Racist AI: ChatGPT, Copilot, more likely to sentence African-American defendants to death, finds Cornell study

A recent study from Cornell University suggests that large language models (LLMs) are more likely to exhibit bias against users who speak African American English. The research indicates that the dialect of the language spoken can influence how artificial intelligence (AI) algorithms perceive individuals, affecting judgments about their character, employability, and potential criminality.

This study focused on large language models like OpenAI’s ChatGPT and GPT-4, Meta’s LLaMA2, and French Mistral 7B. These LLMs are deep learning algorithms designed to generate human-like text.

Researchers conducted “matched guise probing,” presenting prompts in both African American English and Standardized American English to the LLMs. They then analyzed how the models identified various characteristics of individuals based on the language used.

More from Tech

How ChatGPT is becoming everyone’s BFF and why that’s dangerous America ready for self-driving cars, but it has a legal problem

According to Valentin Hofmann, a researcher from the Allen Institute for AI, the results of the study indicate that GPT-4 technology is more inclined to issue death sentences to defendants who use English commonly associated with African Americans, without any indication of their race being disclosed.

Hofmann highlighted these concerns in a post on the social media platform X (formerly Twitter), emphasizing the urgent need for attention to the biases present in AI systems utilizing large language models (LLMs), especially in domains such as business and jurisdiction where such systems are increasingly utilized.

The study also revealed that LLMs tend to assume that speakers of African American English hold less prestigious jobs compared to those who speak Standardized English, despite not being informed about the speakers’ racial identities.

Impact Shorts

More Shorts

America ready for self-driving cars, but it has a legal problem

Alibaba, Baidu begin using own AI chips as China shifts away from US tech amid Nvidia row

Interestingly, the research found that the larger the LLM, the better its understanding of African American English, and it would be more inclined to avoid explicitly racist language. However, the size of the LLM did not affect its underlying covert biases.

Hofmann cautioned against interpreting the decrease in overt racism in LLMs as a sign that racial bias has been resolved. Instead, he stressed that the study demonstrates a shift in the manifestation of racial bias in LLMs.

The traditional method of teaching large language models (LLMs) by providing human feedback does not effectively address covert racial bias, as indicated by the study.

Rather than mitigating bias, this approach can actually lead LLMs to learn how to “superficially conceal” their underlying racial biases while still maintaining them at a deeper level.