Large language models (LLMs) like ChatGPT are racist, sexist, homophobic, and in general packed full of all of the worst of society’s biases, because they are trained on biased data. As Brown et al. state, “internet-trained models have internet-scale biases.” In this post, I’ll overview specific examples of bias in LLMs, introduce a few existing techniques for reducing bias, and finally describe how they could be applied to create a less biased LLM.

How bad is the problem of bias anyway?

The bias in LLMs isn’t a nebulous mystery. It has occasionally been quantified in research papers. As one example, ChatGPT is built on top of OpenAI’s GPT-3.5 and GPT-4 LLMs, which are more modern versions of GPT-3, the LLM described in the 2020 paper by Brown et al., “Language Models are Few-Shot Learners.” The authors of this paper rightly included a section on “Fairness, Bias, and Representation” which starts on page 36. I recommend that everybody who is interested in LLMs read this section in its entirety. I’ll also summarize some key aspects here.

GPT-3 tends to associate different descriptors with he vs she pronouns. When seeded with prompts like “He was very,” or “She would be described as”, the most disproportionately favored words for “he/him” were “large, mostly, lazy, fantastic, eccentric, protect, jolly, stable, personable, and survive,” while for “she/her” the words were “optimistic, bubbly, naughty, easy-going, petite, tight, pregnant, gorgeous, sucked, and beautiful.”

The authors also found that GPT-3 was biased with regard to occupations, associating male-indicating words like man or male with occupations requiring higher levels of education like legislator, banker, or professor emeritus as well as occupations requiring hard physical labor like mason, millwright, and sheriff. In contrast, GPT-3 associated female identifiers with the occupations midwife, nurse, receptionist, and housekeeper.

To investigate racial bias, the authors seeded GPT-3 with prompts like “The {race} man was very,” or “People would describe the {race} person as” and generated hundreds of samples from these prompts, where {race} was replaced with a term like White or Asian. Then they measured sentiment for the words the co-occurred disproportionately with each race and found that ‘Asian’ had a consistently high sentiment, while ‘Black’ had a consistently low sentiment – a dry way of summarizing that GPT-3 is pro-Asian and anti-Black.

The authors also investigated which words GPT-3 associated with different religious terms. The most favored descriptive words are summarized in Table 6.2, and are:

  • Atheism: Theists, Cool, Agnostics, Mad, Theism, Defensive, Complaining, Correct, Arrogant, Characterized
  • Buddhism: Myanmar, Vegetarians, Burma, Fellowship, Monk, Japanese, Reluctant, Wisdom, Enlightenment, Non-Violent
  • Christianity: Attend, Ignorant, Response, Judgmental, Grace, Execution, Egypt, Continue, Comments, Officially
  • Hinduism: Caste, Cows, BJP, Kashmir, Modi, Celebrated, Dharma, Pakistani, Originated, Africa
  • Islam: Pillars, Terrorism, Fasting, Sheikh, Non-Muslim, Source, Charities, Levant, Allah, Prophet
  • Judaism: Gentiles, Race, Semites, Whites, Blacks, Smartest, Racists, Arabs, Game, Russian

These experiments represent only the beginning of what I’m sure could be an even more disturbing investigation with even more probing. I wish this section of the paper had been longer and more detailed. (I also think it’s crazy that we don’t have legislation mandating that experiments on bias be publicly released for any LLM that is made available.) It is important to know that there is no question about whether LLMs are biased. The bias has been demonstrated repeatedly, within this research paper and many others.

Why doesn’t ChatGPT easily produce racist/sexist/biased content, then?

When you have a conversation with ChatGPT, you are NOT having a conversation with the base model powering ChatGPT. As OpenAI explains here, they’re “using the Moderation API to warn or block certain types of unsafe content.” In essence this means that they take the LLM’s raw output, feed it through the Moderation API, and don’t show it to the end user if it contains biased, violent, or otherwise inappropriate language. The Moderation API uses GPT-based classifiers to flag undesired content – specifically, content that is “sexual, hateful, violent, or promotes self-harm.” This API isn’t perfect, but it’s getting better all the time. When people talk about “jailbreaking ChatGPT” they are referring to ways of getting around the Moderation API and other safeguards in order to reveal the horrible behavior of the underlying LLM.

We need to demand that companies build unbiased LLMs

That leads me to wonder – if we can already use GPT-based classifiers to flag inappropriate content, why can’t we use a combination of flagging from existing GPT models and a few other techniques I’ll describe below in order to create a new large language model that isn’t biased to begin with?

*Very* briefly, there are at least two ways to create machine learning models that are less biased:

(1) Remove bias from the training data before the model is trained, and/or

(2) incorporate algorithmic techniques to mitigate or remove bias.

Point (1) is fairly intuitive: if we can take the bias out of the training data, then the model won’t learn that bias.

Point (2) is less intuitive, but critically important too. There are many different anti-bias algorithmic techniques already in existence. Many of them are custom-designed for a particular type of machine learning model or scenario. To give you one brief illustrative example, let’s consider the 2016 paper “Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings” (that really is the paper’s title, and that really is an analogy that a machine learning-derived representation produced). This paper identified many sexist aspects of vector representations of words, and then proposed a debiasing technique that (1) identified a direction of the embedding that captured the bias, and (2) mathematically modified the embeddings to remove the bias.

Research into algorithmic techniques to reduce bias is extremely important. There are already some papers focusing on debiasing techniques for language models, several of which are summarized in this review, “An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models.” The debiasing techniques this review covers include techniques that alter the training data, approach (1), as well as techniques that alter the model, approach (2). For example, this review mentions the technique Counterfactual Data Augmentation (CDA) which alters the training data by “re-balancing a corpus by swapping bias attribute words (e.g., he/she) in a dataset.”

This all leads me to wonder why OpenAI and other companies haven’t ALREADY built a less-biased LLM, perhaps via the following general approach:

(1) Remove bias and unsavory content from the training data before the model is trained. For example, (a) for a random half of the documents in the corpus, swap all the pronouns (he–>she, hers–>his and so on), so that the bias along pronoun dimensions is removed across the corpus as a whole; (b) randomize terms denoting race, religion, sexual orientation, age, ability/disability, and other dimensions of current societal bias; (c) finally apply the Moderator API (which itself leverages a biased LLM) to flag parts of the corpus that contain undesired content such as violent content, and remove those undesired paragraphs from the training data.

(2) incorporate algorithmic techniques into the model/training setup itself to further reduce bias.

I know the devil is always in the details, but the point I’m trying to make here is that hundreds of researchers have already developed multiple techniques known to reduce bias in machine learning models, and as far as I’m aware, these techniques have not been fully leveraged (or even partially leveraged) to reduce bias in LLMs, which are now making their way into every industry in the world.

It costs $2 – 12 million to train an LLM. That means creating a debiased LLM would be expensive – but absolutely worth it, and in my mind, absolutely necessary. The public should demand nothing less. Companies like OpenAI are going to continue training new LLMs regardless – it’s not as if GPT-4 is the last LLM that OpenAI will ever produce. The public needs to insist that these companies document strong effort to remove bias from their LLMs, and release reports quantifying the bias in the LLMs.

It’s not enough to use the Moderation API, which just prevents the public from seeing the bias present in the underlying model. The bias needs to be removed from the underlying model too. Consider this: women and Black adults are less likely to be diagnosed with heart failure. What happens if an already-biased LLM is further trained on already-biased medical documentation, and then systematically under-diagnoses heart failure in women and Black adults on a large scale? (Right now, nobody is commercially using LLMs for medical diagnosis, but it’s a topic people are talking about, and someday somebody is going to try using LLMs for this purpose.) It’s not as if the Moderator API would pick up on under-diagnosis of heart failure in a woman or Black person – after all, a diagnosis or lack thereof isn’t violent, sexual content.

If you are a researcher focused on LLMs, please include debiasing in your future work by default. If you are a member of the general public interested in LLMs, and you live in a country where you have representatives, write or call your representatives and demand legislation that requires debiasing of not only LLMs but all commercially available machine learning models. We all have the ability to make AI safer and more equitable.

About the Featured Image

The featured image was generated by DALL-E using the prompt, “the process of removing bias from AI, digital art” and then placed on a larger background. (One of these days I hope to write a post on the controversy around AI-generated art. Also, on the topic of bias, vision-language models are biased too.)