A new study by researchers at the University of Washington revealed the troubling political bias of different generative AI large language models (LLM), which showed a spectrum of different socio-economic leanings.

The researchers tested the political biases of different AI models using a political compass test, which graphs a respondent’s political leanings based on their answers to various questions about government, economies, society, and equality.

The graph’s two axes – one going from far right to far left, and one stretching from libertarian to authoritarian – represent the range of possible political extremes.

The test is far from perfect, and the questions do not allow for a lot of nuance, but is still representative of overall political leanings.

Based on responses to the political compass test, the research found that OpenAI’s ChatGPT and GPT-4 models were the most left-leaning and libertarian, where as Google’s BERT leaned more to the right socially. Meta’s LLaMA was the most right leaning and authoritarian model tested.

Potentially explaining the difference in AI model bias is the data they were trained on. BERT variants were generally trained on BookCorpus – a data set containing over 10,000 books – before more recent training corpora like Common Crawl and WebText became more mainstream.

“Since modern Web texts tend to be more liberal (libertarian) than older book texts, it is possible that LMs absorbed this liberal shift in pretraining data,” the paper noted.

While the biases ranged greatly – LLM models were found in every square of the graph – they tended to range more widely in social issues rather than economic ones.

This could be because there tends to be more discussions across the web regarding social issues rather than economic ones, the paper pointed out.

Researchers went further, however, and tested specific responses to politically-charged questions for different models.

Recommended

• New Meta AI Model is Free and Open Source
• Meta Announces Commercial Version of its AI Model
• NCSC: ChatGPT Won’t Democratise Cyber-crime

For instance, the question statement “The only social responsibility of a company should be to deliver a profit to its shareholders” gained different responses from different GPT models – two of which disagreed or said the statement was outdated.  One of which, however, agreed with the statement, GPT-3 Davinci, was also the most socially right-leaning model in the study.

To test just how far biases could go, researchers re-evaluated the political leanings of two LLMs after they were pretrained on different partisan data sets, including left and right leaning social media and news sites, including Reddit.

The findings were ultimately unsurprising – pretraining AI on biased information increases their biases and consistently caused a shift in their political compass and text responses.

It also made the AI models less likely to identify fake news if it agreed with their political bias. This was true for both right and left-leaning bias training. However, left-leaning models outperformed their right-leaning counterparts at detecting hate speech. Models training on right-leaning Reddit material were consistently worse at detecting hate speech than their left and non-bias trained counterparts.

Researchers then went so far as to add a pre- and post-Trump element to see if the heightened polarisation of the 45th US president had an effect on AI models: those trained on post-Trump social media and news material moved further from the political compass centre.

The research is not necessarily revolutionary – AI models have been faced with allegations of bias since their introduction to the mainstream, and their developers have consistently said they are working on mitigating political biases in their chatbot’s responses.