New research analysing data from the Prolific research platform reveals how the demographics of people labelling the data used to build and train AI models influences their decisions.

What one person finds offensive, another may find perfectly acceptable – which has major ramifications for the development of AI systems with the danger that existing biases – perpetuated by a lack of diversity – are baked into them and amplified.

Machine learning (ML) and artificial intelligence systems often rely on high-quality human labelling and annotation – people reviewing and categorising the output of language models to train them.

This would include using humans to monitor for offensive language and hate speech, to better understand toxic behaviour and human intentions – often referred to a ‘Human-in-the-loop’ or Reinforcement Learning from Human Feedback (RLHF).

The study, conducted collaboratively by Prolific, Potato, a web-based annotation tool, and the University of Michigan, found that age, race, and education are statistically significant factors in determining how something is labelled.

For instance, when asked to rate the offensiveness of online comments, Black participants tended to rate the same comments as being significantly more offensive compared to other racial groups.

Prior research on annotator background has mostly focused on specific aspects of identity, like gender, and on certain tasks, like toxic language detection.

This study aimed to undertake a much broader analysis, including offensiveness detection, question answering and politeness. The dataset contains 45,000 annotations from 1,484 annotators, drawn from a representative sample regarding sex, age, and race as the US population.

The ratings were compared with annotations from a Ruddit data set which judged 6000 comments from Reddit for offensiveness and politeness.

Offensiveness Detection

The study identified some discrepancies in the rating of offence comments across various demographics.

Across gender, men and women showed no statistically significant difference in their offensiveness ratings – the data did not provide a distinction for non-binary individuals, however.

There were significant differences in offensiveness ratings across various racial groups.

Black participants rated the same comments as more offensive that all other racial groups in the study. The scores of white participants strongly correlated with the original Ruddit data set, potentially indicating that this study mainly had white participants.

There were also differences between ages – those over 60 years of age tended to find comments more offensive than their middle-aged/younger counterparts.

Interestingly, the study found no significant differences with respect to the education level of different participants.

Politeness 

The study went further than offensive comments and asked participants to rate the politeness of responses to see if there were demographic discrepancies.

Women tended to find messages as being less polite than men, and older participants were more likely to give higher politeness ratings.

Those with higher education levels tended to give lower ratings than those with less formal education.

Black participants rated messages as being more polite than their white peers, but Asian participants gave the lowest politeness overall across different racial groups.

Recommended

• ChatGPT to the Left, Meta’s LLaMA to the Right: Stuck in the Middle of AI Bias
• Tech Sector “Diversity Disaster” Contributing to Biased AI Systems
• UK Gov Tasked with Tackling Diversity in STEM

Diversity and AI Bias 

Commenting on the research, Phelim Bradley, CEO and co-founder of Prolific said, “Artificial intelligence will touch all aspects of society and there is a real danger that existing biases will get baked into these systems.

“This research is very clear: who annotates your data matters. Anyone who is building and training AI systems must make sure that the people they use are nationally representative across age, gender, and race or bias will simply breed more bias.”

With tech dealing with a major gender gap and racial gap, not to mention a lack of data for wider gender diversity and neurodivergent workers, AI bias can be exacerbated by a lack of diversity

“Systems like ChatGPT are increasingly used by people for everyday tasks,” says assistant professor David Jurgens from the University of Michigan School of Information. “But on whose values are we instilling in the trained model? If we keep taking a representative sample without accounting for differences, we continue marginalising certain groups of people.”

AI models have recently come under fire for holding political biases as well. Most notably, a study was released tracking the political leanings of ChatGPT according to a Political Compass Test, finding that is aligned more with left-leaning and liberal ideologies.

However, another study on other large language models found that ChatGPT was one of many models that leaned more to the left according to the Political Compass Test, with other models, such as Meta’s LLaMA, leaning to the authoritarian right in its Political Compass results.