The Effect of Demographics on AI Training: A Comprehensive Study

Study Reveals Influence of Annotator Demographics on AI Model Training

A recent study conducted in collaboration between Prolific, Potato, and the University of Michigan has highlighted the significant influence of annotator demographics on the development and training of AI models. The study focused on the impact of age, race, and education on AI model training data and shed light on the potential dangers of biases becoming ingrained within these systems.

The Role of Human Annotation in AI Model Training

Machine learning and AI systems heavily rely on human annotation to effectively train their models. This process, also known as ‘Human-in-the-loop’ or Reinforcement Learning from Human Feedback (RLHF), involves individuals reviewing and categorizing language model outputs to improve their performance.

The Impact of Demographics on Offensiveness Labelling

• The study found that different racial groups have varying perceptions of offensiveness in online comments. Black participants tended to rate comments as more offensive compared to other racial groups.
• Age also played a role, with participants aged 60 or over more likely to label comments as offensive than younger participants.

Demographic Factors in Objective Tasks

The study analyzed 45,000 annotations from 1,484 annotators and covered tasks such as offensiveness detection, question answering, and politeness. The findings revealed that demographic factors continue to impact even objective tasks like question answering. Factors like race and age affected the accuracy of answering questions, reflecting disparities in education and opportunities.

The Influence of Demographics on Politeness Ratings

• Women tended to judge messages as less polite than men.
• Older participants were more likely to assign higher politeness ratings.
• Differences in politeness ratings were observed among racial groups and Asian participants.

The Importance of Accounting for Annotator Demographics

The CEO and co-founder of Prolific, Phelim Bradley, emphasized the need to consider annotator demographics when building and training AI systems. The research highlighted that biases can get embedded into AI systems if the people involved in the annotation process are not nationally representative across age, gender, and race.

Addressing Biases in AI Model Development

As AI systems become more integrated into everyday tasks, addressing biases at the early stages of model development is crucial to avoid exacerbating existing biases and toxicity.

You can find the full research paper here.