According to a recent study, the mere belief in AI assistance can significantly boost performance, which highlights the challenges in assessing the true impact of AI technologies amidst the powerful influence of the placebo effect.

The study employed a clever setup, where participants engaged in a letter recognition exercise, purportedly assisted by an AI system. Half were led to believe the AI was reliable and would enhance their performance, while the other half were informed it was unreliable and would hinder their performance.

However, there was no AI system. The supposed AI assistance was entirely random. Yet, both groups exhibited improved performance when they believed they were aided by AI.

“Participants were led to believe an AI system was assisting them, when in reality, what the sham-AI was doing was completely random,” explains Agnes Kloft, one of the authors of the study. 

In a subsequent online replication study, consistent results emerged, further underscoring the potency of the placebo effect in AI-assisted tasks. Additionally, qualitative insights gleaned from participants’ expectations revealed an overwhelmingly positive outlook toward AI, even among sceptics.

The study challenges conventional methods used to evaluate AI systems, suggesting that the placebo effect may skew assessments. 

“This is the big realisation coming from our study—that it’s hard to evaluate programs that promise to help you because of this placebo effect,” said assistant professor Robin Welsch.

“What we discovered is that people have extremely high expectations of these systems, and we can’t make them AI doomers simply by telling them a programme doesn’t work,” continued Welsch.

As large language models and other AI innovations continue to evolve, acknowledging and mitigating the influence of the placebo effect becomes imperative for accurate evaluation and informed decision-making.