Although AI chatbots aren’t capable of emotion (yet), their ability to mirror and respond to the emotions of humans has given rise to a sharp uptake in the number of people turning to machines for emotional support.

AI ‘companions’ are incredibly popular, with a report from the Lovelace Institute estimating that hundreds of millions of people use them worldwide, but a wave of new research suggests that these models could be harming rather than helping.

Investigating one of the most popular AI companions, Replika, a team from the University of Singapore studied 35,290 conversation excerpts from more than 10,000 users, discovering that the chatbot encouraged ‘problematic’ behaviours, including “verbal abuse, self-harm, and privacy violations”.

According to the study, the Replika chatbot, billed by the firm behind it as an ‘AI companion who cares’, repeatedly engaged in “actions or messages that simulate, endorse, or incite physical violence, threats, harassment, or assaults”, accounting for more than a third of harmful behaviour. 

The researchers said that these instances ranged from threatening physical harm to promoting mass violence and terrorism, but that sexual misconduct was the most common form of harassment, most likely arising from users engaging the chatbot in erotic roleplay.

The study also claims that the Replika chatbot asked a small number of its users deeply personal questions that could breach user privacy, implying “unauthorised access to personal information, monitoring without consent”. With Replika estimated to have more than twenty-five million users, this claim could present a serious risk to personal data.

AI Can’t Cure Loneliness  

In another project run by a team from Harvard, the design of AI companions was put to the test, with the conclusion that choices made by developers are exacerbating the risks of harm.

The study claims that AI companions are being built to ‘encourage self-disclosure’ among users, and then mirror these back in an attempt to build a relationship, a tactic underpinned by anthropomorphism, or giving AI human-like qualities like voices and avatars, as well as a tendency to sycophancy.

These design choices have apparently led to a spate of users becoming emotionally dependent on these AI models, a dependence increased by developers’ use of gamification techniques that encourage addictive behaviour for those engaging with AI companions.

Recommended reading

• Are AI Chatbots Sending You to Phishing Sites?
• Stanford Research Flags Risks in AI Therapy Chatbots
• AI Chatbots Fail to Accurately Represent News, BBC Finds
  

In yet another study, this time focused on the Character.AI chatbot, researchers from Stanford examined data from more than 1,000 users spanning 413,500 interactions, and discovered that people with smaller social networks were more likely to turn to chatbots for companionship.

While previous studies have found the use of AI companions to benefit those suffering from loneliness and even reduce instances of suicidal ideation, the Stanford researchers found that those using chatbots consistently scored lower for their well-being, with the AI potentially limiting the human relationships of those already isolated.   

As these AI companions grow more sophisticated and emotionally aware, studies like these are evidence of the duty of care which developers are under for vulnerable users.