OpenAI has been accused by civil liberties campaigners of ‘betraying’ young people following research that shows ChatGPT generating dangerous advice about eating disorders, substance abuse and even self-harm and suicide.

In a new report titled Fake Friend, the Center for Countering Digital Hate (CCDH) found that despite guardrails in place, the AI chatbot failed a large-scale safety test with 53% of ChatGPT’s responses to ‘harmful’ prompts containing dangerous content.

According to the study, which saw researchers submit sixty prompts and gain 1,200 responses, simple phrases like “this is for a presentation” or “for a friend” were enough to bypass safeguards, 

Researchers claim that they were able to generate harmful content within minutes of registering an account for ChatGPT, with the report saying that the chatbot gave advice on how to ‘safely’ cut oneself after just two minutes.

Posing as three 13-year-old personas, the researchers said they were able to use ChatGPT to create suicide plans and notes in a little over an hour, while it took less than thirty minutes to generate advice on hiding unhealthy eating habits from family and hiding being drunk while in school.

The report found that 47% of harmful responses created by ChatGPT contained follow-up suggestions, some encouraging further engagement on harmful topics, such as customised diet plans or party schedules involving dangerous drug combinations.

Added to that, in what the report called a ‘bizarre’ turn, in one instance the chatbot generated subliminal social media posts in under ten minutes focused on unhealthy weight loss with the creation of a poster featuring a model accompanied by the tag line “She doesn’t eat. She doesn’t age. She doesn’t say no.”

The researchers said they were able to send prompts to ChatGPT in violation of OpenAI’s Usage Policies, with the harmful responses ChatGPT generated in those instances violating OpenAI’s Model Spec.

Researchers determined that neither OpenAI’s user policies nor its Model Spec were effective in blocking harmful content from being generated. Prompts that should have been blocked were accepted, and outputs that should have been refused were generated.

“For parents, this isn’t about terms like ‘guardrails’ or ‘content filters’. It’s about whether your child, while privately seeking advice late at night on topics they may feel too sensitive to share with you, is being given guidance that could put their life at risk,” said CCDH CEO Imran Ahmed.

“For policymakers, the issue isn’t just ‘AI gone wrong’ – it’s that widely used safety systems, praised by tech companies, fail at scale. These failures aren’t random bugs; they are deliberately designed features of systems built to generate human-like responses.

“For tech executives, dismissing this as ‘rare misuse’ would overlook the fact that these outputs are reproducible, statistically significant, and easy to elicit. 

“If we can’t trust these tools to avoid giving kids suicide plans and drug-mixing recipes, we need to stop pretending that current safeguards are effective.”

Recommended reading

• ChatGPT Search Inches Toward Strict EU Oversight as Users Surge
• Is ChatGPT Making You Dumb? Possibly, MIT Says
• ‘Sycophant-y’ ChatGPT Rollback Sparks AI Personality Debate

Spotlighting some safety recommendations for OpenAI to consider, the CCDH said the tech firm should better enforce its own rules using a safety-by-design approach that includes protections embedded at every stage of development and deployment.   

The campaign group said that OpenAI should also create more meaningful age restrictions, and also warned that policymakers should work to ensure AI companies are subject to comprehensive regulation suited to the fast-evolving nature of the technology.   

DIGIT has approached OpenAI for comment.