OpenAI has revealed that some of their AI models have been used to generate fake content, ranging from short social media posts to long-form articles and comments, poised to influence elections.

Since the release of ChatGPT, threat actors have continually used AI tools to support their malicious cyber ventures. From improving their phishing and other social engineering tactics to generating fraudulent content for social media, OpenAI has seen their most prolific AI model being used for criminal activity.

The AI company, which just won a £6.6 billion funding round, said it stalled 20 of these attempts, including a group of ChatGPT accounts that were used to produce fake articles about the US election, among other topics, back in August.

A number of accounts originating in Rwanda were also banned by the company in July, which were used to create comments concerning the country’s election, designed to be posted on X (formerly Twitter).

According to OpenAI, none of these campaigns successfully gained much traction online, Reuters noted from the company’s report.

However, the use of AI in generating and disseminating false information has been a major concern, rising to prominence in 2024 with major elections around the globe.

AI companies have employed heir own methods and even signed accords to quell the use of their technology to spread misinformation and propaganda, but the success of these operations is difficult to analyse.

Recommended reading

• Big Tech Rallies to Counter Deceptive AI Amid Global Elections
• OpenAI Is Trying to Stop AI Meddling in Elections. Will It Be Enough?
• New Measures Announced to Protect UK Elections from Foreign Interference

OpenAI was a signatory of the “Tech Accord to Combat Deceptive Use of AI in 2024 Elections”, which set eight commitments to deploy technology to prevent the use of AI in interfering in elections.

Back in January, the prolific AI company revealed the steps and initiatives it was taking to combat the misuse of its technology, including image provenance tools, and re-directing election questions to independent reliable sources rather than having its chatbot provide answers directly.