ChatGPT is all the rage, and cybersecurity experts are doubtlessly fearful of the AI chatbot’s many applications by threat actors.

But are cyber-criminals actually that keen on using ChatGPT to heighten their prowess? Or if this fear unwarrented?

The research team at Sophos attempted to solve this cyber conundrum by investigating several open forums to see how threat actors – often at the forefront of new technological advancements to better meet their insidious aims – really feel about rising star of tech.

It is no surprise that this is a concern – shortly after the launch of ChatGPT, alternatives deliberately engineered for cyber-crime rose from the ether, including WormGPT and FraudGPT. These removed some of the guardrails of other generative AI chatbot models so criminals could, for instance, access instructions to build bombs, break into safes, or draft convincing phishing texts and emails.

These have been available on phishing and cyber-crime forums on the darkweb, and there is little doubt that they have been used by threat actors to draft phishing emails to steal credentials or receive fraudulent payment.

However, when investigating these uses further, forums of cybercriminals remain unconvinced of the genAI’s revolutionary capabilities, and few have discussed possibilities around its integration outside of drafting cleaner, more correct text.

How are Threat Actors Using LLMS?

After investigating four dark web forums – Exploit, XSS, Breach Forums, and Hackerforums – Sophos’s first impression was that Large Language Models (LLMs) were certainly not a hot topic. They were exponentially less posts about LLMs than cryptocurrencies in the time of investigation, showing that the dark web is not quite buying the hype.

One of the most common threads the researchers did find concerned jailbreaking LLM models. In this case, jailbreaking means bypassing the self-censorship and guardrails that most LLMs come with, which block harmful, illegal, and inappropriate responses.

In conjunction with this, they also discovered a variety of knock-off GPTs that claimed to have these guardrails already removed, ready for threat actors to exploit.

Despite a plethora of options, many users of these forums remained unconvinced by their boasts and capabilities.

Recommended reading

• UK Develops Landmark New Global Guidelines for AI Cybersecurity
• AI Brings Cybersecurity Woes to the Fore
• Is AI’s Threat to Cyber Overblown?

For instance, after garnering a flurry of media attention, WormGPT pulled back on some of its capabilities, saying that is was just a “semi-uncensored AI”, blocking some of its “darker” features.

FraudGPT however received backlash on forums for over-exaggerating its abilities after it claimed to generate “a range of malware that antivirus software cannot detect.”

“Short answer is NO,” one user replied when another asked if this was true.

In the explanation, they detailed how FaudGPT, like all the other forms, relied on publicly available data to draft malware, so it was likely that, even if the malware code could work, it would get picked up by anti-virus software.

As far as any malware actually being generated using AI on these forums, none of it appeared to be sophisticated or novel.

While some threat actors are using the bots to translate text, others are using it to generate simple code or test data.

Despite pushback and scepticism, hacking forums are also employing chatbots to automatically reply to frequently asked questions.

It appears that ChatGPT remains good for somethings, but its use by threat actors is, for now, a mixed bag of simple workarounds, theoretical malware, and translation tools.