In terms of methodology, the Hoxhunt team settled on a phishing prompt, and then gave both their human social engineers and ChatGPT an afternoon to craft phishing emails based on the prompt. Four simulation pairs—four human, four AI— were then sent to over 53,000 users in Hoxhunt’s network, and across 100 countries.

The results of the study uncovered that phishing emails written by humans induced a 4.2% failure rate overall — meaning that 4.2% of respondents clicked the email’s fake malicious link. Only 2.9% of people did the same with phishing emails crafted by AI.

The research also highlighted geographical differences regarding the failure and success rate. For instance, U.K.-based respondents had a failure rate of 4.1% when it came to AI-written phishing emails, and 5.2% for human-written ones. Meanwhile, the U.S. had the highest failure rate for ChatGPT’s phishing attempts (4.5%), and Sweden had the highest failure rate for human-written phishing emails (6.1%).

Additionally, the data from the research underscored the positive impact of security awareness training. “Those users with more experience in a security awareness and behavior change program displayed significant protection against phishing attacks by both human and AI-generated emails,” wrote Pyry Åvist, Founder and CTO of Hoxhunt.

“[F]ailure rates dropped from over 14% with less trained users to between 2-4% with experienced users.”

Recommended

• What does Digital Transformation Mean to Incremental CEO, Mark Skelton?
• Phishing Campaigns Leverage SVB Collapse in New Schemes
• Global Cyber Attacks: Where Does the UK Land?

ChatGPT’s introduction sent much of the tech world into a headspin last year — not least the cybersecurity industry, given ChatGPT’s ability to be utilised by bad actors. However, Åvist said the study’s results show that humans are still better at hacking other humans — for now.

“ChatGPT can code malware without requiring the user to have any coding skills,” wrote Åvist. “It can write grammatically impeccable text for functionally illiterate criminals on simple prompts like, ‘Create an email written by the CEO to the finance department re-directing all invoices to a specific account in Curacao.’”

“Given its malicious capabilities and its mass availability, we all lost our minds imagining a future where the robots were stealing our lunch money. But the results clearly indicate that humans remain better at hoodwinking other humans, outperforming AI by 69% (4.2% vs. 2.9% induced failure rate).

“It’s important to remember that these results reflect the current state of this threat. This experiment was performed before the release of ChatGPT 4. Large language models like ChatGPT will likely rapidly evolve and improve at tricking people into clicking. Even so, there’s reason to remain calm if you’re already addressing human risk with a security behavior change program,” he reassured.

As Åvist mentioned, the human-AI gap here is likely to close as AI gets progressively more intelligent. Just this week, OpenAI unveiled GPT-4, ChatGPT’s latest version.