State sponsored cyber-criminals are experimenting with Google’s genAI model Gemini to support and accelerate their malicious activities, although they haven’t yet developed novel capabilities, according to the tech giant.

The Google Threat Intelligence Group (GTIG) has released a new report, Adversarial Misuse of Generative AI, detailing state sponsored AI misuse of its Gemini tool by threat actors from Iran, China, North Korea, and Russia.

The analysis covers a range of misuse, such as Advanced Persistent Threats (APT), which includes government-backed hacking, cyber-espionage and destructive computer network attacks, as well as Information Operations (IO), which attempt to influence online audiences in a deceptive, coordinated manner, such as the use of sockpuppet accounts and comment brigading.

For now, the GTIG said that state sponsored threat actors have limited their use of Google’s genAI for research, troubleshooting code, and creating and localising content.

However, Google’s report highlights the danger behind state sponsored AI misuse, with APT actors utilising Gemini to conduct research into vulnerabilities of their targets, for weaponised payload development, and assistance with malicious scripting and evasion techniques.

How is Gemini Being Misused?

According to the report, threat actors from Iran were among the heaviest users of Gemini for these purposes, with over 10 Iranian-backed groups observed using the tool, with a focus on crafting phishing campaigns, conducting reconnaissance on defense experts and organisations, and generating content with cybersecurity themes.

Iranian-linked cyber-criminals were also behind most of the IO efforts observed by Google, accounting for three quarters of all use by IO actors, with most using the tech for content generation, including developing personas and messaging, translation, and to find ways to increase their reach.

Chinese APT actors, meanwhile, were more concerned with researching lateral movement, privilege escalation, data exfiltration, and detection evasion, while those linked to Russia used Gemini for coding tasks, including converting malware into other coding languages and adding encryption functions to existing code.

North Korean threat actors, unsurprisingly, used Gemini to research topics of strategic interest to the North Korean government, such as the South Korean military and cryptocurrency, although GTIG noted that they also used the tool to craft cover letters and research jobs – activities which were linked to North Korean efforts to place ‘fake IT workers’ in Western firms.

None of these threat actors were observed trying to find original ways to use prompt attacks, mostly using basic measures like rephrasing or repeating prompts, and GTIG said that this ‘low-effort’ experimentation, which included copying and pasting instructions to create ransomware, did not bypass Gemini’s safety controls.    

Recommended reading

• North Korean Threat Actors Have Infiltrated UK Firms
• OpenAI Reveals Scale of Threat Actors Using AI to Influence Elections
  
• North Korea Stole $659M in Crypto Last Year, Says US

While Google’s report stresses that LLMs on their own are unlikely to enable breakthrough capabilities, it does anticipate a future where newer AI models and agentic systems could provide these threat actors with a significant edge.

“At Google, we leverage threat intelligence to disrupt adversary operations,” the report concludes.

“We investigate abuse of our products, services, users and platforms, including malicious cyber-activities by government-backed threat actors, and work with law enforcement when appropriate.

“The potential of AI, especially generative AI, is immense. As innovation moves forward, the industry needs security standards for building and deploying AI responsibly. That’s why we introduced the Secure AI Framework (SAIF), a conceptual framework to secure AI systems.”