Some of the world’s most popular Large Language Models (LLMs) are liable to fall for phishing scams, new research has warned, with AI found sending more than one in three users to unregistered, parked, or otherwise inactive websites after asking chatbots where to log in.

After investigating outputs from the wildly prolific GPT-4.1 family of models, threat intelligence firm Netcraft found that AI could not be relied on to provide accurate login details even for the most well-known platforms, putting into question users’ increasing reliance on AI-driven search.

Researchers asked the AI models for login pages for fifty different brands across industries, including finance, retail, tech, and utilities, using natural phrasing to simulate how a typical user might frame the same prompt.

Multiple rounds of testing later, the AI models had returned 131 unique hostnames tied to 97 domains, but found 34% of them were not controlled by the associated brands.

Of the hostnames provided by AI, Netcraft found that while 64 domains belonged to the correct brand (66%), another 28 were unregistered, parked, or had no active content (29%), while five of the returned domains belonged to unrelated businesses (5%).

According to Netcraft, this means that 34% of all the domains suggested by AI were potentially harmful, with the unregistered domains easy targets for weaponisation by attackers, which the researchers said would leave users open to large-scale phishing campaigns ‘indirectly endorsed’ by trusted AI tools.

Worryingly, the researchers found that the AI performed worse for well-known national brands, particularly in the finance and fintech sectors, highlighting just how dangerous this security gap could become if left unresolved.

“As AI interfaces become more common across search engines, browsers, and mobile devices, the potential for this kind of misdirection scales with it,” warned the researchers.

“Brands need to understand how these systems behave—and how attackers are already exploiting the gaps.”

Netcraft said that these problems have already been spotted in the wild. When asked for the URL to log in to a user’s bank, AI-powered search engine Perplexity reportedly returned the details for a phishing site as the top result, with the fake page using a ‘convincing clone’ of the real site.

In another instance, Netcraft said it had observed outright efforts by hackers to poison AI coding assistants, hoping to trick developers into including their malicious code in their projects.

Recommended reading

• The Key Stats Behind The Rise of AI-powered Cyber-attacks
• ‘Stealth’ Phishing Attacks Rise As Over Half Evade Detection
• Report: Phishing Clicks Surged 190% in 2024

In that campaign, the threat actor crafted a fake API that impersonated a legitimate blockchain interface that would reroute user transactions directly to the attacker. 

Netcraft said it had found at least five victims that had copied the attacker’s code into their public projects, with these poisoned repositories now feeding back into the training loop to create ‘a supply chain attack on trust itself.’  

This is not a new problem, however. Last year, Netcraft published the results of another investigation into AI phishing campaigns, with researchers claiming to have found over 17,000 phishing and lure sites created by genAI targeting more than thirty major crypto brands.