Like it or loathe it, there is no denying that the emergence of popular and ever-advancing AI models promises to revolutionise almost every industry. 

But while AI tech is evolving at an unprecedented pace, questions around bias, misinformation, and unintended consequences continue to spark debate among security professionals racing to keep up.

It’s a dilemma that renowned expert in cybersecurity, Professor Bill Buchanan, recognises all too clearly, and one that he argues could lead to a bleak future for humanity. 

“AI is more important than fire and electricity,” said Buchanan, speaking at the close of DIGIT’s sold-out Scot-Secure conference.

“This is the greatest technology humanity has yet developed. Perhaps machines might replace us and be much better, this is perhaps the future. We’re not very good as a species anyway.”

To most people, even in the tech sector, that kind of statement might sound like doom-mongering – a worry not to be taken overly seriously (for the moment, at least) as businesses continue to plough funds into more AI tools, chasing productivity gains and ever higher ROI.

For those working in cybersecurity, however, it’s a growing concern, with practitioners already acknowledging that AI has become a double-edged sword. 

It’s not just that AI tools have enhanced the scale and sophistication of cyber-attacks – which numerous studies have shown is an ongoing problem – but the threat that advanced systems pose to human comprehension and oversight.

“Each of our systems are goal oriented,” said Buchanan.

“And if the goal is to protect the infrastructure of the system, then the AI may take over and actually overwrite humans.”

Although that threat is growing exponentially, luckily, according to Buchanan, we’re not in danger of being totally replaced yet.    

The Battle Against ‘Bullshit’ AI

Buchanan doesn’t mince his words when it comes to the capabilities of the most popular AI tool on the planet.

“ChatGPT is bullshit. Basically it doesn’t really know what it’s saying.”

“I know it will eventually, but it’s just regurgitating from Stack Overflow, typically. It’s not thinking about the code, it just finds Stack Overflow and then repeats the code back to us.”

It’s a fiery definition for what could be argued is the most advanced AI widely available right now, but Buchanan’s take is backed up by the work of researchers from the University of Glasgow, who have argued that current LLMs are designed to be convincing rather than accurate, resulting in difficult to address issues around trust.

Misinformation, deepfakes, and AI hallucinations are an ongoing problem for anyone relying on AI tools, but pose real threats to security infrastructure. 

Take phishing as a prime example. Last year, a report from email security provider SlashNext found that since the launch of ChatGPT, and the rush to both create and use AI models thereafter, there has been an astronomical 4,151% increase in phishing messages. This is while another organisation, Egress, reported that 82% of phishing toolkits on the dark web mentioned the use of deepfakes.

Added to that, a report from Gartner has predicted that by next year, AI-generated deepfakes will mean 30% of enterprises will no longer consider facial biometrics a reliable method of identity authentication, and there are even those who see a future where ransomware gangs can use AI to attack multiple targets at one time using evolving code, with malicious AI agents being tasked to search out and compromise vulnerable targets. 

For those working in cybersecurity, instances of threat actors using AI in this fashion has become a top concern, with research finding that although 91% of security experts anticipate a surge in AI-driven threats, just 26% are confident in their ability to detect these attacks.

Specialisation Over Automation: Do Humans Still Have the Edge?

Advancements in AI mean cybersecurity professionals not only have to deal with emerging problems like jailbreaking, reverse psychology and model escape, but also must accept the fact that their traditional roles have changed forever.     

“Like it or not, we will face these problems in all our jobs no matter what,” said Buchanan.

“AI will start to be pentesting and assessing your systems for vulnerabilities and not get tired like humans, and not be expensive, and so on.”

To highlight the problem, Buchanan pointed to studies showing that AI is already capable of beating lower level SOC analysts, especially in areas like the early phases of triaging, with humans only able to compete against machines at levels requiring more nuance and experience.

“What we see now is automated security, and this is a worry because obviously our jobs are at risk. Nobody can stand up and say that none of us will lose our jobs through AI.”

However, despite that sombre outlook, Buchanan doesn’t think that the solution is simply to hand the keys to our security over to faceless systems. 

Rather than try to compete with the broad but shallow defence functions of AI, those working in cybersecurity should look to add specific value. 

At the moment, said Buchanan, AI is good at solving lower level, ‘easier’ security problems, but worse at handling those that are harder to resolve, leaving an opportunity for humans to focus on complex, high-stakes threats that require intuition, strategic thinking, and contextual understanding.

“We need to understand what it is we do and what we’re good at, and we need to get better at it. We need to be more specialised and less general.

“If you’re a Level One SOC analyst, you need to become a Level Two SOC analyst, and a two becomes a three, and so on.”

Perhaps the most important role of future cybersecurity practitioners, however, will be knowing when to pull the plug.

Recommended reading

• 86% of C-Suite Leaders Ready to Spend More on AI
• Deloitte: C-Suite Interest in GenAI Has Declined by 15%
• UK Gov’s New AI Sector Study | Key Data & Insights

Every Company Needs a Killswitch Engineer

As security teams hand more and more responsibility over to AI, Buchanan argued that security teams will need to be more proactive in their defence postures and, most importantly, not be overly trusting of the technology.

“If you leave an AI to do things on your network with any permission, you’re going to be in trouble. It can now link up and understand context, it can perceive, it can reason, and it can then act.

“If there’s a hallucination there, and it goes and deletes all your files, then who are you going to call? I don’t think it’s Ghostbusters.”

There is no question that any successful cybersecurity strategy must include AI, but according to Buchanan it is just as vital not to be blind to the existential threat that the technology might pose.

Consider even those ‘everyday’ AI agents that are available on standard smartphones. As Buchanan pointed out, through our constant interactions with them, these AI systems are able to gather more and more insight, and learn where we are most vulnerable.

That’s why building in guardrails now, while it’s still possible for humans to outthink AI, is important. 

“Go back to your CEO and tell them, we need a killswitch engineer. I can assure you, you do need it,” said Buchanan.

“You need to have your red team set up so that it tests properly for hallucinations, you need an audit trail for your AI agents, you should have a killswitch architecture, and have humans in the loop.”

That’s sage advice, which cybersecurity professionals should pay close attention to for their future plans, but with the end goal of AI being the creation of a superintelligence far outstripping our own limitations, perhaps fingers should already be hovering over the AI killswitch.