“The genie is out of the bottle,” Jordan Schroeder echoed throughout his presentation on generative AI.

The proverbial genie is artificial intelligence, and the “bottle” refers to the deep ethical and theoretical questions surrounding its introduction to society, the potential dangers lurking within the emerging technology, and how this can transform society for better or worse.

Questions, Schroeder says, which are both valid and out-of-date.

The genie is out of the bottle, the world of AI is here.

Schroeder likened GenAI to introducing a fully capable laptop with the full breadth of the internet to a mediaeval farmer – “for good or for ill, it’s going to have an impact for years.”

An impact that cybersecurity experts need to be prepared for.

AI is the fastest adopted technology ever, Schroeder reiterated to an audience that is, at this point, probably aware that ChatGPT was the fastest technology application to reach a user base of 100 million.

“That should scare you, and excite you!” Schroder said, touching on the whiplash generative AI has sent through the technology ecosystem. The emerging technology has introduced a realm of new opportunities for increasingly efficient productivity, more data-driven decision making, and transformative operations with potential implications for our entire socio-economic model.

The possibilities appear endless, but then, so do the dangers.

Generative AI’s purpose, Schroeder attempted to make clear throughout his presentation, is to create new information – “its purpose is to do something novel.”

This, however, leads to its own issues. The level of “novelty” can range, from changing a single word from a copyrighted material to rendering art in the exact style of an illustrator, to creating fake references and providing completely false information in the form of “hallucinations.”

Each end of the “novelty” spectrum creates their own legal and ethical issues, but Schroeder says this phenomenon is the entire point of the technology, to create something new.

“AI takes a lot of source material, tweaks it, modifies it, and stitches it back together into something novel,” he said. “Understanding this model is very important for business and security leaders because it influences how we use it, determine its appropriate use for our organisations, and how we determine that we are able to use it.”

Schroeder urges businesses to keep an eye on the trends as well: a survey from Salesforce showed that 50% of people have tried ChatGPT or another generative AI chatbot, and that out of this half, a third have used it daily, and 60% are on their way to master the technology.

Now, there is no definition for what it means to “master” generative AI, but Schroeder says that these potential “masters” are “our biggest risks, because they are going to use [generative AI] in ways we cannot predict. In weird and wonderful ways, but ways we cannot predict.”

Schroeder took the audience through some of the inherent risks in the trends of emerging generative AI technology, as well as specific security risks, and ways to mitigate these in the company.

GenAI: The Inherent Dangers

Generative AI has its own lurking dangers even without specifically mentioning cybersecurity risks, which Schroeder pointed out.

Most generative AI chatbots function based on prompts provided by the user to then determine an output. These prompts can be manipulated to derive strange or weird answers, or derive from copyrighted material.

Schroeder even said that an active area of research for security leaders is if prompts can be manipulated to let users pull data from other users’ chatbot sessions and their personal data.

“Anything that has user inputs has the risk of injection,” he said, meaning an injection into an AI chatbot’s output in any given session. The impact of this data risk is being lessened as it is researched and resolved, but still presents a data security risk.

This ties into the inherent danger of inputting sensitive information into an AI chatbot – most generative AI chatbots, including ChatGPT, keep your session data to further train their model. So disclosing sensitive data to a third party can be a risk workers are unaware of.

According to Schroeder, ChatGPT itself is very raw as a website – “it doesn’t have a lot of features.” Programmers and developers have therefore developed browser plugins to create extra detail and specifically tailor ChatGPT, but these tend to not be entirely secure since they are typically made by individuals lacking specific cyber training.

The other concerning factor surrounding generative AI is overreliance – despite being introduced to the wider public just this year, many are already overestimating the accuracy of large language models (LLMs), according to Schroeder.

Trust in AI ranges with the ages, with around half (52%) of Gen Z trusting not just the information it provides, but trusting it to make “informed decisions,” according to the Salesforce survey.

Generative AI does not just struggle with hallucinations, but struggles with intermediate coding, according to Schroeder, and often needs human input to not exacerbate bias.

One of the most highlighted issues with LLMs including ChatGPT and other chatbots is how it collects the information it uses to create output.

Often called data scraping, this method can indiscriminately scrape intellectual, copyrighted, and personal information, which it can then “learn” from, and even regurgitate “word for word, something you or your company has written” Schroeder said.

These are just the generative AI concerns at the most basic level, according to Shroeder, and they are riddled with questions surrounding data protection, personal information, copyright infringement, information integrity, and trust.

Generative AI: The Cybersecurity Woes

Schroeder went through the full spectrum of emerging cybersecurity trends utilising AI technology, with his own demonstration highlighting just how pernicious these threats can be.

At DIGIT, we have covered the myriad ways threat actors have utilised generative AI to advance their techniques and stump cybersecurity experts, from using AI to code malware and ransomware not seen before, from using models to craft specific and well-worded spearphishing campaign emails.

WormGPT is even allowing threat actors to scan the internet without guardrails to find illegal information, “its like a ChatGPT without guardrails,” Schroeder explained.

However, one of the ways generative AI is threatening security, is difficult to explain through the written word.

Deepfakes, where AI is used to falsify images, videos, or audio recording to imitate a person’s likeness to a realistic degree, have terrifying implications, as shown through a live demonstration Schroeder presented at DIGIT Scot Secure West.

Using generative AI models and a recording of DIGIT founder Ray Bugg, Schroeder plugged in Ray’s ‘voice’ into ChatGPT, getting to provide live answers in a conversation between Schroeder and ‘Ray’ – with previous permission of course.

In the recording Schroeder could produce a very realistic version of Ray’s voice, with a somewhat realistic version of his countenance and opinions. For example, Schroeder’s deepfake version of Ray is a fan of Manchester United, much to the real Ray’s dismay.

Schroder also got ‘Bot Ray’ to request a payment of £100 pounds to a fake account, which shows that deepfake voice automations could be used to bypass biometric voice recognition services, sparking a conversation about what safety measures will even be left out of reach of generative AI’s insidious reach.

“I think voice biometrics is now dead,” he said. “it is reproducible — I think this will actually kill voice biometrics.”

Recommended reading

• AI: Cyber-friend or Cyber-foe?
• Top Cyber Threat Detections of 2023 Driven by AI Advancements
• Comment | The Dark Side of Generative AI: Unveiling the Cybersecurity Risk

Personal identification is now a “massive puzzle” cybersecurity experts will have to reconfigure as advances in AI overtake current security measures.

AI and Security: How do they Co-exist

So what can companies do to responsibly use AI? According to surveys, not everyone is willing to even try.

75% of organisations, according to Salesforce, are considering outright bans on using AI within their organisation, and 61% planning on making these bans permanent.

However, Reuters found that a “significant number of users continue to use genAI after bans,” Schroeder said, for their work.

“As security specialists, we don’t want to be the Department of No. We want to be the Department of Safely Yes,” he said.

So if bans do not work, Schroeder has some steps companies can take to mitigate the risks of AI – as he said, the genie is out of the bottle, and now it’s time for security experts to reckon with its power.

To actually enforce a ban, Schroeder recommends zero trust tools to ensure that unauthorised sites are blocked, and to keep track of what AI is being used for if some allowances are maintained.

Further, to mitigate against spear phishing attacks, Schroeder mentioned new cybersecurity innovations using AI to scan emails for potential malware and threats.

Overall, Schroder emphasises the importance of policy – “I love policy” he admitted.

“Policies are living documents and should provide a feedback loop so that you know what it’s being used for and what you need to do to protect it,” he said.

Providing training involved in the policy, including training not sharing sensitive data and weeding out potential biases, is integral to this method.

Allowing for feedback and innovation surrounding the use of an emerging technology is also essential.

As the technology continues to develop, and innovations challenge security leaders, Schroeder is often left with comments and questions surrounding what this means for technology, cybersecurity, and humanity as a whole. People often wonder if the advancements promised with AI are worth all the potential risks and dangers, if we are ready for such a new era in innovation, if we can handle it responsibly and ethically.

“And I understand, and agree with these questions,” he said. “But as security people, we have to deal with the genie.”