Artificial Intelligence (AI), powered by large language models (LLMs), is transforming the way we approach a range of tasks. In addition to innumerable consumer use cases, powerful AI tools are also making inroads into the corporate sphere, changing how business is done.

The next frontier for AI is tackling the challenge of cybersecurity, in particular through improving the process of testing for vulnerabilities across the ICT ecosystem. Commonly deployed to support rather than to replace the efforts of human experts, AI is a useful weapon when it comes to finding weaknesses, adept as it is at automating processes, improving speed and scale, and detecting threats that old school methods might miss.

But the kind of language models that power most AI use cases are less effective at dealing with security risks than they are with other areas of business. That is because, by their nature, cyber threats come from the kind of shady and unscrutinized places that lie outside the realm of the typical dataset.

“AI and LLMs work by ingesting everything on the public Internet,” explains Dave Gerry, CEO of crowdsourced cybersecurity specialist Bugcrowd. “Then they use that data to train their models to answer questions based on what they’ve seen. But when it comes to security, that method isn’t very helpful as it will only be able to find out about things that are already known.”

The answer lies in developing models that are trained using Reinforcement Learning (RL) rather than standard autocomplete capabilities and static datasets. “What RL does is act as a training gym for AI models,” says Gerry. “They learn based on new data that companies like Bugcrowd provide, thanks to the work of our army of ethical hackers. This teaches them how to find and recognise vulnerabilities, create exploits that validate them and finally how to fix them. It does that using training datasets that we provide, made up of all the data that our products have seen from around the world – proprietary data, not available on the public Internet.”

“RL is key for training LLMs because unlike traditional methods, it learns through trial and error in real time, adapting to new challenges without needing fixed threat patterns, such as you get with labelled datasets,” agrees Rik Turner, chief analyst with independent consulting firm Omdia’s cybersecurity team, with responsibility for covering emerging cybersecurity technology trends. “This makes RL useful in enabling them to detect advanced threats like zero-days.”

To the next level

The role of AI in cyber protection is now moving to a new evolutionary stage with Bugcrowd’s acquisition of Mayhem Security, a pioneer in AI offensive security and a specialist in the next generation of human-led, AI-powered security testing.
The combined operation will take Reinforcement Learning in new directions, claims Dr David Brumley, Chief AI & Science Officer: “One of the big problems of scanning for security risks has been false positives,” he explains. “The latest research suggests that something like 50% of the findings of a typical scan are not actual threats. You don’t want to be training an AI where 50% of the things you’re training it on are fake. You’re effectively training it on hallucinations.”
RL has the power, he claims, to effectively eliminate false positives, hugely helping with the training of foundational models, and doing so at scale: “We’ve delivered hundreds of 1000s of environments to some of these model companies,” he says. “There are very few players out in the market today that are able to approach this at that scale, using real data, not something synthetic.”

Mayhem Security has, he says, been using RL for a while to train agents to carry out actions and solve problems. Now the combined power of Mayhem and Bugcrowd will enable AI-driven offensive cyber testing in new ways, such as helping to analyse applications and attack surfaces more effectively, build automated test plans, learn from attacker behaviour at scale, and teach AI models how to find real vulnerabilities, all adding to the efforts of human testers.

Keeping the human angle is key to the mission. Bringing together Mayhem’s database of hundreds of thousands of vulnerabilities with the Bugcrowd database will provide even richer training data, working like an AI agent to help do the same things that a human security agent can do.

“Customers will, longer term, get more and more interested in using LLMs for security testing,” predicts Brumley. “It will never replace humans, but it will be able to do a lot of the work, especially the low hanging fruit. This will free humans to focus on the very complex vulnerabilities that AI can’t handle yet.”

Mayhem brings the AI talent to help Bugcrowd grow its existing and future AI efforts, says Gerry: “They have the knowhow to train AI agents, feed in Retrieval-Augmented Generation (RAG) data, create automations, and help us to ‘shift left’ earlier into the planning, design, and coding phases of the software development lifecycle.”
It’s not just Bugcrowd and Mayhem making waves here. In the market for security testing solutions, AI looks set to be a game changer with other testing providers poised to move in the direction of more tightly focused AI training as well.

“There is virtually no area of tech that is completely impervious to the application of AI, and threat detection is no exception,” believes Omdia’s Turner. Like Brumley, he doesn’t think that AI agents are going to replace or do away with the human threat researchers just yet: ”I suspect there will always be a need for the human in the loop. Indeed, given that something like 75% of all alerts are actually false positives, the first task for AI agents will be to separate the wheat from the chaff in the alert tsunami coming at human SecOps teams.”

Turner believes that it is important for CISOs to assertively make the case for continued human involvement: “They should avoid the idea that you can just automate threat detection altogether and do away with pesky humans that need to sleep and take the occasional vacation. In that context, I’d say it’s a smart move for Bugcrowd to team up with Mayhem and get ahead of that trend for a human-led approach.”

Data quality is key

Security testing is an essential process, a vital part of identifying vulnerabilities in an organisation’s software and networks. AI is an increasingly important weapon in the world of testing. But if AI is to become a true game changer, then clearly it is only going to be as good as the data on which it is trained. The more dynamic and rich the data, the better the AI model will be. Reinforcement Learning is the secret sauce to drive AI to the next level. Without RL environments, security professionals are limited by AI that consumes the same data that is available to everybody. In the security world, that is not enough. That’s why RL is a certain growth area in the field of training AI agents to identify risks.

By Guy Matthews, Editor, NetEvents