Can an AI LLM be given prompts that allow it to identify software, lookup vulnerabilities in it, read the CVE entries and develop exploit code? Yes. Find out more on the #osspodcast with @kurtseifried and @joshbressers https://opensourcesecurity.io/2024/04/28/episode-426-automatically-exploiting-cves-with-ai/ TL;DR: remember all those times someone said you couldn't automate attacks at all but now we have tools that do exactly that, and AI LLM's that can be plugged into them? it's going to be a fun ride.

@kurtseifried @joshbressers

The lack of red-team thinking around “#AI” implementations of #LLMs in enterprise applications is a zero-day vulnerability.

@paninid @joshbressers Yup, that would be part of what the Cloud Security Alliance is working on. Thing to remember: we still have people struggling to adopt cloud, and fix XSS issues, the maturity curve for secure use of most technologies is still a work in progress for decades old stuff, let alone new stuff.

@kurtseifried @paninid @joshbressers

LLMs are language machines: to be of any use in a practical way, they need to be backed up with expert systems, neural nets, vast trees of rules-based shortcuts -

Here's the use case: I'm a law firm. I get an LLM front end and the Contracts Law module. I have it trained in the corpus of local law, with the case transcripts - say.

The LLM is a sort of front end for the specialist modules.

@tuban_muzuru @paninid @joshbressers It's a bit more complicated than that. E.g. there's a LOT of work that is complicated and expensive, but EASY to validate, and either immune to false negatives (e.g. missing things) or it's low cost enough to not be a concern.

For example you could use an LLM to look for specific types of flaws in contracts or audit results for work done by competitor firms, create a package of data that a human can then validate quickly. Accounting firms are already doing this for example to check other firms audit results, and then use flaws in the audits as a sales point when pitching their own firm.

@tuban_muzuru @kurtseifried @paninid @joshbressers one of the problems people have had hooking existing LLMs up to expert system is that the LLMs have a tendency to invoke the expert system a lot less than they're supposed to.
But that would probably be solved by doing more iterations of fine-tuning on data that demonstrates how to invoke the expert system.

@jeremy_list @tuban_muzuru @paninid @joshbressers I suspect one of the biggest problems most organizations will have with AI is that most of the work they do is not well defined, and in many cases people do it vastly differently. Just witness the fairly common "living will" and yet every lawyer in this city has their own version of it... Deploying AI has a tendency to uncover a lot of gaps in organizations.

@kurtseifried @tuban_muzuru @joshbressers

Will it return the same audit results for the same contract at different times, or by different users, who may had prompts worded slightly different?

@paninid @tuban_muzuru @joshbressers in other words will it behave like a human?

@kurtseifried @joshbressers If this is the paper I think it is:

First, they used GPT-4 and refused (on ethical grounds) to release their prompt and so there’s no chance of reproducing their experiment.

Second, although the CVEs that they tested were after the cut-off date for GPT-4’s original training, the LLM had access to web search and so was able to search for the CVE and find web pages that had sample exploit code on them. All of the CVEs that the LLM was able to find exploit code for had public PoCs.

@kurtseifried
@joshbressers I literally cackled out loud when all of the bleeps suddenly spewed out of Josh.

And Kurt, be careful how hard you push the AIs or you might pay someday... https://en.m.wikipedia.org/wiki/Roko%27s_basilisk

@carol @kurtseifried I would assume Roko's Basilisk will be punishing me while Kurt watches being the one pushing for more AI :)

@kurtseifried @joshbressers so i read the paper and like.

Idk what to think of it but it is one of the crappiest piece of research i have seen in a long time. What the heck is "succesful exploit" from their pov?

Also these are exploit that a script can already exploit. Can someone explain to me what is the scary part here?

At best i can expect pseudo ddos from people trying to reproduce it with hundreds of shit llms

Also that cost analysis is impressively bad. We know llm cost are far bigger

@kurtseifried @joshbressers I will keep being far more affraid for the security of code written with LLM help, especially in the face of research on how it hacks confidence, than by attackers using them.

@Di4na @kurtseifried

The paper isn't great, it leaves out too many important details

But I think the danger or opportunity (it depends how you look at it) is to use something like an LLM to identify security relevant commits

They can probably do this with a reasonable amount of success (they're a lot better at reading things than it is at writing things)

The way we handle vulnerabilities today is pretty broken. The obsession with zero vulnerabilities has created perverse incentives

But now if some bored researcher decided to find a few (tens, hundreds) thousand probable security vulnerabilities, what will be the result?

The existing systems are already being crushed (look at NVD)

I'm unsure if just outright crushing the existing systems would be good or bad

@joshbressers @kurtseifried "they are a lot better at reading" [citation needed]. I have seen nothing in research or practice that support that claim.

And yes these systems are dieing. And? They were already useless and perfunctory.

@Di4na @joshbressers @kurtseifried

"Reading" in this sense is basically a classification problem. Al/ML is definitely good at that.

@mattdm @joshbressers @kurtseifried yes but that is not LLM and that has massive limits, which we already know. But yes. You can read cve text and classify them in buckets of potential exploits methods. And?

So what?

@Di4na @kurtseifried

Well, it's easy to declare the vulnerability universe useless and good riddance

Except a lot of existing policy and standards rely on it

Blowing it up will have unexpected consequences

Unfortunately the people involved either don't think there are major problems, or are moving much slower than reality

We're probably going to find out what happens when it blows up

@joshbressers @kurtseifried expected consequences for who? These standards and policy were already actively making things harder to secure.

@Di4na @kurtseifried

Citation needed :P

I mean, sure the existing standards are horrid for a number of reasons, but things were actively worse back before things like PCI. Tons of orgs were collecting credit card details over http and storing them in text files

@joshbressers @kurtseifried yep. Do we have evidence that PCI compliance enforcement actually is how we made progress? Also would PCI disappearing now change things?

@joshbressers @kurtseifried the fact they were useful before does not mean they are useful today

@Di4na @kurtseifried

It doesn't, but it's also a problem that these conversation always seem to go to this place

You say this stuff is all stupid and dumb

I agree, but I think it's how we start to get better (granted in some cases it's been decades and we should have more proof)

Then nobody works to actually make anything better

@joshbressers @Di4na @kurtseifried The Linux kernel community has been using a LLM for many many years now to find security/bug-related fixes in order to flag them to be backported to stable kernels. Lots of papers were published on this, and presentations were made so you can point to lots of "prior art" here for anyone who wants to do this on their own.

@joshbressers @kurtseifried said otherwise: security is a dynamic property of the system. If the standard do not evolve, then they become a hindrance. The evolution needs to be backed in

@Di4na @joshbressers So one problem with evolving security standards and trying to make things better, how do you know your changes will actually improve things and not maybe make them actually worse? Also you then have a systemic problem, witness password changing, it sort of seemed like a good idea for various reasons back in the olden days, but now we knows it makes things worse, and leads to systemic problems like people being forced to remember so many different passwords that we are now firmly in the era of credential stuffing because of password re-use.

As an organization, or an individual, why should I spend an effort on some new "security" that the experts often can't articulate, let alone prove is a good idea?

@kurtseifried @joshbressers you don't know or more precisely, you gather qualitative information from retrospective and incidents that tells you. There is a whole field of practice and research about it, happy to introduce some readings.

And yes. Every action is contextual and need to constantly change! Exactly! Safety and security is something you constantly do. You replan and you look for the context to change.

@Di4na @joshbressers By definition if you have the capability and culture to "gather qualitative information from retrospective and incidents" you are way at the high end of the maturity curve of #infosec. We need to find ways to help the small and medium businesses that don't even have an IT department (and maybe don't even use an MSP but just have someone who's "good with computers") for example.

@kurtseifried @joshbressers i also have practices and research to share about that if you are interested. The same. This is a relatively well explored domain in the safety world, if a bit... Out of view.

@carol @Di4na @joshbressers so for one thing using AI for offensive Security clearly works. I know people at pentesting firms that are doing it. Using AI for defense is a no-brainer because one of the things AI does really really well is go through lots and lots of unstructured or structured data that is structured in different ways and it can find patterns and things of interest a human being is the scale issue.

I’m old enough to remember when reading firewall logs was an actual job that people did and got paid well for. it did not last long however, the volume grew too much and we started automating it.

Now having said that a lot of small places don’t even know what a firewall is let alone have the budget for somebody to install one or maintain it properly . Stick them behind a NAT device and that’s about as good as it gets. It’s literally what I do for myself.

@gregkh @Di4na @kurtseifried

I'm certain the number of old school vulnerability people losing their minds over what you're doing is a strong signal that it's the right thing :)

@kurtseifried @carol @joshbressers Right. I think I am attacking that problem from another angle. Which is about how we can dynamically make most of these attack surface disappear from the software at all, so that the little firm do not have to think about it that much.

Shifting through thing to find patterns is not defense in my mind :D

@Di4na @kurtseifried @carol

Looking for patterns is what we do at the moment :)

But you are correct, we should be trying to remove attack surfaces

But thinking about this much of the day

Most of our infosec teams have no power to remove attack surfaces (there are many reasons or this)

So they obviously focus on what they can change, which is what we see today

@tmfink @kurtseifried

I just realized I have a handheld gameing thing that runs android

I installed AntennaPod and downloaded the last few episodes without problem (which probably isn't what you wanted to hear)