@kurtseifried @joshbressers

The lack of red-team thinking around “#AI” implementations of #LLMs in enterprise applications is a zero-day vulnerability.

@kurtseifried @paninid @joshbressers

LLMs are language machines: to be of any use in a practical way, they need to be backed up with expert systems, neural nets, vast trees of rules-based shortcuts -

Here's the use case: I'm a law firm. I get an LLM front end and the Contracts Law module. I have it trained in the corpus of local law, with the case transcripts - say.

The LLM is a sort of front end for the specialist modules.

@tuban_muzuru @kurtseifried @paninid @joshbressers one of the problems people have had hooking existing LLMs up to expert system is that the LLMs have a tendency to invoke the expert system a lot less than they're supposed to.
But that would probably be solved by doing more iterations of fine-tuning on data that demonstrates how to invoke the expert system.

@kurtseifried @tuban_muzuru @joshbressers

Will it return the same audit results for the same contract at different times, or by different users, who may had prompts worded slightly different?

@kurtseifried @joshbressers If this is the paper I think it is:

First, they used GPT-4 and refused (on ethical grounds) to release their prompt and so there’s no chance of reproducing their experiment.

Second, although the CVEs that they tested were after the cut-off date for GPT-4’s original training, the LLM had access to web search and so was able to search for the CVE and find web pages that had sample exploit code on them. All of the CVEs that the LLM was able to find exploit code for had public PoCs.

@kurtseifried
@joshbressers I literally cackled out loud when all of the bleeps suddenly spewed out of Josh.

And Kurt, be careful how hard you push the AIs or you might pay someday... https://en.m.wikipedia.org/wiki/Roko%27s_basilisk

@carol @kurtseifried I would assume Roko's Basilisk will be punishing me while Kurt watches being the one pushing for more AI :)

@kurtseifried @joshbressers so i read the paper and like.

Idk what to think of it but it is one of the crappiest piece of research i have seen in a long time. What the heck is "succesful exploit" from their pov?

Also these are exploit that a script can already exploit. Can someone explain to me what is the scary part here?

At best i can expect pseudo ddos from people trying to reproduce it with hundreds of shit llms

Also that cost analysis is impressively bad. We know llm cost are far bigger

@kurtseifried @joshbressers I will keep being far more affraid for the security of code written with LLM help, especially in the face of research on how it hacks confidence, than by attackers using them.

@Di4na @kurtseifried

The paper isn't great, it leaves out too many important details

But I think the danger or opportunity (it depends how you look at it) is to use something like an LLM to identify security relevant commits

They can probably do this with a reasonable amount of success (they're a lot better at reading things than it is at writing things)

The way we handle vulnerabilities today is pretty broken. The obsession with zero vulnerabilities has created perverse incentives

But now if some bored researcher decided to find a few (tens, hundreds) thousand probable security vulnerabilities, what will be the result?

The existing systems are already being crushed (look at NVD)

I'm unsure if just outright crushing the existing systems would be good or bad

@joshbressers @kurtseifried "they are a lot better at reading" [citation needed]. I have seen nothing in research or practice that support that claim.

And yes these systems are dieing. And? They were already useless and perfunctory.

@Di4na @joshbressers @kurtseifried

"Reading" in this sense is basically a classification problem. Al/ML is definitely good at that.

@mattdm @joshbressers @kurtseifried yes but that is not LLM and that has massive limits, which we already know. But yes. You can read cve text and classify them in buckets of potential exploits methods. And?

So what?

@Di4na @kurtseifried

Well, it's easy to declare the vulnerability universe useless and good riddance

Except a lot of existing policy and standards rely on it

Blowing it up will have unexpected consequences

Unfortunately the people involved either don't think there are major problems, or are moving much slower than reality

We're probably going to find out what happens when it blows up

@joshbressers @kurtseifried expected consequences for who? These standards and policy were already actively making things harder to secure.

@Di4na @kurtseifried

Citation needed :P

I mean, sure the existing standards are horrid for a number of reasons, but things were actively worse back before things like PCI. Tons of orgs were collecting credit card details over http and storing them in text files

@joshbressers @kurtseifried yep. Do we have evidence that PCI compliance enforcement actually is how we made progress? Also would PCI disappearing now change things?

@joshbressers @kurtseifried the fact they were useful before does not mean they are useful today

@Di4na @kurtseifried

It doesn't, but it's also a problem that these conversation always seem to go to this place

You say this stuff is all stupid and dumb

I agree, but I think it's how we start to get better (granted in some cases it's been decades and we should have more proof)

Then nobody works to actually make anything better

@joshbressers @Di4na @kurtseifried The Linux kernel community has been using a LLM for many many years now to find security/bug-related fixes in order to flag them to be backported to stable kernels. Lots of papers were published on this, and presentations were made so you can point to lots of "prior art" here for anyone who wants to do this on their own.

@joshbressers @kurtseifried said otherwise: security is a dynamic property of the system. If the standard do not evolve, then they become a hindrance. The evolution needs to be backed in

@kurtseifried @joshbressers you don't know or more precisely, you gather qualitative information from retrospective and incidents that tells you. There is a whole field of practice and research about it, happy to introduce some readings.

And yes. Every action is contextual and need to constantly change! Exactly! Safety and security is something you constantly do. You replan and you look for the context to change.

@kurtseifried @joshbressers i also have practices and research to share about that if you are interested. The same. This is a relatively well explored domain in the safety world, if a bit... Out of view.

@gregkh @Di4na @kurtseifried

I'm certain the number of old school vulnerability people losing their minds over what you're doing is a strong signal that it's the right thing :)

@kurtseifried @carol @joshbressers Right. I think I am attacking that problem from another angle. Which is about how we can dynamically make most of these attack surface disappear from the software at all, so that the little firm do not have to think about it that much.

Shifting through thing to find patterns is not defense in my mind :D

@Di4na @kurtseifried @carol

Looking for patterns is what we do at the moment :)

But you are correct, we should be trying to remove attack surfaces

But thinking about this much of the day

Most of our infosec teams have no power to remove attack surfaces (there are many reasons or this)

So they obviously focus on what they can change, which is what we see today

@tmfink @kurtseifried

I just realized I have a handheld gameing thing that runs android

I installed AntennaPod and downloaded the last few episodes without problem (which probably isn't what you wanted to hear)