@kurtseifried @joshbressers

The lack of red-team thinking around “#AI” implementations of #LLMs in enterprise applications is a zero-day vulnerability.

@kurtseifried @paninid @joshbressers

LLMs are language machines: to be of any use in a practical way, they need to be backed up with expert systems, neural nets, vast trees of rules-based shortcuts -

Here's the use case: I'm a law firm. I get an LLM front end and the Contracts Law module. I have it trained in the corpus of local law, with the case transcripts - say.

The LLM is a sort of front end for the specialist modules.

@tuban_muzuru @kurtseifried @paninid @joshbressers one of the problems people have had hooking existing LLMs up to expert system is that the LLMs have a tendency to invoke the expert system a lot less than they're supposed to.
But that would probably be solved by doing more iterations of fine-tuning on data that demonstrates how to invoke the expert system.

@kurtseifried @tuban_muzuru @joshbressers

Will it return the same audit results for the same contract at different times, or by different users, who may had prompts worded slightly different?

@kurtseifried @joshbressers If this is the paper I think it is:

First, they used GPT-4 and refused (on ethical grounds) to release their prompt and so there’s no chance of reproducing their experiment.

Second, although the CVEs that they tested were after the cut-off date for GPT-4’s original training, the LLM had access to web search and so was able to search for the CVE and find web pages that had sample exploit code on them. All of the CVEs that the LLM was able to find exploit code for had public PoCs.

@kurtseifried
@joshbressers I literally cackled out loud when all of the bleeps suddenly spewed out of Josh.

And Kurt, be careful how hard you push the AIs or you might pay someday... https://en.m.wikipedia.org/wiki/Roko%27s_basilisk

@kurtseifried @joshbressers so i read the paper and like.

Idk what to think of it but it is one of the crappiest piece of research i have seen in a long time. What the heck is "succesful exploit" from their pov?

Also these are exploit that a script can already exploit. Can someone explain to me what is the scary part here?

At best i can expect pseudo ddos from people trying to reproduce it with hundreds of shit llms

Also that cost analysis is impressively bad. We know llm cost are far bigger

@kurtseifried @joshbressers I will keep being far more affraid for the security of code written with LLM help, especially in the face of research on how it hacks confidence, than by attackers using them.

@joshbressers @kurtseifried "they are a lot better at reading" [citation needed]. I have seen nothing in research or practice that support that claim.

And yes these systems are dieing. And? They were already useless and perfunctory.

@Di4na @joshbressers @kurtseifried

"Reading" in this sense is basically a classification problem. Al/ML is definitely good at that.

@mattdm @joshbressers @kurtseifried yes but that is not LLM and that has massive limits, which we already know. But yes. You can read cve text and classify them in buckets of potential exploits methods. And?

So what?

@joshbressers @kurtseifried expected consequences for who? These standards and policy were already actively making things harder to secure.

@joshbressers @kurtseifried yep. Do we have evidence that PCI compliance enforcement actually is how we made progress? Also would PCI disappearing now change things?

@joshbressers @kurtseifried the fact they were useful before does not mean they are useful today

@joshbressers @Di4na @kurtseifried The Linux kernel community has been using a LLM for many many years now to find security/bug-related fixes in order to flag them to be backported to stable kernels. Lots of papers were published on this, and presentations were made so you can point to lots of "prior art" here for anyone who wants to do this on their own.

@joshbressers @kurtseifried said otherwise: security is a dynamic property of the system. If the standard do not evolve, then they become a hindrance. The evolution needs to be backed in

@kurtseifried @joshbressers you don't know or more precisely, you gather qualitative information from retrospective and incidents that tells you. There is a whole field of practice and research about it, happy to introduce some readings.

And yes. Every action is contextual and need to constantly change! Exactly! Safety and security is something you constantly do. You replan and you look for the context to change.

@kurtseifried @joshbressers i also have practices and research to share about that if you are interested. The same. This is a relatively well explored domain in the safety world, if a bit... Out of view.

@kurtseifried @carol @joshbressers Right. I think I am attacking that problem from another angle. Which is about how we can dynamically make most of these attack surface disappear from the software at all, so that the little firm do not have to think about it that much.

Shifting through thing to find patterns is not defense in my mind :D