We need probably sooner or later better instructions for dealing with LLM assisted patches.

E.g., what to do if the fix itself is alright but commit messages have a continuous trend of being quite bad. I.e. you clearly can observe the patch being pure LLM dump but possibly still a legit fix for critical bug.

This quite defacto dilemma situation. At least for me. We need a more detailed guide on LLMs.

I don't this was the purpose of a patch being LLM assisted because there is zero involvement by the so called author. It's also a risk for kernel because author's own sanity check is our first layer of protection against malicious changes. I make mistakes too and it is now only up to me to explain the patch to myself and acking based on that, making me in a way both reviewer and the actual virtual author in a way.

#linux #kernel #ai #llm

@jarkko the solution is to do more work

@lkundrak i think i'll more proactive nacking and ask authors to do more work at least.

@jarkko Isn't that why you have skill/llm definition files that tell llms what to look for? You can tell them style (for chat bots I start with a 'be terse'.

@penguin42 I don't see connection to this. I'm not an LLM 🤷

@jarkko yes you do more. that's the spirit!

@jarkko Isn't the hope that the LLMs people are using will read the skill/llm definition files in the project so they output stuff in a style you're happier reading?

@penguin42 No. I don't care how your LLM is configured. I care what you do with your own brains and intuition and do you understand what you are doing with or without LLM.

@jarkko Yeh I understand that the authors engaging their brains and checking is important; but the other side is making sure explanations generated by LLMs are readable so you don't have to read through a pile of crap.

@penguin42 So I don't agree on that. If you get the change it takes you no time to make it right. What configuration gives better results is out of scope what I care when I see a patch. This is not a configuration problem.

@penguin42 It's an open problem as of now :-)

@penguin42 This will take time to resolve and there are no trivial answers. It's not a purely technical issue. It comes down minimum expectations from the authors and how to react properly if they are not met.

@jarkko I see 'AGENTS.md' files as different to configuration - in that they're in the control of the project not the (possibly lazy) AI user. Like Qemu's basically tells the AI not to write patches, and what to class as security issues; see: https://github.com/qemu/qemu/blob/master/AGENTS.md

As to replying to the people; well I don't see why that's different from someone sending a clueless patch without thinking it through or without following other guidelines.

@penguin42 This shifts to some other problem while the response and how our guidelines work was the actual problem.

This would ultimately require gathering feedback from maintainers and is not a scoped engineering problem.

@penguin42 My problem definition is not great either and is just a rant :-) Modelling the problem in all its complexity would be better angle based on feedback from multiple maintainers.

@jarkko Oh yeh it's hard - and changing rapidly. I thought that's also some of what https://github.com/masoncl/review-prompts was for - giving each maintainer a space to say what they want.

@penguin42 So AI is worse resolving the right response and would remove checks and balances completely. Not looking for AI solution for a problem that AI caused as a side-effect.

It's not unmanageable issue at this point but it is concerning and ultimately different situations might need different strategies. This is about people and comms not about AGENTS.md and AI skills.

@jarkko I'd thought of AI problems as split into two; ones that hit newbies, and ones that hit experienced users. The experienced people know to check, but also how to kick their AI into normally getting it right; I think the agents.md is what helps the newbies get part of that experience - but yeh it's no substitute for engaging your brain; and sure you have to tell people to fix stuff.

@penguin42 Also I made mistake saying that it is AI problem. It is part of the problem.

@penguin42 not disagreeing what you are saying tho but yeah I can admit that I cannot even at this point fully comprehend what I'm asking for or complaining about really, so there can be only wrong solutions :-) Sometimes it is better admit this and try to navigate for a while.

@penguin42 Off-topic but I get what you say about agents. I've also noticed that I do worse if I use latest GPT, Claude or similar top tier frontier models because they over do things. I do much better with Gemini Flash :-) The thing with those stronger models is that they don't get that what I ask is always within the scope and there is no figuring out to do for them and somehow that eager algorithm absolutely hates that situation :-)

@penguin42 With kernel, given the changes are not large in size, I don't use LLMs other than having a smarter search engine in a browser tab. It gives me 90% of its benefits in this context as gathering data for the decision making takes more time than editing the source code.

A recent example that I should probably resend: https://lore.kernel.org/keyrings/20260924055521.1981957-1-jarkko@kernel.org/

For patches like these I would use LLM to ask "truths" about code base, and when I have good understanding what is going on, I'll figure out the final solution myself.

I don't have any principle that would prevent me doing an AI assisted kernel patch but with the goal of getting the exactly right stuff in, I just have not had a situation yet where that would make sense. I'll do that when it does :-)

@jarkko Yeh they're pretty good for asking questions about source (well as long as you check them and it hasn't invented stuff); I also once did (a Libreoffice) patch where I needed to create a C++ subclass but then create all the virtual functions in it, and it was good for that.