#kr2026 @gregkh on LLMs.

The main point is: DON'T PANIC:!

... At the end 10 bugs fixed, 1hour kernel development...

@KernelRecipes @gregkh

or: keep calm and move along :)

@KernelRecipes @gregkh I love how the LLM stans freak out every time something happens with them. But then every time receipts come without, it's not a big deal.

@KernelRecipes @gregkh Well, we're panicking anyway. Linux is approaching 2000 CVEs per release, and it doesn't matter much how many of those are found by AI. It's made any sensible approach to vulnerability management completely infeasible. And no, "always patch every server all at once immediately" is still not sensible. Everyone I'm talking to is either giving up entirely or actively looking at alternatives to Linux.

@muvlon @KernelRecipes All software is having these same fixes, we are not unique here with Linux. Unless you use an old and unsupported operating system, sure, those don't get updates because no one is actually fixing anything!

@gregkh @KernelRecipes All software has security fixes, and a lot of it has CVEs, yes. But Linux has two orders of magnitude more CVEs than anything else we use. At some point, quantity becomes a quality all of its own. Specifically, the rate of CVEs puts an upper limit on the amount of human analysis that a given organization can invest into CVE triage. Once that dips below the amount of effort needed to even decide "do we run this code path at all?", you have little recourse but either patching everything unconditionally immediately (taking on a lot of operational risk) or blanket accepting security risks. For Linux, many organizations crossed that threshold last year.

And yes, Linux is a huge codebase, but that alone does not explain it. Chromium or Firefox are huge too.

@muvlon @KernelRecipes It has that many listed CVEs because our codebase is huge, but you only use a small fraction, so what is relevant for you is much smaller.
Also bugs at our level are CVEs, while userspace programs don't have those same issues.

This is nothing new, other operating systems have the same issues and lists of CVEs. It's just that the corporate OSes don't actually publish all of their vulnerabilities because nothing requires them to, so they don't. Go blame them and the cve.org people, not us!

@muvlon @KernelRecipes @gregkh On the bright side, it prompts distributions to prune their kernels to a more sensible subset.

@gregkh @KernelRecipes @muvlon Differences are that Linux has historically been exempted from some scrutiny (because Open Source earned trust by default) and simultaneously Open Source makes it easier to identify areas where exploits could be found. It's akin to "the first-mover disadvantage" or "incumbent's disadvantage". Now that it becomes easier to poke at vulnerabilities, the transparency that gave a head-start comes back to bite, temporarily

@KernelRecipes @gregkh minor ipv6 network issues wasn’t on my bingo card today but nice

@gregkh @KernelRecipes @muvlon we've partially addressed this confusion at ADI by generating CVE lists specific to our defconfigs:

https://analogdevicesinc.github.io/linux-security-vulns/#known-vulnerabilities

@philipmolloy @KernelRecipes @muvlon Awesome, that's the way to do it, nice work!

@philipmolloy @gregkh @KernelRecipes @muvlon On what basis do you match a CVE to a certain defconfig? Is there tooling available or is this custom scripting?

@philipmolloy @gregkh @KernelRecipes @muvlon this is great! Hello Qualcomm / Renesas / XYZ whatever / embedded distros (Buildroot, Yocto) I hope you're listening.

@heine @philipmolloy @KernelRecipes @muvlon Has everyone ignored the 'make sbom' kernel build option that will tell you EXACTLY which files your kernel build uses? You can then cross-reference that with the files in the CVE entry (both of which are in json formats, so script away!)

There's also many other older tools out there to get a simple "here are the files I am using for this kernel build" list, here's my very old version that has been used by Android in the past:
https://github.com/gregkh/gregkh-linux/blob/master/scripts/count_lines

@gregkh @heine @KernelRecipes @philipmolloy I definitely have. Not on purpose, this is just my first time learning about it. This looks neat, thanks for the hint!

@muvlon @gregkh @KernelRecipes @philipmolloy
Never heard of this before. To my defence, I see it’s not in 6.18 so it must be rather fresh 😅.

Should this work with a 6.18 kernel?

@heine @muvlon @KernelRecipes @philipmolloy you are right, `make sbom` is newer than 6.18, it showed up in the 7.2 release.