"[…] It is clear that the current process is based on the learnings, and frustrations, the [#Linux #kernel's CVE] team has faced in the past. […] By taking this position, this effort is now duplicated across thousands of engineering teams ad infinitum, […]"

Well, yeah, but guess what: maybe then the companies behind those engineering teams will join up and invest money to handle the problem "[…] at the source, in a central, efficient and reliable manner. […]". 😬

https://amanitasecurity.com/posts/dear-linux-kernel-cna-what-have-you-done/

@kernellogger I think it's clear at this point that this isn't an unintended side effect.
It is intentionally aimed to bring the system down through overload, probably in the hope that something better will emerge later.
At best, it is "constructive destruction" - at worst, it is malicious towards the users and may spectacularly backfire on the trust towards Linux.

@larsmb

Well, you have a point, but the strategy "ignore CVE" apparently became more and more of a problem as well that at some point might "backfire on the trust towards Linux." So maybe escalating the problem rather sooner than later might in the end be the better of two bad choices. Not sure.

@larsmb @kernellogger I don't belive @gregkh is acting in good faith here :-(. https://lwn.net/Articles/961978/ makes it quite clear. How can we stop him?

@pavel @kernellogger @larsmb @gregkh I think it's important to realize that nothing has really changed in how patches are merged into either mainline or stable. The only difference here is that more commits are now designated as (fixing) CVEs. On the whole, that's probably a good thing; false positives are probably better for security than false negatives. This creates more work for distros, but arguably that's work they should be doing anyway. If we can collaborate more, that's even better.

@vegard @pavel @kernellogger @gregkh It's still an intentional attack on the CVE system. That's perhaps a fair aim, but it can't be denied it also causes strive, confusion, and pain for some.

@vegard @kernellogger @larsmb @gregkh Dealing with spammer should not be part of distro's work. And no, "false positives are better" is not true in this case.

@larsmb @vegard @kernellogger @gregkh If there was an SUSE statement explaining "intentional attack on the CVE system is not cool", I could use it :-).

@pavel @kernellogger @vegard @gregkh I'm just posting here as a private person, nobody trusts me enough to speak on behalf of my employer 🙂

@larsmb @kernellogger @vegard @gregkh If there's some good writeup (you, your company, someone else), it would be useful to me. I seen this so far: https://amanitasecurity.com/posts/dear-linux-kernel-cna-what-have-you-done/ .

@larsmb @vegard @pavel @kernellogger I find it hard to understand how this is an "attack", when it was specifically authorized and endorsed by the CVE employees and board of directors? They knew exactly how many CVEs we would be filing on a weekly basis and how we would be evaluating them for inclusion. We went through many meetings and discussions before our application was accepted and our id submissions were allowed.

They also asked us to backfill the database with our previously submitted GSD information, as well as everything that gets merged forward. That backfill work is ongoing and is the large volume of what is currently being assigned right now. That information has already been out there for years in GSD, putting it in CVE doesn't change the actual kernel commits or "security" at all, it's just that you might not have noticed the GSD entries (hint, the "bad guys" sure did...)

The number of "new" CVE entries is a small % overall as we really have only processed the changes from v6.7.0..v6.7.3 or so. We are behind in the new stuff and will catch up soon there, help is always appreciated, if you see a commit you want assigned a CVE, just ask!

As others have said, this hasn't changed how the kernel commits are merged at all, it's just calling stuff out that everyone should have already been looking at and merging already, with way more meta-data and information than has ever been done before for kernel CVEs, thereby actually raising the information level in the CVE database here (which is probably why the CVE board wanted this.)

If companies are somehow dictating that they MUST look at and evaluate all CVEs, then they should be happy as many groups were intentionally abusing the CVE process for the kernel previously, making their life much more difficult (including the community's life, which is why we became a CNA.) That abuse has now stopped, to be replaced with a different workflow with way more meta-data produced to help make decisions easier. So far one of the biggest complainers was the company that was doing that abuse as they are no longer allowed to do that (to be specific, that was NOT SuSE).

@kernellogger you know being part of the kernel community involves actually PUSHING BACK sometimes instead of just repeating the 'company line'.

So far it seems you are simply repeating Greg's points and ignoring the (copious) push back on this.

Quite credibly this is one giant troll designed to essentially attack the whole broken mess of CVEs, many people have pointed out how this is an issue.

It strikes me as quite naive to think that companies should now funnel cash + resources into filtering a ton of 'security' issues that are very obviously not, on top of the already questionable stable practices.

@gregkh @larsmb @kernellogger @pavel @vegard TL;DR: If I ignore enterprise linux all is fine?

I've seen lots of comments like 'nobody uses x.y.z' stated as if <= x.y.z isn't in continuous use in REALITY rather than in a perceived stable linux utopia...

Isn't this just replacing 'why do enterprise linux providers complain about stable commits breaking things' with 'why do enterprise linux providers compalin about arbitrary CVE assignments breaking things'?

There are regulatory and customer demands that dictate CVE take up, yes it's all a mess, but you seem to keep on talking as if this doesn't exist despite multiple people telling you it does?

Am I missing something?

(am happy to stand corrected if I, indeed, am)

@pavel @vegard @gregkh @kernellogger @larsmb the issue for me as an outsider to this (currently not employed within the kernel, though contributing to it a fair bit) is the utter denial that there are consequences like that coming from the pro- side despite the opposing side saying 'yes maybe we shouldn't care but customers care, regulators care, etc. etc' and the other side >/dev/null.

Having a conversation starts with acking or refuting what the other side has to say, not acting like it wasn't said.

Again, happy to stand corrected if I'm wrong here! Just my perspective.

@ljs @kernellogger @larsmb @gregkh @pavel It's really complicated... I'm myself on the distro side here (though speaking only for myself) and I see very clearly the additional work that this is causing. On the other hand... I do think this is actually moving things in the right direction, security-wise. The uncomfortable truth is that the kernel has a TON of bugs, many with security impact. This move really just puts it completely out in the open and forces everybody to acknowledge that fact.

@gregkh @larsmb @kernellogger @vegard What "abuse" are you talking about? There was small amount of questionable CVEs. Now there's huge amount of... commit metadata pasted into CVEs. That's even lower quality than before, and as a bonus there's order of magnitude more of them. It should be pretty plain to see why people view that as an attack. Example: https://nvd.nist.gov/vuln/detail/CVE-2023-52472 . Does the description make sense to you?

@vegard @ljs @kernellogger @larsmb @gregkh It clearly is additional work, true. But the goal is security, and nonsensical CVEs do not help with that. https://nvd.nist.gov/vuln/detail/CVE-2023-52472 . What percentage of spam CVEs do you believe describe real security problem?

@vegard @kernellogger @larsmb @gregkh @pavel yes, I've generally not commented on this as a. I'm not involved with an enterprise kernel at this point and b. the complexities of the issue, but from my perspective it's the lack of acking how the _reality_ of how kernels are used by the companies which fund a huge amount of core dev.

I mean even if you think the way it's done is awful there should be some acknowledgement of that fact, especially when people are explicitly saying 'dude we are in a position where we _have_ to filter through this stuff'.

And I think the whole 'well there's a ton of bugs who knows which could be a security flaw' is dubious at best.

Some bugs are very clearly more problematic or have more clearly been shown to be security flaws than others.

Of course all this speaks to how incredibly crap the whole CVE system is as a whole, but I'm just not sure going nuclear is the way forward.

I think the kernel taking control of the CNA side is _good_, the spamming aspect, seems not so good.

@ljs @kernellogger @larsmb @gregkh @pavel Agreed that the dialogue hasn't been great. However...

I don't think they are actually "going nuclear" here. There are 3 volunteers going through all of the stable commits for things that look potentially security-relevant, and they have a vote. It's not like they are literally flagging every single commit. The CVE volume has gone up, clearly, but I think it's justified.

@vegard @kernellogger @larsmb @gregkh @pavel I am pretty sure there are scripts involved here, look at Pavel's example.

EDIT: Also on the 'go nuclear' point, the original email literally states 'the CVE assignment team is overly cautious and assign CVE numbers to any bugfix that they identify. This explains the seemingly large number of CVEs that are issued by the Linux kernel team."

That very much speaks to going nuclear, every single 'bug fix' (however you want to define that, autosel for instance blurs lines) is likely to be quite a few. And given the use of scripts in stable process highly likely to be at least partially automated.

@ljs @kernellogger @larsmb @gregkh @pavel I posted that one myself a few days ago 🙂 And I disagree with that one having a CVE on account of the whole "small allocations can't fail" thing being so widely accepted to be true in basically all configurations. Greg's response was here, BTW: https://lore.kernel.org/all/2024022654-designer-rack-c644@gregkh/

Looking over the list of recent CVEs, most of these are probably local DOS in various obscure drivers. But they do arguably fall within the definition of a vulnerability according to CVE.

@ljs @kernellogger @larsmb @gregkh @pavel If we really need to know the security impact of each and every patch (which most distros probably do), then somebody has to do that analysis. Yes, it's going to be expensive in terms of manpower and effort. But it still needs to be done. That's why I'm hoping this will lead to more distros collaborating on this and the community effectively only doing it once, in one place.

@vegard @ljs @kernellogger @larsmb @pavel "security impact" means you have to take into account your specific use case, and we have no idea what your use case is Remember, Linux is in cow milking machines, servers, helicopters on Mars, phones, utility meters, watches, mega-super-yacht-stabilizers, wind turbines, printers, cars, planes, trains, and more. Doing an accurate "security impact" is going to be different for all of those cases.

To quote Ben Hawks, "It's hard to capture the fact that a bug can be super serious in one type of deployment, somewhat important in another, or no big deal at all -- and that the bug can be all of this at the same time. Vulnerability remediation is hard."

His full post is worth reading: https://blog.isosceles.com/what-is-a-good-linux-kernel-bug/

@vegard @kernellogger @larsmb @gregkh @pavel this strikes me as a classic mistake people make with probabilistic things.

That anything _could_ be true doesn't mean that everything is _equally likely_ to be true.

There are heuristics that can be used to determine whether something is more/less likely to be a problem, and spamming every single possibility to something that's clearly _intended_ to be more filtered than that is, again, a nuclear option.

@gregkh @kernellogger @larsmb @ljs @pavel Yes, but that's just for the impact. A lot of the analysis will be the same for all distros: Is the code reachable from userspace? Does it require any CAP_*? Is it reachable from user namespaces? Do you need physical access to insert a USB device? etc. These things take time to figure out and will be common for basically everybody. I'm not even talking about the stuff like "what two structs do you need to overlay to manipulate that specific member"

@gregkh @vegard @kernellogger @larsmb @pavel this is what I find weak in this argument is the idea that there is _no_ way to assess the severity.

That's patently untrue. Some bugs are clearly more likely to be more of a risk than others and it's sensible to apply a heuristic.

If you apply no heuristic and just assume that any can be, why stop there? Because you know we don't always identify bugs right, so any commit could be a bug right?

Obviously this pokes holes in the whole CVE thing, but again going nuclear and pretending like no heuristics or filters can be applied gets you to reductio ad absurdum really fast no?

@gregkh @kernellogger @larsmb @vegard @ljs @pavel I'd hazard a generalization and say the 'impact' (any impact) is impossible to know if you consider all combination of systems and workloads. It is only good under the conditions it was tested under. Ie: it works for me 😉 And to be honest, not sure we can do better. Can we even define what full test coverage means? We just need more people to say it works for me 🙃

@vegard @ljs @kernellogger @larsmb @gregkh @pavel note Greg's response argues for taking the fix, but that's not the point. The purpose of a CVE does not mean "why not take the fix?" If the motivation to create an own CNA was some company filling kernel CVEs just so they can justify backporting fixes internally, then this is a rather bizarre twist.

@ljs

don't worry, I will speak up when I see the need; but reg. the CVE stuff and a lot of the critique wrt to how the stable team operates I seem to be somewhat in line with Greg's approach *or* think the problems need to be fixed elsewhere to improve things (the latter is on my todo list).

And 'company line'? Come on, this is the kernel community, there is no company line, just individuals with a lot of different options 😇

/me runs 😬

@ljs

And wrt to "It strikes me as quite naive [...]": maybe, but it's a bit like better docs, regression tracking and some other unattractive work in kernel land: a lot of people want those things to be better, but its hard to find the money to pay people to do all that work.

@kernellogger @ljs Two points here: 1) Yes, the bad guys are already doing this (sifting through changelogs and finding exploitable bugs), we should be doing it too, and 2) various companies and distros probably are probably already doing some parts of the work but it's done behind closed doors and we have no good mechanism for exchanging notes or collaborating at the scale necessary, so it ends up as duplicated for no good reason. I really do see an opportunity here for everybody to win

@vegard @kernellogger @ljs

I did actually make an attempt at gathering the Linux distro security community into an IRC channel a couple of years back to try and help each other.

Arguably the channel has some occasional chatter but could do more with it.

@Foxboron @kernellogger @ljs ...pointer? 😅

@kernellogger I mean you're welcome to your opinions obviously and I'm not trying to be mean it just feels like you speak on these things as if people haven't pushed back.

Obviously you're under no obligation to say or not say anything :P but it seems like, in these posts, you seem to be doing the same as Greg and just basically completely ignoring everything people are saying about the _real world_ consequences of this.

I sort of almost admire the going nuclear option but the issue is (and why I spoke out on it) that it's making kernel engineers suffer for basically no reason.

Also let's not pretend we just 'can't assess' kernel bugs at all. By the same token, how can you be sure a commit doesn't introduce a bug? Why not CVE all commits?

It's reductio ad absurdum in an effort to highlight how stupid the CVE system is imo.

@vegard @kernellogger 1. There seems to be no filtering whatsoever on CVE side, spamming isn't an answer 2. Spamming is not going to improve that.

Actually I think this whole effort will dramatically backfire as it's quite a hostile way of doing things.

@Foxboron @vegard @kernellogger I mean collaboration is the way to go to improve things (great effort from your side to try to push for that!), I think dropping a nuclear bomb on people and denying you dropped it is probably less conducive to that 🤪

@ljs @kernellogger @vegard

I have mostly given up on CVE as a thing as companies misuse it, security researchers pad their CVs with it, the quality of the CVE entries has gotten terrible with the CNA liberation and the signal-to-noise ratio isn't worth it unless you have the tools to support the work.

I was hoping the OpenSSF would fix parts of it, but it has just become another front for companies to FOSS-wash their proprietary things.

@Foxboron @kernellogger @vegard oh don't get me wrong CVE is a giant malodorous pile of excrement as far as I can tell.

I mean it is horrific and needs reworking I think most people feel this way.

I guess security is a really hard one to get right on this because a lot _could_ be a problem but only a few _are_.

Feels like we need some open security foundation or something

@Foxboron @kernellogger @vegard As I've said elsewhere it's pretty obvious you can filter what's likely or not likely to be a security issue (yes you won't always be right), otherwise you're stuck with infinite regression of every single change being a potential problem.

If the black hats are assessing for vulnerabilities they will be looking for certain things, as can we :>)

@ljs @kernellogger @vegard

This assumes you have the capability to filter and analyze. This also assumes you have the manpower.

This is not true for volunteer driven distros.

@vegard @kernellogger @ljs

I couldn't find the original draft, so alpine-devel will do.

https://lists.alpinelinux.org/~alpine/devel/%3C20190301214806.47a05e54%40ncopa-desktop.copa.dup.pw%3E

@Foxboron @kernellogger @vegard right but the answer isn't to mark every bug a security flaw. It kind of speaks to taking a different approach.

@Foxboron @vegard @kernellogger btw love arch btw ;)

@ljs @kernellogger @vegard

Thanks :)

@ljs @kernellogger @vegard

Sure, but I'm thinking a bit broader than the Linux stuff. It's still a small amount of the overall CVE assignments so far.

@Foxboron @kernellogger @vegard it's good that you opened a dialogue there. The issue of course in open source is getting people to actually agree on anything :P

Which is maybe why Greg has taken a more... forthright approach.

I'm wondering whether a distro or some security people just need to start something even if others don't collab and then hopefully if a good solution that gets picked up more generally.

@gregkh @vegard @kernellogger @larsmb @ljs Yep, cow milking machines are clearly reason Greg is allowed to spam CVE database with nonsense copy/pasted from changelogs. Not. There are people who know what "security impact" is, and are acting in good faith, Greg is just not one of them. He complains about "CVE system abuse" then abuses that system 10x more.

@vegard @ljs @kernellogger @larsmb @gregkh Security was quite fine before Greg starting abusing the process. Redirecting manpower from "finding real security issues" to "dealing with Greg's spam" is not an improvement.

@pavel @kernellogger @larsmb @vegard @gregkh @ljs

before surely suse had an engineer evaluate every stable commit for security and back port as part of the support promise. Now gregkh does you a service by doing the first scrub of that and suse only has to evaluate the ones he flags which is a clear subset?

or did you mean to imply that enterprise Linux distros did not do such thing and now you need to?

@pavel @kernellogger @vegard @gregkh @ljs This is getting a bit too personal, could you kindly drop me from the mentions 🙂

@vegard @pavel @kernellogger @larsmb @gregkh Having some experience in maintaining a distro kernel (abet a small distro), it's going to depend on a number of factors, primarily how closely you track a stable kernel tree.

If your regularly tracking a stable kernel tree and don't have a tonne of out-of-tree patches, then the effort should be minimal to at least integrate new stable tree versions, which are likely to be carrying the latest CVE fixes.

@vegard @pavel @kernellogger @larsmb @gregkh

If your trying to support for longer periods than the stable kernel trees run for, then you're probably getting paid to provide that support. If you're trying to "only apply patches that are definitely serious issues" then your probably getting paid for that and arguably should be getting paid more for that than just longer term support.

The kernel providing supported stable trees for free is above what a lot of other open source projects provide.

@MWelchUK @pavel @kernellogger @larsmb @gregkh I think the current headache for distros with this has less to do with patch selection/integration and more to do with dealing with vulnerability scanners, support requests, and escalations.

There's unfortunately a whole ecosystem out there driven by CVEs and the time spent dealing with all of that is proportional to the number of CVEs, regardless of (perceived or actual) severity.

@vegard @pavel @kernellogger @larsmb @gregkh That's a fair point. Though in my experience, there are two types of distribution, those where you pay for support and those which are freely distributed without any guarantees. If your providing paid support, this is what you're paid for, it may now be a bit more expensive. If you get it free, you get to do the work yourself if you want those guarantees.

Maybe those who've been relying on CVEs being exhaustive list of vulnerabilities have been lax.

@vegard @MWelchUK @pavel @kernellogger @gregkh exactly, the "it's a subset" part doesn't help that much wrt the already existing evaluation, but there's more effort once evaluated as not applicable/low score because of the magical CVE stamp and its implications. Would be wonderful if this flood resulted in the magic being lost, but not sure I'm so optimistic as many people are invested in the magic.

@ljs @kernellogger @larsmb @vegard @gregkh @pavel as a system, CVE is not concerned about *probability*. It is only concerned about being a common identifier for a software defect where a security impact is *possible*.

CVSS (as practiced) is similarly unconcerned about probability, it is mainly concerned about capturing characteristics (usually static) of a known *possible* vulnerability.

@msw @kernellogger @vegard @gregkh @pavel ok so explain why you shouldn't make every single commit a CVE?

@ljs @kernellogger @larsmb @vegard @msw @gregkh @pavel obviously not every commit needs to be a CVE. Some commits qualify for CVE allocation under the current CVE system design, and some do not.

@msw @kernellogger @vegard @gregkh @pavel 'obviously'? Why not? You just said probability isn't a factor?

You can't possibly know about all the bugs. If probability isn't a factor then all commits should be CVEs minus perhaps doc/comment-only ones (even then, are you 100% sure about compiler bugs?)

Very plainly the theoretical arguments being applied here are not applicable to the real world.

Also of course this puts aside the point of whether taking a very literal interpretation of what does/does not qualify + adding a ton of downstream effects is a good way of highlighting the flaws in the CVE system.

@msw @gregkh @kernellogger @pavel @vegard The issue with this discussion (in my view) is that one side keeps on repeating points based on a theoretical/principle basis, the other side keeps on pointing out practical outcomes and both talk past each other.

@ljs @kernellogger @vegard @msw @gregkh @pavel dropping Lars from mentions…

While I’m talking a bunch of theory, I am very tuned into the situation in _practice_. CVE as designed has always been different than CVE in practice, same as CVSS as designed (which is FAR more fatally flawed than CVE as designed) is different than CVSS as practiced.

@msw @kernellogger @vegard @gregkh @pavel The issue for me is the wording of the original mail about this essentially saying 'we are very cautious and therefore all bugs must be considered potential security flaws'.

In practice there surely must be some form of filtering. Previously yes there was abuse of the system (and there seems perpetually to be that), but it doesn't seem like previous designation led to particularly bad outcomes.

It's clear CVE as a whole is a mess in _practice_ with people pushing for designation purely for resume reasons/to sell something/misplaced enthusiasm etc.

The issue for me is applying some kind of practical filtering while taking into account down stream impact.

The issue I think (as mentioned in some previous waffle of mine) is that Greg seems to act as if enterprise linux simply does not exist, and that outside pressure to account for CVEs equally does not exist.

But indeed both do exist, and while it is rather asinine that this outside (regulatory, customer, etc.) pressure insists on using such a flawed system (+ of course this should be challenged) - is spamming the right way of doing that?

An aspect in Greg's favour is that - in open source with many competing companies who don't wish to allocate resource to a better solution - does it take bringing the hammer down like this to force a better solution?

I would say this is not a great approach however and quite likely to backfire.

@msw @gregkh @kernellogger @pavel @vegard It's also kind of hard to get away from what I and others think this is really about i.e. the "burn them down!" option, as touted (+ then rejected) by Greg in 2019 - https://youtu.be/dvtZsBlSECk?t=9299 (though a lighter form of what was proposed then i.e. CVE for every commit).

I think it's interesting that the public arguments provided do lead you to question whether there shouldn't just be a CVE per commit which aligns totally with this proposal...

@ljs @kernellogger @vegard @msw @gregkh @pavel by chopping wood and carrying water for this community effort, you earn trust with others and gain influence over system evolutions.

Same as an open source project.

Except for more board structure and government / defense sector contractor involvement…

@MWelchUK @vegard @kernellogger @gregkh People are paying for secure enterprise distribution. And if people doing security have to deal with spammer "trying to burn them down", the result will be less secure.

@pavel @kernellogger @vegard @gregkh I think your making the bold assumption that this only impacts enterprise distros.

Whilst this may historically have been true, there are a tonne of embedded systems out there of varying types, that have historically done a very poor job of security updates, but are coming under increasing scrutiny and regulation.

These systems have differing threat models, hardware and as a result there's going to be more CVEs that don't apply to typical enterprise systems

@fenruspdx @kernellogger @vegard @gregkh @ljs It would be a service if it was done in good faith. It is not, as clearly stated by Greg. SUSE (and others) now have to explain to their customers that they are not fixing 300 CVEs because they are not real CVEs but spam.