@kurtseifried @gregkh @joshbressers
Indeed. Get a stable kernel. Wait for others to test and leave your CVEs hanging out or update fast and beta test the stability. Tough choice indeed.

@kurtseifried @gregkh @joshbressers What a great episode! I can't believe that you managed to explain the kernel security release process in a way that I could understand. Off to go make sure I'm running a stable kernel!

@camdoncady @kurtseifried @gregkh thanks!

@kurtseifried @gregkh @joshbressers This does increase pressure on the Kernel APIs to be more stable however. While "don't break user space" is a mantra, this will get tested much more in the future, systems running ages old glibc on a current stable kernel... . Also, let's see how proprietary kernel drivers do.

I'm all for it!

@divested what do you think about backporting of security patches? @kurtseifried @gregkh @joshbressers

@Kurt
Backporting patches like I do is truly awful, but it is the only feasible option for these old devices stuck on vendor kernels.
If you can run a kernel.org stable release, please do.

@kurtseifried @gregkh @joshbressers

@ljrk @kurtseifried @joshbressers It does not affect anything at all about in-kernel apis, there is no such stability rules or requirements, rather the opposite.

@gregkh @joshbressers @kurtseifried Oh, I’m aware of that! It’s just that I suspect a) companies doing proprietary or at least out-of-tree-drivers will exert more pressure to become in-kernel-stable, a pressure Linux must, and probably will resist against though; and b) the current user space API stability will be put to a more thorough test.

@ljrk @joshbressers @kurtseifried Based on the Android work I've seen happening, I doubt it. We have successfully run Pixel6 on every Linus -rc release for the past 3 years and that had 300+ out of tree drivers, not an issue. If you want to keep your code out of tree, you can, it just costs you money to do so, which companies have long understood for many decades now, it's their business decision. CVEs aren't going to change this at all, it does not affect the rate-of-change of upstream and stable releases at all.

@gregkh @joshbressers @kurtseifried While I’ve followed the Android work quite closely and that’s one of the reason I’m using a Pixel, Google is much different w.r.t. company culture than many other companies that I had in mind when writing this. Against all technical reasons they insist on staying out-of-tree. And I suspect that some of them will maybe use that development (i.e., working “against” back porting) to move in-tree too — but those who remain will do so more fiercely.

Do note, that this is nothing I defend — I basically want them to go in-tree-or-die :’D

@kurtseifried @gregkh @joshbressers thanks for this episode. I'm more and more under the impression, that Americans take the CRA much more seriously than Europeans.

@ljrk @joshbressers @kurtseifried The core Android portion of Google is very much NOT out-of-tree, they merge their code upstream almost always first before applying it to older Android kernel branches. See the yearly report at the Linux Plumbers conference for the past 5+ years where they what has been merged, what is left to be merged, and what is coming next, with pretty graphs and the like.
They have a strong "Get your changes upstream first" rule with their OEMs and you can see the interactions with those companies uptream has increased dramatically over the past years been because of this.

ChromeOS portion of Google also has a strong "upstream first" rule, and is very good about taking updates and getting changes merged properly.

The Pixel out-of-tree stuff is due to other "reasons", but that is a separate division and there have been many patch submissions recently upstream to resolve that delta, so that is being whittled down.

@gregkh @joshbressers @kurtseifried I feel like we're talking past each other. I never said that Android stuff would nowadays be mostly out of tree, nor ChromeOS etc. I don't understand why you bring that up?

I *love* your effort to move a lot of Android (and others) in-tree, I do *know* about the work spent there and recognize it. You keep telling me about it but *I know*.

It's about the *not-Googles*. The less visible out-of-tree stuff that often is bundled in pre-fab appliances and not consumer stuff. By small companies building OT hardware. German "Kleine-und-Mittelständische-Unternehmen" kind of companies. And these feel the pain (rightly so) for staying out-of-tree even more. And there's a choice: Follow the good path and integrate into the tree, or resist and stay out-of-tree – and probably pressure Linux unsuccessfully to become more in-kernel-stable with futile mailing list threads.

That's what I was talking about. I really appreciate you chiming in but I don't feel it makes any sense repeating anymore at this point.

@kurtseifried@infosec.exchange this talk was mentioned by @gregkh@social.kernel.org during the show, but it’s missing from the show notes. https://youtu.be/Kt3hpvMZv8o

@kurtseifried @gregkh @joshbressers I agree with everything said in that episode. Especially the bit about testing before upgrading.

The episode reminded me of a blog post I made a while ago:

https://blog.liw.fi/posts/2022/09/14/not-breaking/

(Shameless self-promotion, sorry.)

@clacke @joshbressers @kurtseifried @ljrk Yes, that is a good overview, and you can point at the managers and developers on the Google Android team for doing all of this hard work both in convincing people that this was a good idea, and actually doing the work to get stuff merged properly. I've watched them do this for the past 10+ years, it wasn't easy but has paid off a lot in my opinion, the security level of an Android device is worlds better than it used to be because of this effort.

@liw @kurtseifried @gregkh shameless self promotion is the best kind :)

@liw @kurtseifried @gregkh

I gave this a read, and my first thought was:

Do successful software projects take on the stance of "don't break your users", or does "don't break your users" create successful software projects?

It's easy to say it's probably somewhere in the middle, but I bet it heavily skews to one side.

I'm not sure if Linux always had the "don't break the users" mantra but it's clear that if they didn't have it, Linux wouldn't be nearly as successful as it is

@joshbressers @kurtseifried @gregkh The answer is always that it's not as simple as you'd want.

@joshbressers @liw @kurtseifried @gregkh in the case of curl and libcurl, we took on the stance to never break our users very early on. I am convinced it is a reason behind our success. They know they can upgrade without worries.