So we got @gregkh on the show to explain Linux Kernel security, both proactive and reactive, and why they sort of can't treat security bugs special (TL;DR: Linux is on everything, so a prenotification list to tell people secretly doesn't work when you tell thousands of people... and that's one of the easier problems), the whole #CVE thing and more on the #osspodcast with @joshbressers and @kurtseifried https://opensourcesecurity.io/2024/02/25/episode-417-linux-kernel-security-with-greg-k-h/ TL;DR: just run an up to date stable Kernel, the era of trying to cherry-pick and backport security fixes is coming to an end.

@kurtseifried @gregkh @joshbressers
Indeed. Get a stable kernel. Wait for others to test and leave your CVEs hanging out or update fast and beta test the stability. Tough choice indeed.

@kurtseifried @gregkh @joshbressers What a great episode! I can't believe that you managed to explain the kernel security release process in a way that I could understand. Off to go make sure I'm running a stable kernel!

@camdoncady @kurtseifried @gregkh thanks!

@camdoncady @gregkh @joshbressers to be fair Greg did all the heavy lifting.

@kurtseifried @gregkh @joshbressers This does increase pressure on the Kernel APIs to be more stable however. While "don't break user space" is a mantra, this will get tested much more in the future, systems running ages old glibc on a current stable kernel... . Also, let's see how proprietary kernel drivers do.

I'm all for it!

@divested what do you think about backporting of security patches? @kurtseifried @gregkh @joshbressers

@Kurt
Backporting patches like I do is truly awful, but it is the only feasible option for these old devices stuck on vendor kernels.
If you can run a kernel.org stable release, please do.

@kurtseifried @gregkh @joshbressers

@ljrk @kurtseifried @joshbressers It does not affect anything at all about in-kernel apis, there is no such stability rules or requirements, rather the opposite.

@gregkh @joshbressers @kurtseifried Oh, I’m aware of that! It’s just that I suspect a) companies doing proprietary or at least out-of-tree-drivers will exert more pressure to become in-kernel-stable, a pressure Linux must, and probably will resist against though; and b) the current user space API stability will be put to a more thorough test.

@ljrk @joshbressers @kurtseifried Based on the Android work I've seen happening, I doubt it. We have successfully run Pixel6 on every Linus -rc release for the past 3 years and that had 300+ out of tree drivers, not an issue. If you want to keep your code out of tree, you can, it just costs you money to do so, which companies have long understood for many decades now, it's their business decision. CVEs aren't going to change this at all, it does not affect the rate-of-change of upstream and stable releases at all.

@gregkh @joshbressers @kurtseifried While I’ve followed the Android work quite closely and that’s one of the reason I’m using a Pixel, Google is much different w.r.t. company culture than many other companies that I had in mind when writing this. Against all technical reasons they insist on staying out-of-tree. And I suspect that some of them will maybe use that development (i.e., working “against” back porting) to move in-tree too — but those who remain will do so more fiercely.

Do note, that this is nothing I defend — I basically want them to go in-tree-or-die :’D

@kurtseifried @gregkh @joshbressers thanks for this episode. I'm more and more under the impression, that Americans take the CRA much more seriously than Europeans.

@ljrk @joshbressers @kurtseifried The core Android portion of Google is very much NOT out-of-tree, they merge their code upstream almost always first before applying it to older Android kernel branches. See the yearly report at the Linux Plumbers conference for the past 5+ years where they what has been merged, what is left to be merged, and what is coming next, with pretty graphs and the like.
They have a strong "Get your changes upstream first" rule with their OEMs and you can see the interactions with those companies uptream has increased dramatically over the past years been because of this.

ChromeOS portion of Google also has a strong "upstream first" rule, and is very good about taking updates and getting changes merged properly.

The Pixel out-of-tree stuff is due to other "reasons", but that is a separate division and there have been many patch submissions recently upstream to resolve that delta, so that is being whittled down.

@gregkh @joshbressers @kurtseifried I feel like we're talking past each other. I never said that Android stuff would nowadays be mostly out of tree, nor ChromeOS etc. I don't understand why you bring that up?

I *love* your effort to move a lot of Android (and others) in-tree, I do *know* about the work spent there and recognize it. You keep telling me about it but *I know*.

It's about the *not-Googles*. The less visible out-of-tree stuff that often is bundled in pre-fab appliances and not consumer stuff. By small companies building OT hardware. German "Kleine-und-Mittelständische-Unternehmen" kind of companies. And these feel the pain (rightly so) for staying out-of-tree even more. And there's a choice: Follow the good path and integrate into the tree, or resist and stay out-of-tree – and probably pressure Linux unsuccessfully to become more in-kernel-stable with futile mailing list threads.

That's what I was talking about. I really appreciate you chiming in but I don't feel it makes any sense repeating anymore at this point.

@kurtseifried@infosec.exchange this talk was mentioned by @gregkh@social.kernel.org during the show, but it’s missing from the show notes. https://youtu.be/Kt3hpvMZv8o

@kurtseifried @gregkh @joshbressers I agree with everything said in that episode. Especially the bit about testing before upgrading.

The episode reminded me of a blog post I made a while ago:

https://blog.liw.fi/posts/2022/09/14/not-breaking/

(Shameless self-promotion, sorry.)

@gregkh @ljrk If I remember correctly there is a huge difference between 2014 and 2024, where Android Linuxes like in Android 2, 3 and 4 had not only drivers out of tree, but their own subsystems and features, some of which had no chance of going upstream, because mainline had similar but different mechanisms.

Some time between then and now some people started working hard on upstreaming what could be upstreamed and leaving behind things that couldn't, so that Android today can run on a mainline Linux, and anything out of tree is device-specific stuff.

Is that a somewhat accurate overview?

@kurtseifried @joshbressers

@clacke @joshbressers @kurtseifried @ljrk Yes, that is a good overview, and you can point at the managers and developers on the Google Android team for doing all of this hard work both in convincing people that this was a good idea, and actually doing the work to get stuff merged properly. I've watched them do this for the past 10+ years, it wasn't easy but has paid off a lot in my opinion, the security level of an Android device is worlds better than it used to be because of this effort.

@liw @kurtseifried @gregkh shameless self promotion is the best kind :)

@liw @kurtseifried @gregkh

I gave this a read, and my first thought was:

Do successful software projects take on the stance of "don't break your users", or does "don't break your users" create successful software projects?

It's easy to say it's probably somewhere in the middle, but I bet it heavily skews to one side.

I'm not sure if Linux always had the "don't break the users" mantra but it's clear that if they didn't have it, Linux wouldn't be nearly as successful as it is

@joshbressers @kurtseifried @gregkh The answer is always that it's not as simple as you'd want.

@joshbressers @liw @kurtseifried @gregkh in the case of curl and libcurl, we took on the stance to never break our users very early on. I am convinced it is a reason behind our success. They know they can upgrade without worries.

@bagder @joshbressers @liw @gregkh I have a suspicion: if software breaks on update and the user is faced with a choice of fixing it, or moving to something else, I bet if they move to something else they never move back. So curl wouldn't bleed users out, and would gain users basically permanently. I just jumped through an annoying hoopwith Python/libcurl, but you have CURL_CA_BUNDLE_AUTODETECT so... it wasn't that bad.

@bagder @joshbressers @liw @gregkh Oh one comment: e.g. (EDIT: libcurl was mentioned, I meant pycurl but typed the wrong thing): pycurl :

(77, 'error setting certificate verify locations:\n CAfile: /etc/pki/tls/certs/ca-bundle.crt\n CApath: none')

maybe mention "have you considered using CURL_CA_BUNDLE_AUTODETECT"

because there's about 5x people asking why it's broken than mention of the solution in Google:

@bagder @joshbressers @liw @gregkh And pycurl doesn't appear to support CURL_CA_BUNDLE_AUTODETECT so for posterity:

self.ca_bundle_path = self.find_ca_bundle([
'/etc/ssl/certs/ca-certificates.crt',
'/etc/pki/tls/certs/ca-bundle.crt',
'/etc/ssl/ca-bundle.pem',
'/etc/pki/tls/cacert.pem',
'/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem',
'/etc/ssl/cert.pem',
])
if not self.ca_bundle_path:
sys.exit("No CA bundle found. SSL certificate verification might fail. Exiting.")
self.content = None # To store the downloaded content

@bagder @joshbressers @liw @gregkh for history: https://github.com/pycurl/pycurl/issues/835