Check out @gregkh 's Mentorship Session: Demystifying the #Linux Kernel Security Process

https://youtu.be/sLX1ehWjIcw?si=obRx2vcNuZSjieWY

So I presented today on EU CRA, NIS2 and other initiatives to regulate code/hardware/services. One consistent piece of feedback I got is that the amount of upcoming regulation is so huge that even dedicated professionals are unable to keep track of it all. So it is not just me (or you). It is _a lot_. https://berthub.eu/one/EU%20and%20you.pdf

How to fix the TTY layer in Linux?
@gregkh explains at #er2023
https://git.sr.ht/~gregkh/presentation-tty/tree/9afefcf3171a02e5eee19639ec62456511902a56/item/tty.pdf

Slides and recording of @gregkh's great @KernelRecipes talk "#Linux #Kernel security demistified" are now online:

Slides: https://git.sr.ht/~gregkh/presentation-security/blob/3547183843399d693c35b502cf4a313e256d0dd8/security-stuff.pdf

Video: https://www.youtube.com/watch?v=xDHTn0auo2w&t=14975s (that's a section of the live stream) #LinuxKernel

Slides for my @KernelRecipes talk from yesterday for "Linux Kernel security demistified" can be found here: https://git.sr.ht/~gregkh/presentation-security and the video will be probably online sometime "soon" for those that missed the live stream.

@davem_dokebi our godfather on stage for a sump up of netconf 2023 #kr2023

A sign of DisplayPort over USB-C (external display) working with the #mainline kernel on the recently launched #Fairphone 5!

This wouldn't have been possible without the great work done by all the people contributing to Linux and this SoC, especially the amazing people at Linaro!

#postmarketOS #Fairphone5 #LinuxMobile #kr2023 #KernelRecipes

@gregkh on stage: Demystifying the Linux kernel security process #kr2023 - serious things coming...

So OSS Europe was an interesting experience, this year, in a way.

I did my usual talk, and started with the usual section on kernel releases. When talking about stable updates I tossed in a quick mention that six-year support from the stable team was being phased out — something I understood to be generally known for about the last year. Way at the end of the talk, as my last topic, I discussed at some length the stresses being felt by kernel maintainers.

@sjvn wrote an article about the talk (https://www.zdnet.com/article/long-term-support-for-linux-kernel-to-be-cut-as-maintainence-remains-under-strain/) and made a connection between the stable-policy change and the maintainer issue — something I had not done in the talk. It was a bit of a shift from what I said, but not a bad article overall.

Then the rest of the net filled up with other writers putting up articles that were clearly just cribbed from SJVN's piece — sometimes with credit, sometimes without. I'm getting emails about what a terrible idea this all is, as if I had anything to do with that decision or can somehow change it. I have, it seems, taken away everybody's six-year support, and they're not happy about it.

All because of a 30-second mention of a change that was made public something like a year ago. My 1.5 minutes of fame has given me a new appreciation for this old quote from Rusty Russell: "when a respected information source covers something where you have on-the-ground experience, the result is often to make you wonder how much fecal matter you've swallowed in areas outside your own expertise."

Here is a hopefully-useful notice about Linux kernel security issues, as it seems like this knowledge isn't distributed very widely based on the number of emails I get on a weekly basis:

- The kernel security team does not have any "early notice"
announcement list for security fixes for anyone, as that would only
make things more insecure for everyone.

- The kernel community does not assign CVEs, nor do we deal with them
at all. This is documented in the kernel's security policy, yet we
still have a number of people asking for CVE numbers even after
reading that policy. See my longer "CVEs are dead..." talk for full
details about how the CVE process is broken for projects like Linux:
https://kernel-recipes.org/en/2019/talks/cves-are-dead-long-live-the-cve/

- You HAVE to take all of the stable/LTS releases in order to have a
secure and stable system. If you attempt to cherry-pick random
patches you will NOT fix all of the known, and unknown, problems,
but rather you will end up with a potentially more insecure system,
and one that contains known bugs. Reliance on an "enterprise"
distribution to provide this for your systems is up to you, discuss
it with them as to how they achieve this result as this is what you
are paying for. If you aren't paying for it, just use Debian, they
know what they are doing and track the stable kernels and have a
larger installed base than any other Linux distro. For embedded,
use Yocto, they track the stable releases, or keep your own
buildroot-based system up to date with the new releases.

- Test all stable/LTS releases on your workload and hardware before
putting the kernel into "production" as everyone runs a different %
of the kernel source code from everyone else (servers run about
1.5mil lines of code, embedded runs about 3.5mil lines of code, your
mileage will vary). If you can't test releases before moving them
into production, you might want to solve that problem first.

- A fix for a known bug is better than the potential of a fix causing a
future problem as future problems, when found, will be fixed then.

I think I need to give another talk about this issue to go into the above in more detail. So much for me giving a technical talk at Kernel Recipes this year...

When your server is doing too many builds at once...

On the always good https://kottke.org/ the other day, there was a link to a series of videos of professional drummers hearing songs for the first time, without the drum track, and then playing what they think would fit. Here's one of them, https://www.youtube.com/watch?v=tbUYVcaF_l0 and while I'm not a drummer, the best part of this is how Dirk Verbeuren explains the process of listening to the song, and how to learn something never seen before and how to adapt to it. The whole series is recommended: https://www.youtube.com/playlist?list=PLThYwnIoLwyWiF5RgHPzOzYNdqQw1-tep

Anyway, got me to thinking, a while ago a friend (also a kernel developer) had the idea of a presentation for the yearly Kernel Recipes conference that would be him sending me a patch, me reviewing it, talking about how it is reviewed, and the back and forth between us on getting it into a mergable state. That process is one that someone else recently asked me "what presentations can you recommend for new kernel developers to explain how this review process works" and I didn't have any suggestions, but it really is an important thing that is not taught at all in school, or in any company that I know of.

So maybe, a series of videos, or talks, where a maintainer gets a patch series and walks through how they review it, what they look for, what they expect, and how its tested (if at all), might be interesting? Or "here's a reported bug, how do you debug it?" type of presentation to put a developer through the steps of attacking something unknown, and figuring out what the problem is, and how it could possibly be solved.

Brings me back to the days of the Plumbers conference session where we had "bring us your laptop and we will get suspend/resume to work on it" tracks that ended up being a lot of fun for everyone involved as crazy hardware/bios issues were debugged live.

Would this even be interesting? I think it might need a lot of good editing, you don't want to see me staring numly at a terminal window for a few hours while reading lots of inscrutable driver code tracing it to find a bug, that would put everyone to sleep...

@kernellogger It's patch Tuesday! {sigh} This has been a pain to get done, odds are it will take a few more cycles and fixups once it hits real testing...

RE: https://fosstodon.org/users/kernellogger/statuses/110855114931496454

Thanks to @jstultz for showing that the monstrosity of USB A-C-A with a magnetic C connector in the middle actually works! Saves me constantly having to plug/unplug my microphone and camera every other day into my well-worn USB hub.

John did it much better with one less adapter than I so this stack could be smaller if needed, but this is all I could find locally.

First talk I've given in person in a very long time, and it's on legal issues (the EU Cyber Resilience Act), I must have done something wrong in a past life:
https://kccnceu2023.sched.com/event/1Lnv0

Should be fun, if you are at KubeCon, stop by and ask questions, it's meant to be a discussion. A recording will be made public afterwards too.

bpftrace fun question of the week I’ve been beating my head against for a while now.

Given the following bpftrace program:

tracepoint:syscalls:sys_enter_open,
tracepoint:syscalls:sys_enter_openat
{
        $g = "magic_command_to_exit_trace";
        $s = str(args->filename);
        printf("%s\n", $s);
        if ($s == $g) {
                exit();
        }
}

I get the lovely warning:

WARNING: Addrspace mismatch
    if ($s == $g) {

which I can understand. But what I can’t figure out is how to resolve this (hint, the program works just fine, when opening the “magic” file, the trace exits), as how to turn a literal string into the proper address space that args->filename is?

I’ve dug into too many bpftrace git commits to try to figure it out, to no luck. Anyone have a hint?

Oh, and if you want to see where this is used, it’s in this “fun” script: https://github.com/gregkh/gregkh-linux/blob/master/scripts/trace_kernel_build.sh

Warning, realpath takes a long time when processing millions of files, be patient when running the script.

There's been a long nerd-sniping thread recently here from @monsieuricon where email message-ids were being discussed and generated in semi-interesting ways that ended up detouring me and @brauner into writing up competing python vs. perl scripts to get `git send-email` to properly use our new bespoke message ids:
https://social.kernel.org/notice/AU5IphRPUsQvvkx732

But why does any of this matter? As most everyone knows, Linux kernel development happens through email, and the Message-Id of an email is a unique identifier that is used to track messages in our patch handling tools and archives (see https://lore.kernel.org for the archives.) By crafting shorter-but-still-unique message ids it's easier to reference those messages in other places, and using words is just prettier overall than random UUID values (https://i.redd.it/64gl4t9s52ra1.jpg for an example)

Bonus to all of this is that people don't realize that most of the patches we send out are actually signed and can be validated as coming from the person that sent them. The tool we use for that looks at the body of the email, and a small subset of the Header tags in the email. By providing to the tool our custom Message-Id, that adds yet another portion of the email that is now able to be signed and validated, providing a tiny bit more security overall in the patch submission processes (very very tiny, I know, but it's real, as I found out when I submitted a patch with a broken message-id from what was signed and our tools caught it.)

Anyway, all of that is a long way of showing off a tiny core change to the kernel that allows some core structures to be moved to read-only memory that I've been working on for a few months now. Here's the last portion of that work being sucked off of the email archives and validated as coming from me:

Me: we have 50TB on our backend storage system that stores kernel tarballs, so this should be plenty for the next 5+ years.
@gregkh: challenge accepted!

You get a stable kernel release, and you get a stable kernel release, and you get a stable kernel release!

Hopefully things now settle down to the normal constant crazy pace we are used to (1-2 releases a week), instead of the mass of releases we had in the past few days.

@killyourfm Just a short note, your recent #mixtape playlists have been the soundtrack for my recent kernel development and release work for the past few weeks, great work! Many thanks for sharing them!