You hear that Mr. Kernighan? ... That is the
sound of inevitability... It is the sound
of improvement...
It's time, Mr. Kernighan....

@ljs lol how embarrassing, they totally fell for the "technical argument" smokescreen. It's stupid to defend the technical points of Kent's replies in that thread, because the technical aspect was never the point.

And yet the "non-technical" aspects are just as important to Linux's technical excellence. I can't remember where I read that Linux's true innovation wasn't its design, but rather its development model, but it's true. Linux isn't defined by a single, great technical decision. Instead, it's great because it's constantly evolving through collaboration. Jeopardizing that is a much bigger deal than any single feature.

@vbabka wow, I didn't know that about m68k. From what I can tell, the rb_node is aligned so that they can tag the parent pointer with the node color, which is just one bit. If that's true, then additional alignment shouldn't be necessary even on m68k, but who knows what else might be relying on the explicit alignment.

• Step 1: While debugging a kernel issue, you want the return value of a function at startup.

• Step 2: Instead of just adding a printk() and recompiling, you try tracing like the cool kids.

• Step 3: Since 2023, the function graph tracer can show return values. You add to the kernel command line and reboot: ftrace=function_graph ftrace_filter=interesting_fn trace_options=funcgraph-retval

• Step 4: The trace shows no return value. You find that ARM return value capture exists only for AArch64, not ARM32.

• Step 5: You add a printk() and rebuild.

struct rb_node in the kernel has this pointless alignment attribute: __attribute__((aligned(sizeof(long)))).

It always makes me laugh because it was added in 2006 (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e977145aeaad23d443686f2a2d5b32800d1607c5) for an obscure architecture that the kernel dropped support for in 2018 (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c690eddc2f3b44b24520f4a77cc3a4c9bde7d571).

For a while now, the kernel's configuration and build systems have been an area of concern for me. Almost nobody truly understands those complex subsystems, which were handled by a single maintainer.

That maintainer, Masahiro Yamada, has just stepped down after eight years on the job:

https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d6841d5cb20

Happily, Nathan Chancellor and Nicolas Schier have agreed to pick up the build system. The configuration system, instead, is now unmaintained. That ... seems less than optimal.

Thanks to Masahiro for doing this work all these years, and to Nathan and Nicolas for stepping up!

@sj recently started using drgn to add more DAMON selftests, which is a super interesting use case that I didn't envision: https://lore.kernel.org/all/20250628160428.53115-1-sj@kernel.org/. It has already found a real bug! https://lore.kernel.org/all/20250719181932.72944-1-sj@kernel.org/

@hipsterelectron you can put the process in its own mount namespace and mount a FUSE filesystem in that namespace.

@jann huh, do you know why the kernel asks for only 8 byte alignment?

@pervognsen @dougall yeah, exactly (although you might already be aware of the long-deprecated uselib syscall: https://man7.org/linux/man-pages/man2/uselib.2.html)

@pervognsen @dougall on Linux, there is a "debugger rendezvous" structure named r_debug which the debugger finds via an entry in the dynamic section. r_debug has a member, r_brk, which is a code address that is hit whenever a shared object is loaded/unloaded. The debugger can set a breakpoint there if it wants to be notified of those events and inspect other members in r_debug for details.

It's very Unix "worse is better" (which makes sense since I think it originated in System V and came to Linux via glibc).

The next Linux Kernel Debugging Tools Monthly Meeting is tomorrow, Wednesday, May 28th at 11:30 AM Pacific time. As usual, see the agenda on the linux-debuggers mailing list: https://lore.kernel.org/linux-debuggers/aDYy37NBkt7mdaJY@telecaster/T/#u and let me know if you want an invite.

What's that mysterious workaround?

Core Huff6 decode step is described in https://fgiesen.wordpress.com/2023/10/29/entropy-decoding-in-oodle-data-x86-64-6-stream-huffman-decoders/

A customer managed to get a fairly consistent repro for transient decode errors by overclocking an i7-14700KF by about 5% from stock settings ("performance" multiplier 56->59).

It took weeks of back and forth and forensic debugging to figure out what actually happens, but TL;DR: the observed decode errors are all consistent with a single instruction misbehaving.

@vitaut I'm pretty sure we've had some of those bubble up as kernel bug reports lol

@brenns10 yeah, it's also an ideal test case in that it's somewhat mechanical. Maybe a bigger corpus of drgn helpers out there would help

@brenns10 I am extremely amused by Cursor's attempts at a drgn helper, thanks for sharing lol

I've hinted at this a bit, but it's finally to a point where I feel comfortable with other people using it.

I've spent the last 8 months iterating on different ways to debug really large and complicated applications. systing is the tool that's come out of this work. I wrote a post about it introducing it, and there's some documentation in the repo for it. It's really just designed for people like me who have a pretty deep knowledge of the kernel and how userspace interacts with the kernel. Hopefully my fellow kernel developers find this useful.

https://josefbacik.github.io/kernel/systing/debugging/2025/05/08/systing.html

@ljs @brenns10 FYI the drgn repo has a contrib directory for anyone to dump their scripts in: https://github.com/osandov/drgn/tree/main/contrib. Totally up to you wherever you want to keep them, obviously

@brauner nice! I think it'd be pretty cool to set the coredump handler to a drgn script that could directly extract only the desired information without needing to capture a full core dump at all. I think I saw this scroll by in your discussion with @jann, but I assume /proc/pid/mem, /proc/pid/maps, etc. are still alive at that point? In particular, drgn would love to use /proc/pid/map_files/ so that it unambiguously gets the correct files, but that requires PTRACE_MODE_READ_FSCREDS, so it couldn't be fully unprivileged