You hear that Mr. Kernighan? ... That is the
sound of inevitability... It is the sound
of improvement...
It's time, Mr. Kernighan....

• Step 1: While debugging a kernel issue, you want the return value of a function at startup.

• Step 2: Instead of just adding a printk() and recompiling, you try tracing like the cool kids.

• Step 3: Since 2023, the function graph tracer can show return values. You add to the kernel command line and reboot: ftrace=function_graph ftrace_filter=interesting_fn trace_options=funcgraph-retval

• Step 4: The trace shows no return value. You find that ARM return value capture exists only for AArch64, not ARM32.

• Step 5: You add a printk() and rebuild.

struct rb_node in the kernel has this pointless alignment attribute: __attribute__((aligned(sizeof(long)))).

It always makes me laugh because it was added in 2006 (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e977145aeaad23d443686f2a2d5b32800d1607c5) for an obscure architecture that the kernel dropped support for in 2018 (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c690eddc2f3b44b24520f4a77cc3a4c9bde7d571).

For a while now, the kernel's configuration and build systems have been an area of concern for me. Almost nobody truly understands those complex subsystems, which were handled by a single maintainer.

That maintainer, Masahiro Yamada, has just stepped down after eight years on the job:

https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d6841d5cb20

Happily, Nathan Chancellor and Nicolas Schier have agreed to pick up the build system. The configuration system, instead, is now unmaintained. That ... seems less than optimal.

Thanks to Masahiro for doing this work all these years, and to Nathan and Nicolas for stepping up!

@sj recently started using drgn to add more DAMON selftests, which is a super interesting use case that I didn't envision: https://lore.kernel.org/all/20250628160428.53115-1-sj@kernel.org/. It has already found a real bug! https://lore.kernel.org/all/20250719181932.72944-1-sj@kernel.org/

The next Linux Kernel Debugging Tools Monthly Meeting is tomorrow, Wednesday, May 28th at 11:30 AM Pacific time. As usual, see the agenda on the linux-debuggers mailing list: https://lore.kernel.org/linux-debuggers/aDYy37NBkt7mdaJY@telecaster/T/#u and let me know if you want an invite.

What's that mysterious workaround?

Core Huff6 decode step is described in https://fgiesen.wordpress.com/2023/10/29/entropy-decoding-in-oodle-data-x86-64-6-stream-huffman-decoders/

A customer managed to get a fairly consistent repro for transient decode errors by overclocking an i7-14700KF by about 5% from stock settings ("performance" multiplier 56->59).

It took weeks of back and forth and forensic debugging to figure out what actually happens, but TL;DR: the observed decode errors are all consistent with a single instruction misbehaving.

I've hinted at this a bit, but it's finally to a point where I feel comfortable with other people using it.

I've spent the last 8 months iterating on different ways to debug really large and complicated applications. systing is the tool that's come out of this work. I wrote a post about it introducing it, and there's some documentation in the repo for it. It's really just designed for people like me who have a pretty deep knowledge of the kernel and how userspace interacts with the kernel. Hopefully my fellow kernel developers find this useful.

https://josefbacik.github.io/kernel/systing/debugging/2025/05/08/systing.html

I've done a series that adds support for AF_UNIX sockets in coredumps. Userspace provides an AF_UNIX socket path via core_pattern and the kernel connects to it, shuts down the read side and writes the coredump to the socket.

This means no more super privileged usermode helper upcalls and makes for a very nice API experience. I captured coredumps simply via socat:

https://lore.kernel.org/20250430-work-coredump-socket-v1-0-2faf027dbb47@kernel.org

The receiver can use SO_PEERPIDFD to get a stable handle on the crashed process.

#kernel #linux #coredump

@monsieuricon does social.kernel.org not support verified profile links or am I dumb?