I didn't really work on my legs much before, but ever since I started PT, my legs hurt with every move I make

On diet day 3

Uh I don't want to misread code

Book cover revealed :)

Very happy with it!

https://nostarch.com/linux-memory-manager

First day of PT, quite tired.
…now my trainer says only two sandwiches are allowed every day. Oh no!

For a while now, the kernel's configuration and build systems have been an area of concern for me. Almost nobody truly understands those complex subsystems, which were handled by a single maintainer.

That maintainer, Masahiro Yamada, has just stepped down after eight years on the job:

https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d6841d5cb20

Happily, Nathan Chancellor and Nicolas Schier have agreed to pick up the build system. The configuration system, instead, is now unmaintained. That ... seems less than optimal.

Thanks to Masahiro for doing this work all these years, and to Nathan and Nicolas for stepping up!

Still digesting it, and may be slightly incorrect, but a summary of what I've learned: properly using statistics to compare the performance of computer systems [1] [2].

0. When you get a sample and calculate sample mean, it is likely different from the population mean. Simply comparing two sample means from different computer systems may therefore lead to misleading conclusions, since the observed difference could be due to random sampling variation rather than a real performance difference (especially when the variance is high).

1. Central Limit Theorem indicates that, regardless of the underlying distribution, the sampling distribution of the mean tends to follow normal distribution when the sample size is sufficiently large (typically n >= 30). But to apply CLT, the observations should be independent and collected from the same distribution.

2. Based on the CLT, we can estimate how close the sample mean (to be precise, any population parameter) is likely to be to the population mean. A confidence interval [x, y] with a confidence level of p% means that if we repeated the sampling process many times under the same conditions, about p% of those intervals would contain the population mean.

3. To compare two distributions, confidence intervals can help determine whether the difference between mean values is statistically significant. This can be done by:

- Checking whether the confidence intervals of the two samples do not overlap, or
- Examining the confidence interval for the mean of the differences to check if it does not include 0.

4. Caveat: The CLT assumes that the data is collected independently, meaning one observation does not affect another. In computer systems, this assumption usually does not hold. Caches, memory layout, scheduling decisions etc. can introduce some degree of dependencies between observations.

This can be mitigated by 1) reducing dependence between experiments as much as possible or 2) by applying the bootstrapping method [3].

[1] Jan Kara, Measuring performance regressions, https://youtu.be/HAHhW13ofrg?si=drgegMwXUDegHsQf
[2] Dev Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, https://www.amazon.com/Art-Computer-Systems-Performance-Analysis/dp/0471503363
[3] https://en.wikipedia.org/wiki/Bootstrapping_(statistics)

Got a new small bookshelf but the user experience isn’t great

Oh god let me fall asleep

@sj recently started using drgn to add more DAMON selftests, which is a super interesting use case that I didn't envision: https://lore.kernel.org/all/20250628160428.53115-1-sj@kernel.org/. It has already found a real bug! https://lore.kernel.org/all/20250719181932.72944-1-sj@kernel.org/

linux-stable is a vibe-backported frankenkernel

All microconferences (MCs) at LPC 2025 have been accepted! It is time to submit topics to your favorite MCs.

Please check out our latest blog post for the list of MCs, and how to create a ideal MC topic.

https://lpc.events/blog/current/index.php/2025/07/25/all-microconferences-have-been-accepted/

I've been playing with the LLM code assistants, trying to stress them out with the Linux kernel code. So far, I've had success with them writing reasonable unit tests:
https://lore.kernel.org/lkml/20250717085156.work.363-kees@kernel.org/
https://lore.kernel.org/lkml/20250724030233.work.486-kees@kernel.org/
These both saved me some time since it emitted quite a lot of good boundary testing code, but I had to massage them both a bit until I was happy with the coverage. But it was a net win on time spent.

And then I walked it through fixing a buffer overflow. This one didn't save me any time because I had to tell it how to look at the problem. Since it was a shorter/simpler session, I included my exact prompts just for anyone interested in what I did:
https://lore.kernel.org/lkml/20250724080756.work.741-kees@kernel.org/

This graph is the one I'm most excited about: the lifetime of security flaws in Linux is finally starting to get shorter (and the number of fixed flaws continues to rise).

https://hachyderm.io/@LinuxSecSummit@social.kernel.org/114750428620118674

Recently, I was debugging some intermittent boot failures and discovered that, on x86, forgetting to synchronize kernel page tables when installing new PGD entries can lead to kernel crashes.

Most of the time, PGD entries are populated during the boot process, and new tasks inherit page tables derived from the swapper process, so everything works fine.

However, if you're adding memory at runtime via memory hotplugging and populating new PGD entries for the vmemmap (i.e., the struct page array) and the direct mapping area, you need to iterate over all page tables in the system and update them accordingly to make it visible to all tasks.

Failing to handle this properly can lead to a situation where you think the page tables are set up correctly, but in reality, only init_mm.pgd (page table of the swapper process) was updated, not the page table of current task. This leads to kernel crashes since the PGD entry is not set up properly.

x86 code has mechanisms to handle this kind of synchronization, but it’s easy to be overlooked and introduce kernel crashes as the code evolves. Here’s my patch series that addresses the issue and aims to make it more robust: https://lore.kernel.org/linux-mm/20250709131657.5660-1-harry.yoo@oracle.com/

GNU Butt

It took a long time and over 60 articles but, at @lwn, we have finally managed to complete our reporting from the 2025 Linux Storage, Filesystem, Memory Management, and BPF Summit. If you want to know what is going on in those core parts of the kernel, this is the place to look.

We've put together an EPUB version of the whole set as well — good bedtime reading!

https://lwn.net/Articles/1026338/

In case you want to join my company (#SUSE), we're looking for an #infrastructure engineer!
#FediHire
https://suse.wd3.myworkdayjobs.com/Jobsatsuse/job/Czech-Republic-EMEA/Senior-Software-Engineer--Core-Platforms-_71006946

How does the kernel manage LRU lists and perform memory reclaim when memory cgroup feature is enabled? I wrote an article to answer that question.

Memory Cgroup Naturalization in the Linux Kernel v3.3
https://vmscan.org/memory-cgroup-naturalization-in-the-linux-kernel-v33

In the early days, memory cgroup was “bolted onto” the memory management subsystem, using a double-LRU scheme: each page was linked to both global and per-memcg lists—leading to a moderate memory overhead and complicating reclaim logic.

Linux v3.3 changed that. Memory cgroups were fully integrated into the MM core, global LRU lists were removed on memcg-enabled systems, and global reclaim began walking memcg hierarchy and scanning the per-memcg LRU lists, scanning each group in proportion to its memory usage.

This reduced memory overhead and simplified the reclaim code. More details in the article!

Linux 6.15 Released With Continued Rust Integration, Bcachefs Stabilizing

As anticipated the Linux 6.15 kernel is out today in stable form. Linux 6.15 brings a lot of new hardware support, security improvements, various other kernel innovations, and more...
https://www.phoronix.com/news/Linux-6.15-Released