Just posted a proof of concept that allows you to read a trace after a reboot, including a kernel crash!
https://lore.kernel.org/lkml/20240306015910.766510873@goodmis.org/

@rostedt Awesome, thanks for that POC! I read through your POC description and it would be a very helpful feature!

@rostedt such a feature used to be super useful back then in #OpenWrt, helped a lot on #embedded devices. And would love to have a comeback for it. Don't know why the OpenWrt version never made it upstream though, nor why it was then suddenly dropped in #OpenWrt 21.02 / why it was not ported to #Linux 5.4 anymore.

https://github.com/openwrt/openwrt/blob/openwrt-19.07/target/linux/generic/hack-4.14/930-crashlog.patch

@T_X @rostedt indeed, it's cool to do this for tracing, but would be even more impactful for printk buffer imho.

@vbabka @T_X there is a printk trace event that writes all the printk output to the tracing ring buffer. Enable that event, and you have that feature. 😁

@vbabka @T_X @rostedt don't we already basically have that feature? The pstore subsystem maintained by @kees can store normal dmesg output (CONFIG_PSTORE_CONSOLE), oops/panic messages (CONFIG_PSTORE_RAM), and other stuff. (It can even store ftrace data, I think, CONFIG_PSTORE_FTRACE?)
The pstore backends used for this include RAM regions carved out via a kernel command line flag or in Device Tree (typically used on Android) and storage in EFI variables (typically used on x86), and systemd has integration to automatically copy oopses from pstore to disk on the next boot.

I think if you install a normal Linux distro on a normal x86 machine nowadays, you probably have EFI pstore.

One "fun" thing about storing persistent data in RAM is that at least on many older Android devices, when you reboot, the RAM apparently doesn't stay properly powered the whole way, so it used to be that you'd have random visible bitflips in the stored dmesg copy... but that has also been adressed by adding reed-solomon encoding to make it possible to recover from some bit errors. (But the EFI pstore typically used on X86 does not have that problem.)

@jann @T_X @rostedt @kees good point. I knew about the EFI pstore, but was afraid to use it as it can brick machines with broken firmware IIRC?

@vbabka @T_X @jann @rostedt the EFI backend is enabled by default in at least Fedora. There's even systemd-pstore that moves any crashes and stuff from pstorefs to /var/lib/systemd/pstore
https://www.freedesktop.org/software/systemd/man/latest/systemd-pstore.service.html

@jann @vbabka @T_X @kees Note, the CONFIG_PSTORE_FTRACE just adds hooks into the ftrace infrastructure to have it write into the pstore. What I did is different. Here you give the ftrace infrastructure a block of memory (starting address and size), and it will map its ring buffer on top of that. There’s no hooks. All functionality of ftrace will write into the that range of memory.

Yes, if pstore can give me a block of memory, I’ll use it. Really, the code I wrote just lets you use any block of memory. How I get that block of memory is part 2 of his story. 😉