trace-cmd 3.4 has been released!

What’s new? The biggest change is that if you record with function or function graph tracer and enable the func-args or funcgraph-args then trace-cmd will save the BTF file (/sys/kernel/btf/vmlinux) in the trace.dat and use it for trace-cmd report to show the functions:

 function:             mutex_unlock(lock=0xffffffff831dbbe0)
 function:             __mutex_unlock_slowpath(lock=0xffffffff831dbbe0, ip=0xffffffff814a7154)
 function:             __f_unlock_pos(f=0xffff8881538de000)
 function:             mutex_unlock(lock=0xffff8881538de090)
 function:             __mutex_unlock_slowpath(lock=0xffff8881538de090, ip=0xffffffff816e8ed1)
 function:             mem_cgroup_handle_over_high(gfp_mask=0xcc0)

Also, if you run trace-cmd extract on a persistent ring buffer, it will save the last_boot_info file (and /proc/modules) and use it to properly calculate the addresses of the functions from the previous boot to the addresses of the same functions of the current boot so that it can properly get their names from the saved kallsyms. This way you can even see all the functions that did the reboot!

boot_mapped:            <...>-1     [000] d..1.  6869.910269: function:             native_machine_emergency_restart()
boot_mapped:            <...>-1     [000] d..1.  6869.910269: function:                tboot_shutdown(shutdown_type=0x0)
boot_mapped:            <...>-1     [000] d..1.  6869.910269: function:                machine_real_restart(type=0x0)
boot_mapped:            <...>-1     [000] d..1.  6869.910269: function:                   _raw_spin_lock(lock=0xffffffff95d03214)
boot_mapped:            <...>-1     [000] d..2.  6869.910269: function:                   rtc_cmos_write(val=0x0, addr=0x8f)
boot_mapped:            <...>-1     [000] d..2.  6869.910276: function:                   _raw_spin_unlock(lock=0xffffffff95d03214)
boot_mapped:            <...>-1     [000] d..1.  6869.910277: function:                   load_trampoline_pgtable()
boot_mapped:            <...>-1     [000] d..1.  6869.910277: function:                   __flush_tlb_all()
boot_mapped:            <...>-1     [000] d..1.  6869.910278: function:                      native_flush_tlb_global()

libtraceevent 1.9 has been released!

What’s new? Support for loading BTF and printing function arguments. Also, processing of the persistent ring buffer. It adds APIs to be able to map the function addresses from the previous boot to address of the current boot so that their names can be found from kallsyms.

@vbabka @DesnoyersMa I’ve sent Linus default y configs before. But I always mention “this keeps previous behavior”. What gets him pissed off is when you send a default y for a new feature. If it’s something that doesn’t change his config, he hasn’t complained to me about it before.

@DesnoyersMa The default needs to keep how it is today the same. For hiding, it would be default N.

@DesnoyersMa I would actually make it CONFIG_HIDE_TRACE_PRINTK to allow those that don't want it included to be able to have that option. As right now it's something people want that build their kernels.

I’m fighting to keep trace_printk.h in kernel.h. If you use trace_printk() for debugging, and do not want to have to add:

#include <linux/trace_printk.h>

to every file you you want to add a trace_printk() to, please make yourself heard and respond to this email thread.

Otherwise, trace_printk() will become a bit more tedious to use.

https://lore.kernel.org/all/20251229111748.3ba66311@gandalf.local.home/

I’ve once again compiled my server and workstation with the branch profilers enabled. You can see the data for my server here

The branch_annotated file is where “likely”() and “unlikely()” are correct or not.

The branch_all is if the if statement was true or false.

The data is updated nightly on a live machine. I’ll run his for a couple of weeks to see where the Linux kernel gets it wrong.

The currently running version is v6.18.2.

@cyberia @torvalds It is a shame. I love “right to modify” and “right to repair”, but honestly, GPLv2 does not have a clause about the device itself. It’s only about the software. If the device requires keys and such to run, the GPLv2 says nothing about allowing it to run.

It’s not about what we want the license to mean (although Torvalds has the right to state his intent), it is also about what the license actually says. The judge mentioned:

If this was the intent of the Agreements, the Agreements could have been readily modified to state that users must be permitted to modify and reinstall modified software on products which use the program while ensuring the products continue to function.

The GPLv2 is a little ambiguous with the install scripts, and the judge is saying that due to that ambiguity, it does not cover reinstalling on devices. If it did, then the “agreement” (the GPLv2 license) should explicitly state so.

@MichaelOpdenacker @fosdem @openembedded

Good luck on your talks. I won’t be going. There’s a comedian coming to my home town that weekend and I booked tickets fro my wife and myself. She’d be pissed if I left for a conference instead. Maybe next time!

@codonell BTW, the wait list has a priority order where people who have submitted content gets first dibs. Did you submit to the Toolchain MC?

Linux Plumbers in-person tickets are sold out! But you can still add yourself to the wait-list. Read the blog for more details. https://lpc.events/blog/current/index.php/2025/09/19/in-person-registration-is-sold-out-2/

Linux Plumbers 2025 in Tokyo’s Call for Proposals is nearing its end! https://lpc.events/blog/current/index.php/2025/09/08/the-call-for-proposals-is-nearing-its-end/

@itaru Of course it is. It’s part of our tradition!

@monsieuricon You should see the emails we get for the Linux Plumbers Conference mailing list!

All Linux Plumbers Microconferences have been accepted for 2025!

You can now submit to a specific MC topic of your choosing.

https://lpc.events/blog/current/index.php/2025/07/25/all-microconferences-have-been-accepted/

Linux Plumbers Refereed track and Microconference proposals are now open for submission! https://lpc.events/blog/current/index.php/2025/05/14/submission-time-for-linux-plumbers-2025/

ANNOUNCE: The following has been released:

trace-cmd 3.2.2

libtracefs 1.8.2

A ftrace selftest just failed due to the differences between bash and dash on how they interpret backslashes (\).

Bash will evaluate it once, where as dash will evaluate it every time.

i=123
TEST="\\$i"
echo "$TEST"

Bash will show: \123

where as dash will echo out the escape sequence of “123”.

A kprobe event self test needed to test the max number of arguments, where each argument held a backslash followed by a number. It created it with the loop:

   TEST_STRING=$1
   # Acceptable
   for i in `seq 1 $MAX_ARGS`; do
    TEST_STRING="$TEST_STRING \\$i"
   done
   echo "$TEST_STRING" >> dynamic_events

Which worked fine when /bin/sh was bash, but when /bin/sh was dash, it failed!

And checkbashisms did NOT catch it!

Just added tab completion for trace-cmd sqlhist for making synthetic events with the sql interface so much easier!

I’ve been wanting to add to trace-cmd start a way to make it start quicker. It’s currently slow because it resets most of tracefs before starting, and that can take seconds.

I forgot I already have this feature! trace-cmd set acts just like start, but it does not do the reset before starting. It’s very quick!

Note, it also does not re-enable tracing if it’s disabled. So you may want to run trace-cmd restart right after that, if that’s the case.