Learning to compile distro kernel for Fedora. I took a random patch set from mm just to learn this process [1]. Has taken couple of days because I don't know anything about RPM and have compiled so far only distro kernel for Ubuntu. I'm recent convertant to Fedora because between 2004-2021 I used Ubuntu util I had had enough ;-)

Glitches or things to reconsider:

- I have now kernel compilation ongoing with "fedpkg mockbuilld". I hope this is the right command. I got the kernel by "fedpkg clone --anonymous kernel ".
- I have no idea how to point out linux-next so had to backport a single patch from kselftest to v6.12-rc4. Any pointers how to use that nice fedpkg command AND have linux-next is appreciated.
- I hope this will ensure that the kernel will signed with my MOK key: [2].

[1] https://lore.kernel.org/all/cover.1729440856.git.lorenzo.stoakes@oracle.com/
[2]
I have a key created:

$ sudo certutil -L -d /etc/pki/pesign

Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI

Machine Owner Certificate Pu,Pu,Pu

And I hope this will pick it for the kernel build:

--- a/kernel-local
+++ b/kernel-local
@@ -1,2 +1,3 @@
# This file is intentionally left empty in the stock kernel. Its a nicety
# added for those wanting to do custom rebuilds with altered config opts.
+%define pe_signing_cert "Machine Owner Certificate"

#fedora #linux #kernel #rpm

@jarkko cc: @Conan_Kudo

@jarkko "Learning to compile distro kernel for Fedora": which way?

Modify the SRPM?

Modify dist-git (https://src.fedoraproject.org/rpms/kernel – upstream for the SRPM)?

Modify kernel-ark (https://gitlab.com/cki-project/kernel-ark – upstream of dist-git)?

@jarkko reg -next:

I'm biased, but using something based on[1] (based on kernel-ark) might be the wises if you want to do that more often in the long run.

I really should write a short "how to use Thorsten's stuff for your own use cases as a kernel developer" entry for the faq…

[1] https://copr.fedorainfracloud.org/coprs/g/kernel-vanilla/next/ ; https://fedoraproject.org/wiki/Kernel_Vanilla_Repositories & https://fedoraproject.org/wiki/Kernel_Vanilla_Repositories-FAQ#Where_is_the_code_from_which_the_packages_are_built?;

@kernellogger I don't know. And also: I don't know what dist-git is :-) I spotted that maybe in some error message along the way.

So fedpkg is not the way to go for kernel dev?

Anyway I found many:

1. https://docs.fedoraproject.org/en-US/quick-docs/kernel-build-custom/
2. https://docs.fedoraproject.org/en-US/quick-docs/kernel-testing-patches/ (I'd pay a note taht this page specifically speaks about kernel dev task and uses fedpkg. Thus one will asssumee that a kernel developer should stick to fedpkg, right?)
3. https://fedoraproject.org/wiki/Building_a_custom_kernel

Pretty hard to get from A to B IMHO ;-) I wish there was a tutorial page on "how do upstream kernel development with Fedora and build compatible packages with opt-in secure boot. Here's how you do it."

Ubuntu had smooth procedure where you had actual kernel git with packaging patches on top. This broke of course with snap kernel images so does not hold any more.

@kernellogger dumb question perhaps: what is kernel-ark?

@jarkko

then you most likely want kernel-ark: https://cki-project.gitlab.io/kernel-ark/

But it's a bit tricky to get used to it. :-/

And not idea wrt to secure boot signing. :-(

@jarkko

> But it's a bit tricky to get used to it. :-/

I really should write something up to make this easier, as the stuff in the "ark-infra" branch is really handy once you know about it and how to use it.

@kernellogger If you can append to kernel.spec, e.g. to kernel-local file, then that does the job. Thanks I give a shot.

I exactly picked one of @ljs patch sets because it is from alien subsystem. Using the stuff that I review myself and send to Linus eventually, could make me blind. This way I believe I can develop a robust workflow :-)

I mean 9/10 I just do trivial BuildRoot build but sometimes it is nice to be able to do proper packages and eat your own dog food (or should it be cat food given the patch set)...

@kernellogger i'm sure i've seen worse don't worry ;-)

@jarkko I'm not sure if this is helpful or not, but for several years now I've been building Fedora Rawhide kernel packages with the LSM, SELinux, and audit linux-next patches included. It isn't perfect, but it works reasonably well for me and barring an odd merge conflict it runs completely automated.

https://paul-moore.com/blog/d/2019/04/kernel_secnext_repo.html

@kernellogger for me the last draw was this FDE or whatever crap where you either have deb based kernel or snap based. too confusing for a developer like me - especially since the snap based had all the stuff I was interested (e.g. TPM2 boot) is the snap version. And as we know the Snap Store itself is as proprietary as Apple's App Store...

@jarkko I've been on Fedora since before Ubuntu existed :-D Contributing to Fedora is actually how I did my first steps in contributing to Open Source.

And yeah, I prefer to not touch Snap due to the Snap Store situation; there are also other problems that make it a bad candidate for corss-distro packaging tool. like the confinement depending on AppArmor for example.

@securepaul @jarkko

FWIW, I started using copr for my builds (https://fedoraproject.org/wiki/Kernel_Vanilla_Repositories ) a while ago; build times are still a problem, but never saw something like the "the build chroot often breaks when Fedora Rawhide moves to the next release" you describe.

@securepaul this absolutely useful as i have absolutely no idea what i'm doing :-) i'll experiment with your and Thorsten's suggestions and try to find my own way...

@kernellogger @securepaul thanks!

@kernellogger @securepaul OK, this I've done successfully in the past but being ablet to do local builds is an asset :-)

For instance I use these sometimes just so that I can pre-check anything I push to Gitlab/Github just to check locally what CI will do:

- Gitlab: https://github.com/firecow/gitlab-ci-local
- GitHub: https://github.com/nektos/act

@kernellogger @jarkko That COPR build chroot problem may have gotten better over the years, I can't say for certain as I no longer rely on the COPR builds, I've found my local builds (repo.paul-moore.com) to much more reliable/quicker.

FWIW, I do still send the kernel-secnext builds to COPR, I just don't worry about it too much when the COPR build breaks.

@jarkko what's wrong with "make install" ?

@jarkko @kernellogger The automation is a big part of this, both in doing the build and executing your tests. You want something that can do what it needs to do with minimal manual intervention.

I'm not a good example here as my solution is a monstrous general job scheduler written in Bash back when the only other realistic option was Jenkins. If you're curious there is a slightly outdated version posted here, but be warned it is hacky beyond belief :)

https://github.com/pcmoore/pjob

@securepaul @kernellogger does not matter. i could grab some ideas. so far i don't have RPM packages that i'd be happy with using any method.

this is exactly why i was so committed to ubuntu even when i hated most of their decisions. only when they put kernel to snap my marriage ended.

for any distro, i still think, would be optimal to find a way to describe the build using a kernel tree with extra patches adding the packaging shenanigans on top. i think all major distributions fail misserably serving upstream kernel developers :-)

@Aissen it does not fulfill the test case i'm lookin for. using the distro kernel as it was meant to be. and e.g. integrating properly on how a distribution has chosen to integrate with secure boot, which is essential in my case.

for ad-hoc testing i have already a solution, which does the job perfectly generating both image for USB stick and scripts for running same payload in QEMU: https://codeberg.org/jarkko/linux-tpmdd-test

What you are suggesting would be a worse version of this, contaminating my rootfs.

@jarkko But the kernel you build isn't signed by your distro in this case?

@Aissen Most distributions (including Fedora) provide reasonable ways to integrate your own additional MOK key, and get the build sign everything with that. And ubiquitos way to make shim to recognize that is "mokutil --import".

For Fedora I found the way it manages extra MOK keys from here: https://docs.fedoraproject.org/en-US/quick-docs/kernel-build-custom/

And the extra flag I need to add to file called "kernel-local" is %define pe_signing_cert "Machine Owner Certificate", so that RPM build shenanigans will pick it up.

@Aissen See, I have my own here:

$ sudo certutil -L -d /etc/pki/pesign

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Machine Owner Certificate                                    Pu,Pu,Pu

@Aissen That said I would not recommend those instructions otherwise. I've found at least four different "how to build kernel" tutorials even from Fedora's domain and all of them are broken. You have to pick right bits here and there ;-)

@jarkko @securepaul

> a kernel tree with extra patches adding the packaging shenanigans on top

1/ It should work round about like this with kernel-ark [untested, so I will have gotten something wrong or forget something]::

@jarkko @securepaul

2/

sudo dnf install git make gcc flex bison bzip2 rpm-build python3-pyyaml
git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/
git remote add -t ark-infra kernel-ark
git fetch kernel-ark
git merge kernel-ark/ark-infra
sed -i '/# We are using a recursive / i include Makefile.rhelver\n' Makefile
sudo dnf -y install $(make dist-get-buildreqs | grep "Missing dependencies:" | cut -d":" -f2-)
make dist-srpm
mock redhat/rpm/SRPMS/kernel-*.src.rpm

@jarkko TIL. Thanks !

@jarkko can't wait for your article with "just" the right bits ;-)

@jarkko @securepaul

3/ obviously you can add patches to the tree; that mock build will take a while, as it will build non-debug and debug kernels; disabling this requires commands I would have to look up.

not sure if "git merge kernel-ark/ark-infra" really works these days; maybe it's better to use "git archive --format=tar "kernel-ark/ark-infra" makefile Makefile.rhelver redhat/ | tar -

@Aissen definitely will do a cheat sheet and write it all down once i found a way!

@kernellogger @jarkko @securepaul and also if you go this way, you can just run pesign at the end instead of trying to test if my rpm macro bits from 2012 that nobody really uses have survived the 12 years of buildsystem-hack wack-a-mole they sit in the middle of.

@kernellogger @jarkko @securepaul
to be clear the rpm macros and related shell scripts are used all the time, but I don't think anyone actually regularly tries just setting pe_signing_cert. It's supposed to work, but... all the code around it is heuristics to figure out where RPMs are getting built, if something is a scratch build, if some other signing mechanism is present, etc., and that's all churned quite a lot in the intervening decade.

@vathpela @kernellogger @securepaul Why is it in one of many Fedora's "how to build a kernel" pages? This the one where I grabbed it:

[1] https://docs.fedoraproject.org/en-US/quick-docs/kernel-build-custom/

That is not even wiki, it is official documentation. Not slandering just pointing out. Deprecated stuff that gets stagnated over time if totally normal!

Also, why nobody talks about fedpkg which is contained to the only page, which *specifically* talks about kernel patch testing:

[2] https://docs.fedoraproject.org/en-US/quick-docs/kernel-testing-patches/

And finally, I'm now a bit lost how do I get pesign to use my cert and key created following [1] to sign RPM packages (which I don't know how I should build it)? I'm don't actually know what pesign is but I guess it is something that signs a PE/COFF binary? I.e. UKI signing perhaps?

@jarkko @kernellogger @securepaul I mean, I don't think it's wrong, I just think the userbase that actually tries to use it that way is vanishingly small.

@vathpela @kernellogger @securepaul OK so how would you sign RPM's? I don't have a preferred way.

It's not a priority question tho because I've not yet after 2-3 days of trying produced successfully RPM's even without signing...

@jarkko @kernellogger @securepaul rpm signing is different, pesign is just for signing kernel/bootloader/etc. Why are you bothering with RPMs to do kernel development, though?

@vathpela @jarkko @kernellogger FWIW, I build and provide RPMs because there are people who would like to help test things but are not necessarily comfortable with building a kernel. It also makes test automation easy; add a repo to your test system and then you just `dnf update`, test, and repeat.

@vathpela @kernellogger @securepaul

This came up in the thread but here we go again :-)

I have my polished framework for testing kernel patches for any possible Git tree:

https://codeberg.org/jarkko/linux-tpmdd-test

You can point it to any tree (by using LINUX_OVERRIDE_SRCDIR) and it will build kernel, rootfs and disk.img and wrapper scripts for QEMU. The disk.img EFI boots on real hardware. The base system is grub + systemd (grub gives option to not use systemd thus the choice) and has all tracing include like bpftrace for instance.

Sometimes, however, one would like run a kernel that is packaged like the real one in the distro but with a different git tree base and/or patches. This could e.g. some issue that does not easily pop up with normal testing. Artificial images have their limits.

I think distributors are making mistake by not taking this seriously and making it robust to do cool stuff with the distro kernel or like make any kernel packaged and signed like the one in the distro. Canonical used to have this asset but the feature I'm interested in are available only in the recent'ish snap packaged kernel.

Or they might think exactly like "why you use RPM's anyway for kernel development". There's good reasons to do that. Also one use case has been few times in the past that you have user space project, which requires tailored kernel. In such case you would like to "emulate" as they were part of the distribution.

@vathpela @kernellogger @securepaul One ugly but totally working hack my kernel testing project: I don't use cmake as a build system. I use it to download source code and get away using Git submodules :-)

https://codeberg.org/jarkko/linux-tpmdd-test/src/branch/main/CMakeLists.txt

It's ugly but solid ;-)

@vathpela @kernellogger @securepaul pretty interesting discussion this came to be. but yeah, it is essentially a flaw in developer experience.

for more complex patch sets it would be beneficial to be able to do such thing with low barrier, i.e. it could also have positive effect on quality for stuff sent to LKML in the end. you always end up doing fairly narrow and targeted testing without this feature in the distribution, which means in practice missing bunch of "side-channel bugs".

@jarkko @kernellogger @securepaul okay "I want to include package install as part of my testing" would have sufficed ;) Do you really need your /own/ cert/signature for testing? Because I think if you just use the kernel-ark ./redhat/fedora-srpm.sh workflow and build the srpm in a mock root, you should get a signature from the "Red Hat Test Certificate", which you can enroll in VMs and whatnot and have that just work.

@jarkko @kernellogger @securepaul ... and kernel-ark gets you a tree you can just cherry pick upstream packages into (with fixups when versions mismatch, obviously, but mostly you get jforbes and others doing rebases for you, which is nice.)

@vathpela @kernellogger @securepaul I'll revisit this thanks :-) Cannot really say but appreciate all this info. Now I need to digest all this for a while.

@jarkko @kernellogger @securepaul okay. If it helps, this is basically the workflow I've been using for making sure the aarch64 parts of pre-boot NX work using the Windows Dev Kit 2023 (a snapdragon desktoppy kind of device), which I've pushed here (not my kernel work, someone else's on a fedora kernel tree) https://gitlab.com/vathpela/kernel-ark/-/tree/blackrock-v6.8.y?ref_type=heads , which is on top of the fedora-6.8 tree (also here: https://gitlab.com/vathpela/kernel-ark/-/tree/fedora-6.8?ref_type=heads ). But I'm doing all that with Secure Boot disabled for expedience.

@jarkko @kernellogger @securepaul (yes, you heard it here first, Secure Boot guy does non-Secure Boot bootloader and kernel work with Secure Boot disabled ;)

@vathpela @kernellogger @securepaul Yeah all this helps! It's just before I take a few days and play with all these cool ideas that you and others kindly provided, anything I would right now would probably be uttermostly incorrect :-) Lot's of stuff to process. Thank you!

@jarkko

1/ I took a closer look and that's how I'd do it:

git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/
cd linux
git remote add --no-tags -t ark-vanilla-mainline ark-vanilla https://gitlab.com/knurd42/linux/
git fetch ark-vanilla-mainline
git archive --format=tar ark-vanilla/ark-vanilla-mainline makefile Makefile.rhelver redhat/ .copr | tar -x
sed -i '/# We are using a recursive / i include Makefile.rhelver\n' Makefile
sed -i 's!define with_realtime 1!define with_realtime 0!' redhat/kernel.spec.template
…

@jarkko

2/…

sed -i 's!^\(UPSTREAM_BRANCH \?.\?=\)\(.*\)!\1 origin!' redhat/kernel.spec.template
git add Makefile makefile Makefile.rhelver redhat/
git add -f .copr redhat/kabi/kabi-module/kabi*
git commit -q -s -m "import ark-infra"
make -j $(nproc --all) dist-srpm
mock redhat/rpm/SRPMS/kernel-*.src.rpm

Yes, this uses a repo I maintain; this nicely avoid a few obstacles that using kernel-ark directly had, but you you use kernel-ark/os-build as well.

3/ Updating…

@jarkko

3/…that stuff in redhat/ over time and a few other aspects are not covered here, but that is relatively easy.

I maybe should create a script and write a blog post about it.

@kernellogger Finding out what would be a sustainable approach to test kernel patches with a distribution kernel is already a conference presentation. If I was in Linux conference, would not miss that one. It is *rarely* covered topic.

@kernellogger I'm also going to try out your instructions, try to polish them and come up with something :-) I'll share the results when I have something. Compiling kernel is my thing (I guess) ;-)

@jarkko

+1 - but not feeling partly motivated right now to go down that path currently; but who knows what the future might bring.

Side note: sadly the kernel missed hooks to easily and cleanly plug-in a distro-specific packaging infra as that from kernel-ark; its maintainers had the idea to upstream something, but haven't heard anything about it for a while…

@jarkko

Compiling kernel is my thing as well (I guess), but I do it for different reasons: so that others don't have to and thus are hopefully more willing to participate in testing upstream…

https://copr.fedorainfracloud.org/groups/g/kernel-vanilla/coprs/

@kernellogger ;-) https://gist.github.com/jarkkojs/d95bbfaf9b35a18fc9199ab72e51a91c

It's a start.

@jarkko

nice name 😂 😂 😂

Anyway, okay, you convinced me, I will generalize the stuff I use to maintain the SRPM generation for https://fedoraproject.org/wiki/Kernel_Vanilla_Repositories, as the script that I used for that (https://gitlab.com/knurd42/ark-vanilla-scripts/-/blob/main/vk-gitmon.sh) does a lot of what will be needed already (and a lot of stuff that won't be needed, so I will have to rework a few things).

But it will take a few days.

@jarkko

> But it will take a few days.

curl --silent 'https://gitlab.com/knurd42/linux/-/raw/ark-scripts/arkify.sh' | bash

One to three minutes later you'll have a src.rpm you can build with rpmbuild, mock, copr, or koji. Used the later to give it a try:

https://koji.fedoraproject.org/koji/taskinfo?taskID=125255353

@kernellogger I'll give this a try once the week starts.

@kernellogger It started building but it needs tweaking to get packaging to sign binaries with my own MOK key, doesn't it? I.e. this would not pass secure boot?

@jarkko yeah, TBH, I never bothered with the secure boot stuff because I can't really use it for my copr builds anyway (the private key would become public).

Adjusting the stuff that is there for signing in redhat/kernel.spec.template should work, but as @vathpela mentioned: maybe nobody used it and that stuff is broken.

@jarkko @marcan does some foo with SB in Asahi

@kernellogger ok it's not a show stopper! thanks for doing this

@jarkko yw, but it's only a start; next steps: README and a way to update the stuff you import to the branch you want to build, as sooner or later that often will be needed.

@kernellogger OK so I also only now found: https://cki-project.gitlab.io/kernel-ark/

It was informative ;-) Now I know what ark is. So if you put your stuff that Git at some point (e.g. in Gitlab) I can look at secure boot at some point.

My take here is the ability to build distro kernel with secure boot and TPM2 encryption with the goal to get an environment with most popular security features enabled in order to see how they play together.

Normally I just run BuildRoot image in QEMU or similar but this would be a good test to do e.g. before each pull request to Linus (once or twice per release cycle).

@jarkko

rearranged and cleaned a few things up, created a README: https://gitlab.com/knurd42/linux/-/tree/arkify-arkify

The command provided earlier thus now is:

curl --silent 'https://gitlab.com/knurd42/linux/-/raw/arkify-arkify/arkify' | bash

Next step: updating the arkify-infra* branch and then importing the ark infra to target branches that already contain it.

@kernellogger thanks used the bookmark-feature in mastodon that i just remembered :-) i will try this as soon as i have bandwidth