1️⃣2️⃣ Here's the 12th post highlighting key new features of the upcoming v258 release of systemd. #systemd258

I believe a fundamental aspect of OS security must be a secure way to return the OS into a well-defined, secure state if a compromise has been identified.

Fending of an attacker is one thing, accepting that it might happen anyway and that you can recover from that in a reasonable way is another. And that's the case not only if you think about "cattle" style installs…

… (i.e. lots of similar systems running the same OS image), but equally on "pet" style installs (i.e. your personal device).

Or to turn this around, a system that doesn't come with a clean, well defined mechanism to reset it fully, erase all unauthenticated data and return it to a vendor image without any local modifications, is highly problematic I believe.

Now you might say: "I can always reinstall my Linux" and start from scratch. And to some point you are right even.

But only to some point. First of all "reinstalling your Linux" is not precisely a trivial operation, and secondly, in today's world it's actually not really going to suffice in many ways. That's because it's not sufficient to erase your HDD an reinitialize it. There's more state to reset. One of them in particular is the TPM: local key material is derived from or protected by a "seed" key stored on the TPM.

If you do not invalidate the old seed key and generate a new one, then all secrets associated with the compromised install (and thus tainted) will remain valid.

And there's more: there are various bits stored in EFI vars NVRAM that need to be reset too (shim keys for example).

systemd for a longer time has had really nice support for resetting your HDD (i.e. securely erase partitions, via the systemd-repart FactoryReset= knob) if a factory reset is requested.

With v258 we considerably extended this work.There's now a well defined way to extend the factory reset logic so that other subsystems can be reset too. And we do ship one plugin for that (which is enabled by default), that requests a TPM reset from the firmware.

If your OS is properly set up for that you can now issue "systemctl start factory-reset.target". This will initiate a reboot, but a special one, where first the firmware will reset the TPM, and then the OS will reset its own state.

And once the system comes back again the system is fully reset.

This currently does not cover EFI var NVRAM reset natively, but there's a clean plugin interface now to do that eventually (two actually: one place where you can plug in such code *before* we reboot, and one place *after* we reboot). [I'd expect that in one of the next releases we'll add another plugin for this, that erases certain EFI vars in NVRAM].

There's also now a Varlink based API to query the current factory reset state (i.e. is a factory reset pending for the next reboot, or are we currently executing one for the current boot, or did we already finish one for the current boot, or is none pending nor executing).

If you want to learn more about the factory reset concepts in systemd, there is a new document for this:

https://github.com/systemd/systemd/blob/main/docs/FACTORY_RESET.md

Oh, and of course: our ParticleOS images have been been updated to expose this kind of thorough factory reset that covers the TPM already. You can simply select "factory reset" in the boot menu, and all non-authenticated data is gone.

And finally: if you run an OS in production and it has no clear story for factory reset, maybe talk to your vendor, and ask them to get back to the drawing board, and fix their mess.

@pid_eins Is the factory reset in the boot menu password-protected?

@anselmschueler which password?

It's supposed to be a mechanism to get a working systemd back if you forgot your disk encryption passwords, so that the system is not bricked for good, you know.

It's the same as it is for your android phone for example, where you can do a factory reset from the boot menu too – without authentication.

So no, we do not ask for authentication: if you managed to get this kind of access to the boot menu to be able to select it, you are god anyway.

@pid_eins The threat model is unclear here. How can you trust the system to really reset itself if you think it's compromised and run this command from a compromised environment?

Later you mention a boot entry, which makes more sense to me, but we would need something to not expose it too much as it would be too easy to select by mistake on boot. You would also likely need two reboots there, one to reset the TPM, and then do the reset?

@siosm you definitely need to reboot, to get back into an unbroken chain of trust. Now you have two ways to ensure this works. First of all you physically request the reset early during boot. That's why we have this in the boot menu. Or you ask for the reset from the compromised running system, and then use some form of attestation to validate that the reboot was genuine. The latter is stuff bigger deployments (i.e. companies which actually do attestation) can do, …

@siosm … but for the local personal device case ("pet") is indeed not really realistic. At least not right now, hence boot loader reset it is for that case, if your factory reset is about trust. (if your factory reset is not about trust, doing factory reset via UI is of course always fine).

@siosm if you request a factory reset from a running system a single reboot should suffice: we ask firmware to reset tpm on reboot, set a flag to reset disk on reboot, then reboot. done.

if you request a factory reset from boot menu, we'll boot twice: first one will set those flags, second one then does the deed like the other case. or in other words: we need some userspace to be able to run the pre-reboot works, and if we are still in boot menu we first need to go there.

@siosm note that the first boot doesn't transition into rootfs however, it's does its stuff from the initrd, and reboots from there, without transitioning. So from the PoV of the firmware it's two reboots in total. From PoV of rootfs userspace it's one. If you understand what I am trying to say here...

@pid_eins Hi Lennart, what do you think about @rfc1036 factory reset proposal?
https://hostux.social/@rfc1036/113205554708605481

@sstagnaro @rfc1036 I love it, of course. It's the exact right thing to do, and if Debian implements that, more power to them!

@pid_eins What prevents an attacker from just hooking into that systemd target then? Why would you trust the command to do what you expect after the system has been compromised?

Either we still trust it or not. You cant have your cake and eat it too.

@agowa338 see my comments about this elsewhere in the thread: https://mastodon.social/@siosm@floss.social/114635361092180697

@pid_eins sounds dangerous considering there are still enough shitty OEM devices that brick when you delete the wrong NVRAM variable..

@agowa338 well, the iea is not erase all efi vars... just the OS-owned subset. the way I envision this is to work is to primarily rely on the namespace of efi vars (you know, that uuid that indicates ownership of the efi var) to filter which ones to keep and which ones to erase.

@pid_eins @siosm

Lol, so now we should boot the compromised system again? You know that normal security practice is to wipe it before it lays eggs right?

Also if just rebooting was enough then we wouldn't literally need to do the factory reset...

@agowa338 @siosm not sure what you are going on about, but the way the trust chain concept works on modern computers is that you boot up in a trusted state, and then chain everything else from there. Hence of course, you reboot to reset the state, to get your trust chain into a clean state again?

Note that all of systemd's factory reset work actually runs from the initrd, i.e. under the assumption of an UKI world in a fully vendor signed part of the OS with only minimal input from elsewhere.

@pid_eins I know of at least one case where deleting the Windows Bootloader stuff caused one cheap asus(?) notebook to brik and not even power on anymore.

I hope you've thought about this carefully this is a deep rabbit hole...

@agowa338 Hmm? the logic is supposed to reset stuff relevant to *our* OS, i.e. the OS we actually want to reset. Some windows efi var does *not* belong to that set, and is clearly recognizable as that, precisely because it uses a different namespace uuid.

@pid_eins @siosm

But why should I need to break the trustchain as an attacker wanting to exfiltrate secrets? I can do most of that as regular user. Therefore we never turn compromised systems back on once we know they're compromised.
That just gives attackers opportunities to have their scripts ran. Also there is still the risk of exploits.

It is just bad security practice to boot from a compromised disk. At most you'd attach it to an air gapped offline system to do analysis and such...

@agowa338 @siosm sure, if you can afford to dump all hw on the trash and never use them again, if you have the suspicion that they got compromised, knock yourself out. Not cool for the environment, kinda wasteful, but whatever.

I am pretty sure most people outside your bubble probably prefer a reasonable mechanism to maybe not waste all those resources, and get a clear and reasonably secure way to get the systems back to work.

@pid_eins That was an example.
Basically what I was saying is "hic sunt dracones" ("Here be dragons") if that makes more sense to you.

@agowa338 frankly, to a large degree that's cargo culted paranoia I am sure. At this point various components of the boot set and delete efi vars they own already (shim, sd-boot, systemd otherwise), and the reports that this bricked computers have not really happened.

@pid_eins Thanks for the clarifications. This matches what I had in mind and this is looking great.

@agowa338 @siosm

also, it's actually mostly fine to boot from a compromised disk as long as every resource involved is properly authenticated. Of course, the less you boot the better, but again when resetting the disks like this we do this from the initrd, i.e. from the signed, vendor-supplied UKI in a short-lived environment, not from the rootfs that might be user (and thus attacker) controlled.

But I think ultimately we can just agree to disagree on the security model, no?

@pid_eins @siosm Well only high security context dumps the hw into the trash. Others just wipe the HDD, reset the TPM, (cross fingers there wasn't a UEFI exploit granting attackers persistance in hardware), maybe do full factory reset of the UEFI, and reinstall the os.

@agowa338 @siosm oh, hey, you are describing exactly what systemd is providing you with now.

@pid_eins Not really, if you haven't had to deal with these issues yet, go into the next tech store and buy 2-3 notebooks of the cheapest consumer crap on display and you'll see.

Stuff like this was the reason for e.g. back then the windows 8 upgrades to bluescreen. Vendors cheaped out on actually implementing the UEFI spec fully and took shortcuts...

@agowa338 You think I am not sitting at the firehose of systemd + core OS bug reports already? But you are or what?

@pid_eins @siosm
Only that I don't trust systemd at that point. Except you mean the systemd on the read-only install media...

@agowa338 @siosm it's the systemd embedded in the UKI, i.e. fully signed by the OS vendor, immutable, not modifiable by an attacker, without network access and so on.

I mean, sure you can decide not to trust SecureBoot and stuff at all, sure. But if you are at that level of paranoia, then you cannot trust computers at all, and might as well become a sheep farmer?

@pid_eins @siosm

to be frank I just don't want to think about how trustworthy it is and just dump the entire state into the trash.

You're making things way more complicated than they should have to be.
Now one actually needs to check if the chain of trust is still valid or if the attacker was able to just use e.g. some exploit to disable secure boot (or it may not even have been enabled to begin with) or re-signed parts of the trustchain using and so on...

@agowa338 @siosm

we are dumping the *state* on the trash. But a signed UKI isn't precisely "state". It's firmware-authenticated immutable vendor code.

But dunno, I think we should end this here. You have your security model, I have mine, let's just agree to disagree on it.

@pid_eins
I don't know the full extend of what you're planing. That's why I said "Here be dragons".
I wouldn't want to be touching this topic so I wish you luck...

@agowa338 You think I haven't been touching this topic since a long time already?

@pid_eins @siosm

Oh, I think I see where the misunderstanding lays. How does your UKI get signed? I always forget that you develop systemd with a focus on distros like Fedora and such.

On ArchLinux the keys to sign it generally also lay on the system itself it gets signed locally...

@agowa338 @siosm sure if you make your UKI signing keys accessible to an attacker then of course there's no way to recover. Maybe don't do that then...

@pid_eins @siosm
Yea, that's why clearly defining the thread vector as someone in another post asked you about is so important.

I think I by now understand that you're targeting this as a feature for system integrators selling devices and not to everyone that installed Linux on some random device.

Because then the question of how to get your kernel signed automatically after updating or after you recompiled it locally doesn't even arise to begin with...

@agowa338 @siosm if you compile your own kernels you are your own OS vendor. In that case you better take care of your signing keys.

But this is not how most Linux distros or big installs are set up: they run kernels put together and signed on build systems, and those are distinct from the systems they are run on, and the signing keys are not available there.

@pid_eins this sounds like a fantastic addition.

Out of curiosity (more lack of knowledge): does this means that the OS must maintain outdated versions of the packages installed so it can recover back to that state?

I'm thinking on a scenario where one package update bricked the system and a factory reset could help.

@jlhertel that's a different issue, i.e. automatic rollback. systemd got you covered there too (i.e. keep multiple UKIs around, sd-boot will boot newest, but count it, and if OS never reports back that it worked, will mark it bad, pick older UKI next time).

@pid_eins @siosm
kinda true, however why would I need a dedicated build system for building something tailored to a single notebook. Guess that systemd feature is just simply not for me then.

I'm kinda looking forward to seeing how this feature gets used in the wild (if at all) and if it is actually going to be provided in a secure way that doesn't just leave it open to be bypassed or compromised by attackers (e.g. them resigning and enrolling their keys or an unprotected script or something)

@agowa338 @pid_eins @siosm that's really not a problem these days, you can build such images with mkosi on build.opensuse.org and the signing key never leaves the hsm in the datacenter

@bluca @pid_eins @siosm
Ok, but I have 192 cpu cores here right next to me. Also that may be something for distros to do I think it may be a bit unreasonable to setup something like mkosi for everyone that wants to have a slightly differently built kernel.

Also I don't see how using someone elses keys should be more secure than using ones that only work for a single device and are not available anywhere else...

@agowa338 @pid_eins @siosm you can't at the same time be all worried about attackers getting a foothold on the machine and doing nasty things that invalidate the factory reset feature described here, and also prioritize using those 192 cores over securing the keys. Pick one.

@agowa338 @bluca @siosm you can store the UKI signing key on any pkcs11 device btw, i.e. a yubikey if you like. An attacker who takes over your system shouldn't be able to get to that, if you lock it with pin + fingerprint.

@bluca @pid_eins @siosm

You misunderstood. I secure the key by not having it available in a datacenter. Also the Image it signes will only boot on a single computer so the interest of anyone stealing it area also quite slim.

On ArchLinux every user typically generates their own keys and enrolls these self generated keys within their devices.

Also one can use a HSM if they want to but because of the limited scope it isn't really required.

As I said above entirely different use case...

@agowa338 @pid_eins @siosm it's not a different use case, it's simply that the threat model you are describing makes no sense. You spent a dozen posts above going against a perceived flaw in a feature because of local attacks, and now you are claiming that it's fine to leave private keys in plain text locally. This is completely contradictory, if you are worried about local attacks, the first thing to do is to never, ever have local signing keys. Average users are not security experts.

@bluca @pid_eins @siosm

In earlier posts I misunderstood the new feature. It is obviously way more meaningful when the intended audience is distro that ship a single static and pre-signed kernel or environments with e.g. a golden image.
It also was already outlined above that I'm therefore not within the target audience and that a distro where the final UKI and kernel are created on each individual machine with a different set of parameters also was never one of the intended scenarios either.

@bluca @pid_eins @siosm
In my case these keys are solely protected by the fact that you have to have booted at least once to have the TPM release the key to decrypt the drive.

The only thread vector that is still valid considering the actual intended audience is how one would ensure against an attacker with root access permissions enrolling their own signing keys, patching the kernel, and re-signing it. But that probably only applies to expert users configurations only anyway...

@agowa338 @pid_eins @siosm the most common threat model for normal users is malware breaking out of browsers and getting local execution privileges. At which point the next step is finding way to persist. Leaving uefi keys available at runtime presents an extremely juicy target as it allows to then attack the firmware, and then that's it. So unless you are never using a browser, or an email client, you really, really, really should not leave secure boot signing keys lying around in plain text.

@bluca @pid_eins @siosm
"for normal users" ArchLinux isn't necessarily for normal users.

What is worse in having the uefi signing keys compared to being able to place your malicious script within the build path of an UKI and thereby getting pulled in and (because you don't know about it) signed by you after entering the pin?
None. Both actions require the attacker having root permissions on your system...

@agowa338 @pid_eins @siosm any such script would run long after ESB(), in userspace, so magnitude is a lot lower - still not great of course, so yes don't build locally, use a build system that isolates the build, uses an HSM and produces a SBOM

@bluca @pid_eins @siosm

Furthermore at that point an attacker also could just place their own uefi signing key within /boot/loader/keys/auto and systemd-boot would provision them for them...

@bluca @pid_eins @siosm

So what you're saying is being able to control which files get included in the initrd, as well as which file is used as kernel, and therefore ultimately end up being signed isn't giving an attacker full control?

Sure....

@agowa338 @pid_eins @siosm no, I'm saying that loading arbitrary code before ESB is worse

@agowa338 @pid_eins @siosm no, that only happens in VMs, and only if in setup mode

@bluca @pid_eins @siosm
And how is that not also loading arbitrary code before ESB when the attacker basically controls which file gets signed?

I don't see a difference between attacker controlled input and an attacker controlling the input and the signing keys. It's literally the same in this case.

The only reason the UKI is signed at all is because it isn't protected by LUKS2+TPM and could be tampered with offline.

@agowa338 @pid_eins @siosm if you really can't tell the difference between being able to stash away private keys for any arbitrary usage at any time for any purpose, and having to intercept a build process at the exactly right time in exactly the right place and hope it doesn't break anything, then I'm afraid we are just not speaking the same language