I had a weird situation where I had a root shell on a remote Linux server that was experiencing a file system issue. No userspace commands were working except my shell builtins (echo, cd, and so on). I decided to reboot, but the "reboot" command, "init 6", "systemctl reboot", and "shutdown" commands wouldn't work. I couldn't visit the system in person and there was no remote power option.

Just a reminder that when all else is lost you can hard reboot a Linux system remotely via /proc/sys (assuming you happen to have a root shell on the machine):

echo 1 > /proc/sys/kernel/sysrq
echo b > /proc/sysrq-trigger

This is a hard reboot. There's no file system sync, your databases go down hard, and this can really ruin your day. But just remember this little trick if you get painted into a corner.

@hal_pomeranz That’s a new trick to me, thanks! I only knew the keyboard version. From the kernel docs it looks like e.g. echo _sub > /proc/sysrq-trigger should work (sync, remount read-only before reboot), but who knows how much good that would do with a filesystem broken that badly. https://www.kernel.org/doc/html/latest/admin-guide/sysrq.html

@hal_pomeranz I’ve expanded that and remember I can reboot my computer via ‘usb’

# unmount (mount r/o) filesystems if possible
echo u > /proc/sysrq-trigger
# sync
echo s > /proc/sysrq-trigger
# (re)boot
echo b > /proc/sysrq-trigger

@arrjay Seems like you should sync BEFORE trying to remount read-only: “sub” not “usb”

@hal_pomeranz y'know, I'm really not sure what the semantics of "go read only" vs "flush buffers" get to be at this point, I'm mostly hoping and probably get to repair the system afterward anyway :/

@hal_pomeranz I recently did similar things to get install on Librem 5. I could not figure out how to unmount the filesystem, so I nuked it using sysrq, and then wrote image over it over ssh ;-).