the network card i'm poking at, which has two different CPU clusters with Linux running on it, apparently has eMMC and NVMe drives onboard

also it has docker^W containerd installed by default

the operating system is booted by grub??

@whitequark That somehow seems like the least weird part of the whole setup…

@whitequark it's got systemd installed and running too right?

this is amazing

@arch naturally

@whitequark From my experience with such things I assume it has a full UEFI firmware too.

@whitequark …can it run Doom

somebody here was making a joke about the network card running kubernetes

i regret to inform you that the network card does in fact run kubernetes (there is kubelet in ps)

okay there's "kubelet" in ps. it's definitely running it

@whitequark jokes aside fwiw the way to run P4 on the ASIC is to load the compiled P4 using grpc through a service running on kube https://docs.nvidia.com/doca/sdk/loading-dpl-applications/index.html

@joew whats p4

@whitequark that is brilliant. A kubernetes native network card. Point it at a control plane, taint it, stick your workloads on with the right toleration, tell everyone you're doing edge computing like it's 2013

@whitequark I mean, what should they do? Boot their linux via a tinier linux like IBM pesants? (looking at you, petitboot)

@krono the BMC does run Linux and is booting the main SoC

@whitequark it's more like a cluster of linux boxes which incidentally have a GPU and can communicate over PCIe with a host computer

@confusomu it does not have a GPU

@whitequark 🤣 why grub then?

btw, what kind of bmc is it?

@whitequark O dear it *is* the BMC supermicro uses, too. I hope it is a newer, less...laggy variant.

@krono very dfiferent firmware

@whitequark what the card is it?

@deafbeef bf3 dpu

@whitequark @krono BMC on a network card?
This is giving me stronk "Bigfoot Killer 2100" vibes (NIC from ~2011 that had PowerPC SoC and used U-Boot and Linux, I have one of those)

@whitequark "Kubernetes running on a network card" is an entire new level of cursed.

@whitequark Dante better get inventing some new layers of hell

@whitequark oh then is it running a version of SONiC?

@4censord what is that

@gsuberland @whitequark at my first job I had to get PHP running on a network card

@whitequark it's a weird language to do network packet manipulation in hardware. NICs like bluefield, AMD pensando and even some switches with the right ASIC (tofino) support loading up custom code this way. So you can do things like firewalls, load balancing and etc in hardware rather than in the OS.

@joew ah!

@whitequark at first I was surprised then remembered ISPs/carriers. Due to how modern networking systems work (think 5g), they run a lot of service stuff in containers and sticking it on the NIC is probably giving faster network connectivity by bypassing PCIe to CPU translations.

@lethedata the bf-3 is mostly for AI stuff as far as i know. also like. the CPU on this thing is connected to the NIC (the actual NIC part of NIC) over... maybe PCIe, maybe AMBA? not sure. some sort of bus. but you're definitely going to still have NIC to CPU translations, is my point

@whitequark I wasn't joking! Bluefield Kubernetes is a real thing and can hurt you

@annathyst can confirm, am hurt by this knowledge

@gsuberland @whitequark
Between Dante and OSI, we have 18 already. Do we need even more?

@gsuberland @whitequark
(OK, I included "financial" and "political", which OSI didn't officially include. So sue me.)

@brouhaha @gsuberland that's the judicial layer! above the other two

@whitequark kuberNICes

@whitequark Then especially don’t look at the doca-hbn or doca-snap manifests. Or the hbn setup scripts.

@falk_ terrifying

@whitequark I'm assuming that is a Bluefield DPU, not a Connext-X7 SmartNIC?

I'm expecting my second DGX Spark tomorrow, and of course I don't have the QSFP56 DAC cables yet.

@fazalmajid yes, bf3