I've had multiple people at XDC 2023 tell me that they don't contribute to the Linux kernel due to community and process problems.

This is an event where most of the attendees are already seasoned FOSS folks. We're talking people contributing all across the Linux graphics stack and more. Many of them employees of well known corporations. And yet the kernel makes them nope out.

If you want people to contribute to your project willingly, and actually want to help out with hard problems, especially if they are not getting explicitly paid to do so, you need a welcoming environment with good tooling and processes to make it as friction-free as possible.

This is why when I started Asahi, the first thing I did was spend a ton of time on tooling, so people would have fun contributing to our project, with super-short test cycles and advanced debug and reverse engineering tools. And then of course, actually sending me code is just opening a PR. I don't even care about target branches, I'll just cherry pick whatever you send me into the right place and thank you for your contribution. At XDC this year, I spent a lot of my time discussing CI so next up I can help put together good kernel+Mesa CI for Apple platforms that makes our developers happy, not sad.

The Linux kernel is the polar opposite of that. With terrible processes, a culture of gatekeeping, total lack of common standards, and a good dose of toxic maintainers, sending stuff to Linux is a huge chore and a drain on your energy. And when you create that kind of environment, you are scaring away huge numbers of contributors, and burning out the rest so that they have no incentive to go the extra mile or solve hard problems, and just optimize for the least amount of friction.

I'm already at the point where I just go for the simplest/least controversial workable solution where possible, because I'm already having trouble maintaining motivation to upstream everything we have queued, never mind add on extra work on top. And I'm getting paid to do this (by all my wonderful sponsors and patrons). If I weren't I'd have thrown the towel a long time ago.

Once our CI is brought up, I'm going to move soc/apple upstream development to a forge. And I hope more kernel trees start doing this (a few DRM drivers already are). Maybe with some push for sanity from a few bottom maintainers we can slowly fix this trainwreck. Because this has to change for Linux to remain a viable kernel that runs everywhere for many years into the future.

@marcan Do you see this as a problem with Linus in particular?
I get if you don't reply, it's a bit of a question!

@greenpete It's a problem with everyone. Linus doesn't care, if anything. But there is room for subsystem maintainers to push for change, and it largely isn't happening either.

@marcan I just think, like you, it's sad to have this going on.

@marcan I don't understand this well enough, but I feel that what you are describing is very bad for Linux as a whole, am I right?

@greenpete Yes, this is bad, a lot of people agree that it's bad, but nothing seems to ever happen :(

@greenpete @marcan it kinda used to be (check his debconf Q&A for the before), but he's worked a lot on improving himself and growing as a human being, so not anymore. He isn't abrasive anymore, just blunt and very direct. I don't have experience contributing to the Linux kernel, but I've heard that the mailing lists make it hard for people who aren't used to them, that the procedures are very tedious, and that some maintainers are... Not nice.

@Alonely0 @greenpete That's more on the personal side yes, but I haven't seen any indication that Linus has changed his mind about kernel processes or pushed for much change in the kernel in general, unfortunately.

@Alonely0 @marcan So who is in a position to change this, Linus?

@marcan vaguely related thought (and open question, if anyone wants to chime in) but what does someone do if they’re curious about contributing to these kinds of projects but don’t have any kind of foothold in OS dev type stuff? I’ve always thought it sounded neat but i don’t know what process of thoughts would lead me to “i need to fix/change/improve this OS thing”

@chrisisgr8
idk, write your own os?
i've been genuinely doing that for the past few months on the side and it's a lot of fun
though i don't think i'd be anywhere near good enough to contribute to the linux kernel
@marcan

@marcan I don't think the _entire_ process is wrong. Obviously the various subsystem maintainers are comfortable with their workflow.

I do, however, agree the workflow is unfriendly for new developers. There's little documentation on how to get started, patches via email are unfamiliar to many, and maintainers are picky in what they accept -- to put it nicely.

An email workflow can still work if you don't blame contributors and accept patches liberally, e.g. as an email PR.

@marcan Essentially I think what we need are friendly interfaces for contributors, which for example could be implemented as a software forge. And friendly contributors willing to help devs get their contributions upstreamed.

And thanks to git being a decentralized VCS, that can be mostly detached from the workflows in the background.

@lschuermann No, email is a major part of the problem. It multiplies interpersonal friction by making it a mentally much more significant step to send a new revision, as opposed to pushing new commits to a forge. And with zero introspectability and metadata, it makes managing open PRs, searching, organizing, etc. a nightmare. I had to write my own goddamn email to status dashboard bridge server to keep track of even just the things I'm sending.

You cannot fix this by wrapping more layers of nonsense around email. It needs to go away. The rest of the world has moved on.

@marcan @lschuermann Wrapping more layers around email won't work, but maybe we can wrap a layer of email around a forge, so the browser-allergic old-schoolers can keep using mutt, but the canonical info is in a forge? (GitLab etc already have email notifications, and replies even appear as comments, but that's not enough)

@marcan curious what forge you’re considering…

@ross gitlab.freedesktop.org, since a few kernel trees are already on there, and ours has been manually synced there for a while too.

No particular preference for GitLab, but anything is better than email and fd.o is already an established community of like-minded folks :).

@marcan @ross Please don't help fd.o become the monoculture they aspire to be. It is important to keep the lower levels organizationally separate from all the desktop crap.

@marcan If you don't mind me asking, could elaborate a little on what you mean by 'toxic maintainers' and 'terrible processes' in particular?

(I work in the embedded Linux space and I'd love to do some work on the kernel, so it feels important to understand this right now...)

@bars The toxic maintainers... well, I think that basically speaks for itself. Unfortunately the Linux kernel does not have any process in place to discourage power-seeking narcissistic and abusive personalities from reaching positions of power over other contributors, and once that happens everyone under them suffers. It's simply not an environment that values (never mind encourages or requires) friendliness or cordiality. Not all maintainers are like that, but a significant enough fraction are to be a major problem.

As for the processes, the email-based workflow is archaic, painful, and exacerbates the people problems.

@marcan another project like this is qemu, i have a few random useful things i made that i absolutely cannot be bothered to even try to contribute given other people’s experiences

@harmonia Yeah, unfortunately a few kernel-adjacent projects have inherited and continue to use the same broken process for some reason... :(

@marcan

I've heard this from people internal to AMD on the kernel contributors team. Our AMDGPU team is apparently pretty happy, but the people on the kernel teams spend a ton of time politicsing, to the point that it drives skilled engineers away from those positions.

@aer FWIW at least one AMD employee doing kernel stuff is part of the problem crowd :P

(He's the reason our GPU driver upstreaming has been derailed/delayed for half a year ish, because he thinks instead of fixing poorly designed common code originating from amdgpu to be usable by more drivers, other drivers should just work and be designed like amdgpu, even when that is outright impossible. And he's rude and demeaning about it all on top.)

@marcan @lschuermann EMail will stay unless there is another way of maintaining a project without a webbrowser or some janky platform specific cli tool.

Strictly from the PoV of a maintainer, reviewing patches in my mail software is an order of magnitude less frustrating than Github or Gitlab. I won't claim it's good, I won't claim it's easy to get into. But it unironically is the better workflow for me and a fair share of other people.

And being kind arguably fixes the main problem with mail.

@lhp @lschuermann The rest of the FOSS world is doing much better with web browsers and janky platform specific cli tools (for the minority that refuses to use web browsers) than the trainwreck that is email. Email has to go.

Strictly from the PoV of a maintainer, reviewing patches in my mail software is an order of magnitude less frustrating than Github or Gitlab.

That just means you're part of the minority that is entrenched in that flow, and dead set on keeping the status quo, against the well-being and future of the kernel. This opinion is not shared by the vast majority of the FOSS community today; that much is plainly clearly evidenced by the vast dominance of forges across the rest of the ecosystem. The Linux kernel and a few adjacent projects are practically the only notable projects that continue to use email today. The opinions of a minority should not dictate process for everyone, and said minority retaining control over a single, but major FOSS project with worldwide impact is a problem that needs to be fixed, not a sustainable state of affairs or something to celebrate.

And being kind arguably fixes the main problem with mail.

It only makes it less hostile. It still sucks.

My job, as a project lead and maintainer, is not to build and use the processes that I like. It is to build and use the processes that maximize the happiness of my downstream developers. And there is no planet where that process would be email.

@marcan @lschuermann Now I am not claiming the kernel is doing a good job with their lists btw.

Also I dont exactly care how people submit changes. Whether via mail or some slow web interface, their choice. All I care is that I can review and merge them on my own terms.

@marcan @bars One of the worst things about working in the kernel — one of the most toxic parts — is the constant stream of nastiness toward our community that comes from outside.

The kernel community is far from perfect; we have a lot of problems and we have been actively working for years (decades) to improve on them.

We are, nonetheless, a project that manages to incorporate nearly 100,000 commits per year, from over 5,000 developers, into a single code base while maintaining a level of quality that — while also certainly in need of improvement — is good enough for deployment into billions of devices.

As for the use of email...email is painful and broken, but we have found nothing better that will work at the scale we need. See https://lwn.net/Articles/702177/ from a few years back. For all its faults, email is distributed, non-proprietary, scriptable, and gives everybody the freedom to choose their tools; it is a highly inclusive solution in a way that proprietary web forges (for example) are not. Someday we'll find something better and move on with a cry of joy, but that day has not come.

Rather than crapping on the kernel community from afar, why not work with us to try to make things better?

@ska @ross

Please don't help fd.o become the monoculture they aspire to be.

Citation needed.

It is important to keep the lower levels organizationally separate from all the desktop crap.

Why? It's just a forge. I can move to any other GitLab instance the next day. fd.o happen to host a useful service for the community that is already used by some kernel subtrees and has a significant overlap in developer base, plus a generally healthy community (much healthier than the kernel). Why wouldn't I want to take advantage of that?

@marcan what is a forge here?

@bas GitHub, GitLab, Gitea, etc.

@corbet @marcan

I think it is reasonable to object to "from outside". marcan is literally a seasoned kernel maintainer. I don't think he needs to be explained what the scale of kernel dev is.

This is a meaningless non-defense of the status quo.

For example, what if there was a forge using open software for each subsystem? You could then integrate whatever tooling you'd like with it.

Bram (rip) made an email based workflow on top of GitHub for vim e.g.. It can be done.

@VulpineAmethyst @marcan @ross I'm not even talking about systemd here; more about stuff like D-Bus and PolicyKit. Don't your desktop environments use these? (Good for them if they don't; I'm curious about their inter-app communication model.)

As for being a standards-setting organization... yeah, that's exactly the problem. You know what I think about fd.o's standards, by experience. From my pov, they make development for UNIX platforms harder, not easier. I'll never get over XDG_RUNTIME_DIR being unusable to host user services.

Who else is doing that work? I don't know, X11 existed before fd.o came along, and wasn't so full of crap. Maybe I'm just a grumpy dinosaur, or blinded by privilege, but their added value is dubious to me. I stopped using GUIs when desktops became popular in part because I couldn't wrap my mind around all the bloat and insanity.

And this has nothing to do with the community. The fd.o community may be perfectly healthy and lovely, but I still don't think they're going in the right direction.

It may be "just a forge", but association is important. If I see the fd.o forge hosting projects I respect, I'm going to be pretty confused - but if it works for @marcan, well, who am I to judge.

(I'm too tired and not interested enough to go to war; this is not an invitation for people to tell me how wonderful fd.o is and how it will save FOSS. I may be wrong; I may or may not realize it at some point in the future; I don't need people to explain it to me.)

@corbet @bars @marcan I think Hector can be a bit strident in his messaging, but I don't think he's *wrong*. From my perspective (Fedora QA) the kernel is one of the things that makes me sigh the most. Just about every other project I deal with - which is, like, nearly all of them - makes it easier to file and find bug reports, follow the progress of debugging and fixing them, and submit and follow pull requests (or whatever the system they use call it).

@corbet @bars @marcan real world example: https://bugzilla.redhat.com/show_bug.cgi?id=2113005 . trying to keep up with what the heck is going on with that bug upstream requires painful searching through awful mailing list archives for patch series with cryptic titles (I actually just wind up searching for maintainer names and wading through pages of unrelated gunk). this could and should be easier.

@corbet @bars @marcan there are F/OSS forges: there's the F/OSS core of gitlab, there is Gitea, and there is pagure. None of these is perfect, but they're all there and they work. I'm sure either gitea or pagure would welcome (this is a *massive understatement*) some kernel folks showing up to help maintain and improve them.

@leftpaddotpy @marcan How is a forge for each subsystem ever going to work? This is one program we're dealing with, and an awful lot of work is not contained to a single subsystem.

@adamw @bars @marcan Bug tracking is clearly a place where the kernel project falls down badly, agreed. We finally got regression tracking funded, but that's just barely the beginning of the problem.

For bug tracking, one aspect of the problem is a simple unwillingness on the part of many maintainers to bother with a bug tracker. That does not help at all.

The other part is something I'm going to poke people at the LF shindig about next week. Almost everybody who works on the kernel is paid to do it, but there are many areas that no company thinks it needs to worry about funding. Of the 5,000 developers who work on the kernel each year, not a single one of them is tasked with documentation — my own pet peeve. But (almost) nobody is paid to work on tools, and it hurts us in all kinds of ways, including bug tracking.

@corbet @bars @marcan

email is distributed, non-proprietary, scriptable, and gives everybody the freedom to choose their tools; it is a highly inclusive solution in a way that proprietary web forges (for example) are not.

Could a federated git forge help with that? People would have the freedom to choose their tools while still going to a simpler and more modern thing. Gitea seems to be working on that for a while, but I don’t think they have an expected release date. Pretty sure gitlab was working on it too.

@corbet @adamw @bars @marcan That sounds like something the @sovtechfund would cover, if someone applied for it.

@corbet "Of the 5,000 developers who work on the kernel each year, not a single one of them is tasked with documentation"

Do you mean full time? All Oracle Linux kernel devs have an explicit mandate to work on upstream docs. I'm by no means a frequent contributor, but I've submitted a few patches; most of them got rather lukewarm responses at best. I think this actually ties in with what @marcan says: it's draining and not very rewarding when you have to fight for every attempted change.

@marcan I increasingly feel that Linux will need an EGCS-style moment to effectively change. And without changing this, I can't see how it remains relevant.

@marcan To expand on this a bit --

I think there is a powerful and growing pressure for a *sharp* culture shift / reboot of Linux. I think a culture-shift-centric fork or competitor will eventually unstick this.

And I do mean culture-centric:
- Contribution culture (as pointed out in OP)
- Safety & security culture
- Testing & correctness culture
- Tools & infrastructure culture
- ...

It's honestly remarkable to me that Linux holds on given the deficits it faces on *every* dimension here.

@fyrkbury @lhp @lschuermann Upstream already doesn't have a "mail interface". Changes are pushed to upstream Linux via pull requests already (using mail notifications, but the Git protocol for changes). I'm a maintainer and I do that. asahi/soc changes already flow to soc and then to Linus directly via the Git protocol.

Maintainers already know mail doesn't scale. It's just all the downstream contributors that get screwed and are forced to send patches via email.

@marcan @lschuermann Ok, typical forge workflow: Receive notification, click on link, wait for firefox to startup (~2 seconds), wait for site to load (~2 seconds), wait for FF to calm down (~2-5 seconds depending on planetary alignment I guess), in case of gitlab wait until auto-scroll decides to do its job (~5-10 seconds, depending on size of discussion). All while listening to the lovely cacophony of the laptop fans spinning up 0 to 100.

Sorry, but in what world is that good UX/UI?

@lhp @lschuermann 🤦‍♂️​

Look, I'm not here to sell you on forges. There's a reason the rest of us use them. The benefits more than outweigh the downsides. If you can't see that, that's on you.

Also your browser should already be running, and you need a new laptop if pulling up GitLab is causing the fans to spin up to 100%. But we all know that's a bullshit argument anyway, because kernel developers already need beefy machines to build the kernel in any reasonable amount of time.

Edit: Took 3 seconds from clicking a GitLab notification to a fully loaded page, on WiFi, from another continent to the server, laptop fans did not make a peep. So yeah, take that BS elsewhere. And if 3 seconds is a signfiicant slowdown to your patch reviews, you suck at reviewing patches, because it's supposed to take way longer than 3 seconds to review a patch and type out a reasonable reply. Heck, patch reviewers not paying attention and replying with nonsense comments is another gripe I have with kernel people.

@vegard @corbet @marcan

Maybe that "lukewarm response" is because the maintainer is already totally drained for various reasons.

No idea if that's the case, but I guess I would be if I was put in Jon's position, because (1) the work on docs is even less rewarding than other maintainer positions (2) it's a huge amount of work without any financial backing for the maintainer (3) a big rework is needed since years to bring some order into things, but there are no resources in sight for that.

@marcan @ross @ska That could be easier if federation between different gitlab/gitea/whatever instances was finished soon. Also would make it possible for people to self host their stuff without sacrificing possible contributions.

I don’t really see the problem in git send-mail meanwhile tho?

Also I’m not really a fan of fd.o and it’s limitations on new accounts. Sure, there was a spam problem but then again if your account needs to be approved in order to open a pull request it’s probably faster to send via email anyway (federation could potentially fix this too :p)

There also seems to be a culture in there to praise wayland without taking its problems into account, while many projects gatekeep support depending on your hardware (sure, no one is obligated to support anything, but that kind of attitude does nothing but drive future contributors away - at least be kind or say that contributions from other people would be welcome if you can’t do it yourself)

@luana @ross @ska The "praise wayland" problem is a lie perpetrated by people who want Xorg to continue to exist, be maintained, and work well, while putting zero effort into it themselves. The reality is that Xorg is unmaintained, dead, and nobody wants to work on it. The developers acknowledge that and move on, whether Wayland is a perfect replacement already or not, because it's the only way forward.

The people complaining about this aren't doing any work to resurrect Xorg, they just want their legacy system to continue to survive while someone else magically takes care of the 30 years of technical debt and fundamental problems it has which is why it's dying in the first place. Sorry, I have no spoons left for people complaining about Wayland and whining that we should "support" Xorg (=fix all of Xorg's problems because nobody else will, even though it's not our job).

The new account limitations are unfortunate and hopefully will be fixed at some point. fd.o have some infra issues, but hopefully they'll get better funding and improve that.

I don’t really see the problem in git send-mail meanwhile tho?

Then you haven't had to deal with kernel development to any significant extent. It sucks. Big time. It makes contributing to the kernel 10 times more annoying than any other project. And it exacerbates the major people problems the kernel has to the point it is insufferable.

That could be easier if federation between different gitlab/gitea/whatever instances was finished soon. Also would make it possible for people to self host their stuff without sacrificing possible contributions.

Federation is overrated. The Linux kernel patch review model is already fast-paced. If Linux used a forge and had to migrate and lose in-progress reviews, the transient impact would be miniscule compared to the ongoing pain and suffering caused by not using a forge. Email makes reviews and PRs basically ephemeral since they are forgotten days after they are posted. Even if you wiped the database on GitLab every 3 months, it'd still be better than email.

@marcan with CI Pipelines beeing the industry norm, I really wonder how the Linux kernel achieves to ship so few regressions without a really good test harness.

Maybe, they need to be hostile, to keep the current ship afloat. More developer == more changes == less time to find all bugs by starring ad patches.

@hikhvar BS. Upstream Linux breaks our platform all the time, from Bluetooth issues to core memory management getting borked and making everything unbootable on our machines to breaking locking across the board to borking our IOMMUs entirely. For fuck's sake, I found out that atomics were broken on ARM64 for two years when working on Asahi. And I've had to panic backport changes to our branch twice already because there were regressions in stable Linux releases on the "this may corrupt your filesystem" level. And those affected all platforms, not just ours.

There's no magic quality level that Linux achieves and nothing else does. The only reason stuff keeps working is there are a lot of users and when things break, which they do all the time, they get fixed relatively quickly.

@kernellogger @corbet @marcan Just to be clear, I wasn't singling out Jon here -- it's a complex issue for sure. I think drained contributors and drained maintainers feed back into each other's drain.

What rework/order do you mean? I'm interested in documentation in general. I've tried cleaning up the documentation front page before and it went nowhere. I'd be happy to take a stab at fixing things but what's the maximum ROI thing I can do that has a chance of actually making a difference?

@vegard @corbet @marcan

"[…] feed back into each […]" Yup… 🥴

"What rework[…]" I got the feeling that Jon would like to do a lot of things if he'd time, but maybe I'm wrong with that.

If I'd be in his shoes, I feel like there is a massiv amount of work waiting to be done, like

- cleaning up old, obsolete stuff
- remove dups (Documentation/process/submitting-patches.rst vs. Documentation/process/[1-8].*.rst)
- separation of content by target audience (users, userland devs, kernel devs, …)

@lhp @marcan @lschuermann what *is* your mail client that it makes viewing code less janky than tools explicitly designed to view code? Do you have syntax highlighting somehow? I feel like I’m missing a trick here.

@gadgetoid @lhp @marcan @lschuermann I'm over here wondering why people won't use a web browser. +1 what email client allows high quality diffs? Would be great for use cases outside code as well.

@binarypie @gadgetoid @marcan @lschuermann I use aerc. Here is a mockup of how I review patches with it.

Why use it over a browser? Faster (both usage and startup), familiar editor and theme, can be used keyboard-only, easier to make small changes before merging/applying, responds better to being resized meaning I can pull other stuff up side-by-side, my laptop fan does not spin up, etc..

@lhp @binarypie @gadgetoid @lschuermann So no code syntax highlighting within the diff, no sub-line change highlighting, no easy permalinks to comments to share stuff with people, no way to see the full discussion in one page once threaded conversations start happening, no way to embed other code snippets with proper highlighting, a single not-perfectly-configured email client will mangle the formatting for every reply downstream, ...

Seriously, if your problem is your laptop fan spins up, get a better laptop. This workflow is inane and horrible.

@corbet @bars

One of the worst things about working in the kernel — one of the most toxic parts — is the constant stream of nastiness toward our community that comes from outside.

You say that as if I'm not part of the kernel community. I'm in MAINTAINERS too, you know? I think I'm allowed to criticize the communities I'm part of and have to deal with on a regular basis.

The kernel community is far from perfect; we have a lot of problems and we have been actively working for years (decades) to improve on them.

And yet somehow much of the rest of the FOSS world is managing to do much better on many fronts, by simply making choices like... not tolerating abusers and toxic people, and using modern tooling instead of increasingly trying to work around the failings of an obsolete process.

We are, nonetheless, a project that manages to incorporate nearly 100,000 commits per year, from over 5,000 developers, into a single code base while maintaining a level of quality that — while also certainly in need of improvement — is good enough for deployment into billions of devices.

See my latest top-level toot. Linux kernel code quality is average for a large FOSS project and regressions happen all the time. The only reason Linux looks like it's a high quality project is that it has so many users all the regressions and bugs get noticed quickly. Not because the underlying dev process is any good, never mind better than average.

For all its faults, email is distributed, non-proprietary, scriptable, and gives everybody the freedom to choose their tools

All the tooling people are building to work around email's failings, like lore.kernel.org or patchwork or indeed the mailing lists themselves, are not distributed. If you find yourself making centralized tools to work around the failings of a "distributed" system... maybe you shouldn't be using a distributed system in the first place.

As for scriptability, that's a joke right? Email's lack of structure makes it impossible to reliably script. I had to write my own damn patch tracker to do things that are built in to every forge, and it's a fragile mess of regexes trying to guess the intent of email replies, because there is no way to do better when it's literally a pile of ASCII with no structure and no standards.

it is a highly inclusive solution in a way that proprietary web forges (for example) are not.

Please tell me you are kidding. Archaic processes like this are a highly exclusive solution because they have a high barrier to entry, a higher mental cost to work with, and are seemingly deliberately distancing themselves from workfows the vast majority of the rest of the FOSS community are already used to.

Edit addendum: And I'm not advocating for proprietary web forges. Plenty of FOSS solutions exist.

Rather than crapping on the kernel community from afar, why not work with us to try to make things better?

I am. That's why I'm moving my soc/apple subtree to a forge soon, once I get CI infra going (which will be my excuse for it). And we'll probably end up with drivers/gpu/drm/asahi there too, which won't be the first drm driver doing upstream dev in a forge either (msm already is). I'm hoping more people follow and we can slowly bring sanity to this mess.

@corbet @leftpaddotpy Same way asahi/mesa is already a downstream sub-forge of mesa/mesa and it's working great for us (and solving problems with everyone working directly upstream). Cross- subsystem work goes directly upstream. Intra-subsystem work happens downstream and gets merged up periodically. Which... coincidentally happens to already be exactly the way Linux already does things, minus the archaic email process.

Git and Git forges already have all the workflows we need and Linux is already effectively using. Email has nothing to do with any of that. You can ditch the broken email workflow and keep the same structure.

@marcan I think we are on the same site. I want to have CI Pipelines for the kernel as well.

As a consultant, the Linux kernel was often an argument for some people to not introduce pipelines in their codebase. "The kernel doesn't have some either and is a very stable code base. Like ours."

Thank you for all these counter examples!

@hikhvar To be fair, testing kernels is hard since much of the time you're testing drivers and that means bespoke hardware setups. Building some kind of Linux-global monolithic CI would be a massive undertaking, and probably end up being a trainwreck (mesa already has problems here).

But Linux definitely needs hardware-agnostic CI (VMs), since that's the easy part and would help catch a lot of really bad core issues. And for hardware, we need more easy-to-deploy CI platforms so individual platform stakeholders can do their own thing and catch problems without trying to merge it into one monolith (and that is exactly what I'm trying to add to with our little corner of kernel CI).

Centralized CI will probably never work for Linux, and Mesa has taught us trying to do blocking CI at scale will end in tears, but distributed CI works, and we need a lot more of that.

@marcan and also, thank you very much for improving the kernel and show how to do better.

Also sharing your insights with us. Learned a lot!

@corbet @adamw @bars

Almost everybody who works on the kernel is paid to do it

Have you considered that this is a disease symptom, and the direct cause of lack of work on

areas that no company thinks it needs to worry about funding

?

Most other FOSS projects have either mostly unpaid or a healthy balance of paid and unpaid work. That Linux is mostly paid work is an outlier.

But indeed, when most of the work is paid, people end up working only on things they get paid to do. For companies that excludes quality-of-life work or reducing technical debt, because they don't care about those things. For me, that means focusing increasingly more and more on just getting stuff upstream for my platform, not the overall picture, because I'm losing motivation to do good for the sake of good.

After being in this community for a while, it has become clear that the Linux kernel's hostility and notoriously bad process is simply driving away unpaid contributors, and with them, motivation to work on things that don't get rewarded with money. And that's how you end up nobody wanting to work on things like documentation. Because we just don't have the spoons left for it.

And when someone tries anyway and gets shouted at by a maintainer because they guessed what the docs should say wrong (because the right docs were in the maintainer's head only), it's easier to just give up on writing docs and spend time on something downstream users actually appreciate, like working code. If the developer community wants people to work on things that matter directly to them only, like good docs, they need to actually start appreciating volunteers and their time more.

This isn't a hypothetical by the way, this situation has already happened in Asahi-adjacent threads and I think you're aware of at least one such instance. It also isn't a jab at you personally, I know you are not personally part of the asshole maintainer problem. I'm just trying to explain how I think we ended up here, and why the email workflow is a much bigger problem than you think.

@marcan @lhp @lschuermann

> That just means you're part of the minority that is

> The opinions of a minority should not dictate process

Golly

We should not make engineering decisions based on popularity.

If we took a vote the Linux kernel might be in Node.js

@worik @lhp @lschuermann This isn't an engineering decision, it's a process decision. And yes, we should make process decisions based on popularity, because the goal is to maximize developer happiness on average, not for a tiny subset of people to the detriment of the rest.

@marcan @lschuermann
Well than I profoundly apologize for everyone not having the funds to acquire the hardware needed to accommodate your taste in UI 😊

It's not a slow-down in total time, it's an interruption, a needless distraction.

I agree with you on a lot of things, but here you are needlessly dogmatic. It would be fine if you just said "I like forges". But no, you have to go out of your way to say "you are wrong for not liking what I like". Come on.

@lhp @lschuermann You're entitled to like whatever you like of course, but I can objectively point out a ton of things that email does worse than forges, and it shouldn't be hard to see why a lot of people value those things over trivialities like a couple extra seconds of delay or laptop fans.

Part of being a productive member of a diverse community is recognizing when you are in the minority, and that the preferences of the majority should drive the direction of developer tools. I know people who use GitLab but hate the web interface and do patch reviews with CLI tools, a la email. It's not ideal but it's a much more practical state of affairs than trying to force email on everyone just because you (and the rest of the status quo crowd, having driven away everyone else) like it.

Like come on, you don't think forges are popular by accident or because they were somehow forced down everyone's throats across the vast majority of the FOSS landscape, right? It's because most people think they're better.

@corbet @bars @marcan With all due respect, I highly disagree with almost everything in this comment. First off: The criticism comes from people who are involved or *were* involved but got burnt. Excluding those because they don't contribute anymore is excluding precisely those one should listen to.

Whatever the case, email is none of these things if you look closely enough. It's as hard as ever to self-host email (distribute) and email as SaaS is often proprietary or makes it irrelevant. It's crazy unstructured and a PitA to automate. The only choice you get is your mail client and what scripts you try to use or roll your own. Hector wants to use a git forge – obviously *not possible*. It's perceived inclusivity is the result of a filter: If you ask those who still use the system, they're fine with it and will, of course, argue against change and how that'd make their life harder.

Web Forges definitely scale that high, k8s as mentioned in the article still use GH. Seeing 5000 open PRs is "bad"? Well, if you'd actually be able to see how much work is lost or forgotten in the Linux Kernel... that'd also be a startling number. They're not perfect, but you can have each maintainer level (subtrees & cross-subtrees, up to for-linus and linus) in different forks or label their PRs, etc.

@corbet @bars @marcan They actually *are* highly scriptable, can be hosted by oneself, there are good opensource forges and they actually allow you to use whatever interface you want for interaction since they give you an API. There are CLI tools and GUI tools and Websites consuming those. You can even still send emails and for where you cannot you can definitely write an integration to trigger and API endpoint for that.

What you say about email being scriptable and automatable is not that: It *requires* heavy scripting to add features that are already common place in every forge. It's not allowing you to do something, its forcing you.

The talk by GKH and the attitude of the kernel devs regarding smart people is just elitist bullshit. And it's absolutely reasonable that people do *not* want to put up with that, *except* if they're part of the ingroup or highly paid for that; and even in the latter case, they only mind their own business.

What Hector is doing is the right move. There will always be protest and a top-down solution from Linus will draw ire. But the discussions are pointless. He moves his subtrees to a forge, and more and more maintainers will follow. It's the only way of change I see going forward.

@corbet @bars @marcan The take on Gerrit is outdated. It came a long way since 2016 and received a ton of investment. I've used it full time since 2015 and it became a real pleasure to use and manage. It's not perfect but the complaints Greg had 7 years ago are not true anymore.

@marcan Thanks! This is very much spot-on!

My own personal workflow for contributing to the kernel:
* spend days perfecting a single (1-10 lines long) patch
* send it weeks later (so I can deal with the energy drain of the process)
* receive comments, usually in the form "that's nice, you should also do this gigantic rework to improve the driver and get your patch in"
* never send a v2 and swear never to bother with this shitty process again
* repeat

@marcan I never contributed changes and fixes I made to internal Kernels for this reason.
I'm currently in progress of making a (fairly small) addition to an existing driver. The only thing on my mind was "how do I reduce the chance of upstream objecting to this" and not what the best architectural approach would be.
I would make much more changes to make the code more maintainable and easier to read but that would likely conflict with previous maintainer sentiment for this specific driver.

@maruel

It also simply talks about wrong things.

When a project of the scale of Linux kernel chooses its development tool, it doesn't choose a product to use, it chooses a project to collaborate with.

And rather than assessing the current state of the UI of the tool, you need to assess the fundamentals, the architecture and the potential to collaborate and design the solution for your tasks.

1/2

@corbet @bars @marcan

Linux Kernel will never get a ready to use forge which it can simply install and run.

It needs to *invest* and *work* together with the wider community to create the forge it can use, to design and build the workflows for it and with it.

And it needs to recognize it as an essential part of the project.

@maruel @corbet @bars @marcan

@corbet @bars @marcan I hope you take this in the spirit it's intended, but this reads exactly like the kind of statement a police union representative would make after 6 of the union's officers were recently arrested for dealing narcotics.

And I have every right in the world to crap on the community from afar, so I will. Thanks.

@vaurora @corbet @bars @marcan I stepped down from all roles to be able to do my little talk

and I waited until Linus' public mea culpa after that New Yorker article until I again picked up anything reassembling a formal role

"do it from the inside" is bullocks in this community, much better people than me tried and failed

@kernellogger @vegard @corbet @marcan People who are "totally drained" should quit. Or at least take a long hiatus. Especially if it means they can't do their job properly.

@marcan @ross @ska X11 is unmaintained and no one wants to work on it, but no one seems to want to work on making Wayland support all that X11 does either. Some wayland compositors fail to work properly on some hardware which makes wayland unusable for some (and lots of software don’t work yet, tho that’s on the software devs themselves not on wayland).
Screensharing doesn’t work at all on some wlroots compositors on nvidia, for example (on this case nvidia is not to blame, what they do is explicitly supported by wayland specs). And at least to my experience some apps constantly flash to invisible. Yes, it sucks that nvidia drivers are proprietary and no one is obligated to fix anything but stopping people with their hardware from even opening an issue is a straight up asshole move. I’m sorry but X11 just won’t die while wayland isn’t usable for everyone, and even if it does damn I prefer using a dead thing that works over a living thing that’s unusable anyday.

@luana @ross @ska

but no one seems to want to work on making Wayland support all that X11 does either.

What? Wayland is getting new features and improving towards feature parity with legacy X solutions all the time.

Yes, it sucks that nvidia drivers are proprietary and no one is obligated to fix anything but stopping people with their hardware from even opening an issue is a straight up asshole move.

No it's not. If things are broken on Nvidia due to bugs or missing features in Nvidia's drivers, then open bugs with Nvidia, not elsewhere. It is not an asshole move to refuse to support broken proprietary code.

I’m sorry but X11 just won’t die while wayland isn’t usable for everyone, and even if it does damn I prefer using a dead thing that works over a living thing that’s unusable anyday.

If X11 works for you and you are happy with it then why are you complaining about Wayland? You can keep using X11 and legacy software all you want.

See the funny thing is that everyone whining about Wayland on Nvidia seems to have an irresistible need to annoy everyone about it, even people who have nothing to do with Nvidia or Nvidia platforms, like me. They are somehow simultaneously upset with Wayland and would love to stick to X11, yet somehow just doing that doesn't work for them and they feel the need to complain.

Maybe don't do that. You can't have your cake and eat it too. Stick to old legacy software or embrace the future, the choice is yours. But you don't get to just complain without proposing any solutions or stepping up to help.

@marcan @ross @ska

Federation is overrated

Some people just don’t want a gitlab account I guess? If at least gitlab supported prs via email it could work. Also I wouldn’t say federation is overrated at all, centralization on git is a big problem and I can’t really switch to a self hosted gitea instance without sacrificing any possible contributions if people can’t contribute from like github or their own self hosted gitea. Maybe it doesn’t help on the specific Linux kernel case, but it definitely helps on many other problems

@marcan @aer I have stories on how a vendor backed off from having a mainline kernel driver for their GPUs because of upstream...

Upstream today has a driver developed independently from them, which that vendor just doesn't really care that much about.

@corbet @adamw @bars @marcan

Compiling without assistance from bug wranglers, doc writers and designers is like trying to do a DnD campaign called "woops all rouges"

The impact of Nathan Lee, Suv and now Krlr on inkscape's development has been monumental. None are developers, all kept the issues tracker producing actionable and detailed reports. But all were volunteers.

The ecological failure in foss can be solved, but not with dichotomised economics. Who pays matters. Who says matters.

@doctormo @corbet @bars @marcan true, but it's not just that. my job is QA but I spend a lot of time coding. I've written fixes for GNOME, KDE, and hundreds of other projects. it is *much* easier to do drive-bys for projects with decent issue tracking and an understandable pull request system with CI hooked up to it (so I know my fix isn't totally busted). the kernel is extremely resistant to drive-by contributions. obviously this is partly due to its nature, but only *partly*.

@adamw @doctormo @corbet @bars Yup, I've sent drive-bys to just about everywhere, including KDE and Inkscape, and there just isn't anything anywhere near as horrible as the kernel. KDE is particularly good in fact: https://social.treehouse.systems/@marcan/111274442140911850

Sometimes after getting burned out on the kernel I start to feel reluctant to contribute to everything else too, but then once I actually do I get reminded that it's just the kernel that sucks, not the rest of the world, thankfully.

@never_released @marcan @aer but is is a true story? I hear the excuse but often it's to cover for internal issues around IP or development practices. The GPU upstream is mostly welcoming and we are having words with out outliers who have over optimised their communication style in favour of asshole.

@marcan you're 100% on the money here. the processes are high friction and poorly documented, which causes new contributors to make process mistakes and ask lots of questions, which quickly becomes tedious for the maintainers, so they get burned out and snippy with everyone, and the culture has long been poisoned by abusive personalities who would rather shout at potential contributors than fix the root causes. we really need to break that cycle.

@gsuberland @marcan @quixoticgeek
We know someone in Australia who spent about a year working on a new website for #FDroid but the PR was rejected because of an Oxford comma!

@dsfgs @gsuberland @marcan @quixoticgeek

I will be the last to deny F-Droid has it's own problems with being unwelcoming to new contributors.

Wanting to change that is one of the reasons I volunteered for our new Community Council.

But a PR hat was "rejected because of an Oxford comma"? If that's true I'd very much like to look into what happened. Do you have a link for me?

@dsfgs @gsuberland @marcan @quixoticgeek do you have any source for that? I doubt that the comma was the real reason...

@jr @dsfgs @gsuberland @marcan having seen some people involved in FOSS, and the views some have against the Oxford comma, this feels incredibly plausible.

@quixoticgeek @jr @dsfgs @gsuberland We actually had a (tongue-in-cheek) argument about the oxford comma at XDC 2023 😂​

@marcan This made your point much clearer for me. Thanks!

@marcan @bars
I only ever tried to get (and got!) one patch into the kernel, and was incredibly lucky that it was about a bug I found that Linus introduced himself, so I mostly dealt with him directly and he was extremely friendly in the process, so I didn't get any toxicity.

But that email-based workflow with inline-patches and specially formatted mails that Thunderbird couldn't even create (without an addon) etc was quite painful. There must be a better way..

@marcan @corbet @bars Years ago, I wanted to try to push Pagure as an incremental improvement over the current situation. Unlike other Git forges, pull requests can be created to submit any branch from any Git repo anywhere on any Git server, and all project data is stored as Git repos that can be archived, replicated, mirrored, etc. for data backup and project distribution purposes.

I got discouraged from this because nobody was even willing to consider even that as a minor upgrade.

@bookwar @corbet @bars @marcan What do you think needs to happen for the culture to change so the project wants to invest to design and build workflows? (I happen to have experience in this area, it'd be fun to chat)

@maruel

I look from the outside so I may be wrong, but I see that kernel folks somehow identify themselves via disrespect to the tools.

It looks like the fit for the kernel community is not defined by the knowledge of the kernel, rather by fit in the existing workflow

And if a kernel developer raises the issue, he is then treated like an outsider

So kernel community is actively forcing people who may work on this out of the community.

It is a negative selection.

@corbet @bars @marcan

@maruel

And when you have such a negative selection process, I don't think you can just sit and wait until the natural grassroots initiatives slowly improve the situation.

Because how could they?

@corbet @bars @marcan

@bookwar @corbet @bars @marcan This sounds very unfortunate.

@marcan @lschuermann
This is what RISC-V kernel devs have set up:
https://github.com/linux-riscv/github-ci/wiki

It may inspire others, just leaving it here for reference.

I've just sent out my first patch to LKML a few weeks ago and already don't know how to reply (properly), so I got stuck - 110% agreeing with what you describe. 🙃

@CyReVolt @marcan @lschuermann What was the problem with replying? (It should be fine as long as you use "Reply all" and set your email client to use plain text.) This guide by @monsieuricon got posted a few days ago: https://subspace.kernel.org/etiquette.html

@vegard @marcan @lschuermann @monsieuricon

"set your email client to plain text" - does GMail do that? I ran git send-email, which I set up for uni-directionally using the sendgmail MTA. That is not suitable for replying.
Otherwise, I only use email for three other uni-directional purposes:
- notifications where I do not want a special app
- mailing lists where a forge is not first tier (LKML, U-Boot and EDK2)
- resetting passwords

For conversations, I use messenger/chat apps (no IRC).

@CyReVolt @marcan @lschuermann @monsieuricon When you compose or reply in gmail there's a 3 dot menu you can use to select "Plain text mode". This has worked for me.

@vegard @CyReVolt @lschuermann @monsieuricon That is not enough not to mangle patches and other kernel stuff. My understanding is GMail is outright unusable for that. Even in Thunderbird, you need to edit hidden config settings to get it to work properly in all the subtle cases. Yes, this is ridiculous.

@marcan @CyReVolt @lschuermann @monsieuricon Yeah, for patches you should always use git send-email. I've used the gmail web interface to reply to messages for years without any issues. That said, https://docs.kernel.org/process/email-clients.html#gmail-web-gui mentions that it will base64-encode any messages with non-ASCII characters, I'm not sure if that's still true or a problem -- if it is, I've never encountered it personally.

@marcan I’m sure many of your criticisms of the Linux process are entirely accurate.

But you seem to have an underlying assumption that gatekeeping = bad, more contributors, onboarded with minimal hurdles = good — a model that seems to work for you so far. But I’m not convinced it would scale to the Linux kernel as a whole.

@marcan “Working in Public” <https://press.stripe.com/working-in-public> discusses in detail how some contributions (and contributors) create more work for maintainers than they’re worth, and how an overly accommodating culture can lead to maintainer burnout. And maybe the toxicity of some maintainers is a product of such burnout.

@marcan Should a project, e.g., encourage Hacktoberfest style drive by PRs that tweak some wording in a README file just so the author can score a free T-Shirt? I would think not.

@microtherion That is a strawman. Those are not the commits that are being gatekept. Real contributions are.

If maintainers get burned out because there are too many contributors, that means the kernel needs more maintainers, and that can only happen by accepting more contributors in the first place. A toxic gatekeeping culture is a self-perpetuating problem.

@vegard @CyReVolt @lschuermann @monsieuricon It's not just patches, e.g. I had to fix those Thunderbird settings to send email pull requests properly using it as an editor (unlike git send-email for patches, that process generally wants you to edit the resulting message). But even for patches, just replying without those settings done right will end up mangling stuff. It's a mess.

@vegard @marcan @CyReVolt @lschuermann @monsieuricon imho, anyone who's getting into kernel development or who is open to changing their workflow should use b4 right away and avoid using git-send-email

https://b4.docs.kernel.org/en/latest

@marcan @corbet @adamw @bars yeah i have had multiple time when i wanted to contribute docs to kernel (or even glibc but different ball of crap), knew what to write, was ready to.

Then i remembered what it would take to actually upstream it. And just. Fuck off. I will go do it in a project where it is 3 clicks and 5min inside my usual workflow. And where i get a line of hearts emojis from the maintainer and CI that guide me.

@Di4na @marcan @corbet @adamw @bars Would you be willing to try [1] and give me your feedback?

https://b4.docs.kernel.org/en/latest/contributor/overview.html#what-is-the-b4-contributor-workflow

@monsieuricon @adamw @bars @corbet @marcan sure. First of all, fuck email and fuck having to put my email in there. And yes, i know git depends on it. Secondly, what the heck is a patch serie?
Git amend is one of these banned git command because it is too much of a pita to use and noone understand it.

Write commits in the right order? Wat? Cover Letter? I am not applying for a job. Next version? changelog? Wat?

Code review and feedback where? in email? fuck that. fuck email.

trailers? wat?

@monsieuricon @adamw @bars @corbet @marcan basically, no idea what this is saying and not something i would touch. I kinda get where this is coming from and what it is trying to do, but i just want to send you a few documentation. Theses are basic strings. In one single commit. You read it, you know what it is. And email? in 23? I have yet to find a single email client that can do replies correctly even less editing text. Why the heck do i need this stuff?

@monsieuricon @Di4na @bars @corbet @marcan I think it's a good effort, but it's fundamentally in the category of "tools attempting to slightly improve the experience of dealing with a difficult workflow". it looks better than *not* having it, but it can't really get to the heart of the workflow just being, in some respects, unnecessarily difficult and cumbersome.

(it also doesn't really help my typical case of 'trying to follow what's going on without being a regular list member' much at all).

@adamw @monsieuricon @Di4na @bars @marcan I have a different experience; I follow a vast number of projects on mailing lists quickly and efficiently - mostly without actually subscribing to the lists. But once a project disappears into its own special little web silo, it's gone from sight.

@corbet @Di4na @bars @monsieuricon @marcan I think the difference is between following a *project* and following a *bug*. I don't want to follow every project I might encounter a bug I need to care about in (well, it'd be nice, but there aren't enough hours in the day). I need to deal with specific *bugs*.

I think the mailing list setup works *much* better for "following a project" than "dealing with bugs".

@adamw @corbet @Di4na @bars @monsieuricon @marcan I think it's worth noting (in case it's not clear to everyone) that @adamw is doing QA for a whole (suite of) linux distros; tracking all the relevant projects is quite impossible at that scale.

@Di4na @adamw @bars @corbet @marcan Mmkay, sorry I asked.

@corbet btw, ironically, i mostly use email for dealing with the 'silo' problem. forges usually sign you up for email notifications for any issue or PR you create or comment on. i filter those emails into specific folders, then i can easily see from my email if there's something that I need to check in on. but then I can go use the nice forge workflow to actually interact with it...

@monsieuricon @Di4na @adamw @bars @corbet @marcan hmm, @monsieuricon, no need to be sorry. For receiving such toxic answer? The toxic complaining about toxic? Awesome.

@monsieuricon @adamw @bars @corbet @marcan no need to be sorry :) this was me giving feedback as naturally as possible. I understand where it come from and why it is difficult to do something about it. I don't blame you for it.

But i also doubt the pain we deal with can be properly realised without that kind of language. This is a situation where a bit of a shock treatment is necessary to get the divergence in context.

@brauner @vegard @marcan @CyReVolt @lschuermann @monsieuricon First time I have heard that being mentioned anywhere...