Reminder that if you get jemalloc errors on software running on Asahi, that's because the jemalloc maintainers refuse to make jemalloc builds portable to all ARM64 systems by default.

Some downstream projects also refuse to override the default for their builds when the issue is raised with them, like Telegram Desktop.

Unfortunately, choices like these are sabotaging the ARM64 Linux ecosystem by explicitly making binaries non-portable, and if people don't listen to our feedback, there is very little we can do to help. These "build for the host page size" approaches are as non-portable as -march=native, but people seem to think it's okay when you're breaking ARM64 systems and not x86_64 systems for some reason.

To be clear, it's perfectly okay to allow non-portable builds for people who want to rice out their system a la Gentoo. That seems to be the motivation for the jemalloc behavior in the first place.

But the default should always be portability. Always.

@Zergling_man This is not about architecture support. This is about bad defaults. It's okay to have bad defaults because you don't know. It's not okay to keep the bad defaults after being told they're bad and making everyone's builds non-portable.

There is no testing required, the build option is right there and we know it works and makes the builds portable. They just refuse to flip the switch out of optimization concerns, because apparently "go fast and break systems" is okay when your target architecture is ARM64 but not okay when your target architecture is x86_64 or anything else, otherwise everyone would be defaulting to -march=native everywhere.

@marcan Can the broken applications be run in the microVM that steam uses, since that is built for 4K pages as a workaround?

@RahulVadhyar Yes, but that's immensely silly for FOSS software, and it comes with downsides. We do not advocate nor support using the VM solution as a workaround for poor FOSS software packaging, the problem needs to be fixed at the source. The VM is for proprietary x86_64 software run under emulation.

@marcan You didn't guess which one properly.

@Zergling_man Yes, "not giving a shit" is the root of the problem here.

@marcan Exactly, so build it yourself, post link somewhere sane and call it a day

@Zergling_man And do that for every single downstream project using and bundling jemalloc, including things like Flatpaks which are now nonportable in the canonical Flathub location and would need redundant builds under a different app name and redundant maintainership and...

vs.

Fix the damn defaults in jemalloc and end this mess.

@marcan Haha, skill issue, makepkg go brrrrr

@Zergling_man You mean the makepkg from the distro with no upstream ARM64 support at all, and where the only downstream ARM64 fork is maintained by a team of approximately size 1 who can't keep up and keeps regressing systems?

There's a reason why we dropped Arch Linux ARM...

@marcan thanks for another reminder that you love to use your large social media following to incite dogpiling. Your comment on that github issue was even called out as abusive: https://github.com/jemalloc/jemalloc/issues/467#issuecomment-2307656698
Still no self-reflection on your side? Too bad.

@marcan Dynamic 4k/16k pages support may affect performance, so it seems, jemalloc just need dynamic switch between two implementations (ifunc may be useful)

@mittorn @marcan

It is 4/16/64k pages. There are users (like HPC) which go 64k for some % of performance.

@vbabka Your "inciting dogpiling" is my "getting maintainers to actually care about non-x86 users" (and the thumbs up/down ratio on that comment seems to agree with my side).

This has been going on for *years*, and at this point social media pressure is absolutely warranted because it's ridiculous we keep running into this over and over and over again and the obvious solution isn't being implemented because the maintainers just don't care.

Hint: Calling out user- and packager-hostile behavior on social media isn't automatically "dogpiling" just because you don't agree with me.

@marcan @Zergling_man
sounds exactly like running a distro...

@wolf480pl @Zergling_man@sacred.harpy.faith And in fact Fedora fixes this nonsense downstream, but that doesn't solve the issue for every other distro, or for non-distro things like Flatpak.

The problem could be fixed once in one place, and save hundreds of people the trouble.

@vbabka @marcan I'm confused on what there is to reflect on? Without digging into details, this seems like an easy fix.

@vbabka @marcan Marcan's comment is pretty much saying

"Your defaults cause hundrends of runtime issues for the benefit of some optimisations that can happen on smaller sized pages - when it works.

Change the default to be sometihng that works and let the developers define a different page size if they want to benefit from the compiler optimisations on top of that"

How is that an abusive comment ? sounds like the person that got offended forgot that "being offended does not mean you are right" thing...

@marcan I point out an issue with abusing social media popularity and you argument with the number of comment thumbs up/downs? Amusing.

And I actually do agree with you about the underlying issue, but certainly not the way you try to achieve it.

@xsk @vbabka Maybe they take offense at the "sabotaging the ecosystem" comment?

Unfortunately, when you make a choice that happens to create pain and make things not work by default for a subset of a given architecture ecosystem, and refuse to change that choice when it is brought up, yeah, that's pretty much what you're doing. Especially for something as widely used (and widely vendored) as jemalloc.

@marcan
Is there any solution for software that's already using jemalloc? Would setting a 16k page size on linux-aarch64 builds work? Or would that also break stuff?

@mithicspirit There is no reasonable solution for software that is already compiled with 4K-only jemalloc, no. It just won't work on 16K/64K kernels, period. It needs a recompile with the proper jemalloc options.

Just like there's no reasonable solution for software that assumes 4K in other ways, it's just that jemalloc are basically the only upstream refusing to fix this problem by default. We already got basically every other similar issue in major software fixed.

ARM64 Linux toolchains have been using 64K alignment for a long time now, for this same reason (that setting is compatible with all page sizes).

@marcan @vbabka

The problem with that comment claiming abuse on the PR is that it is inherently focusing on producing feelings and misdirection in an otherwise perfectly formed argument with data points and suggested solution, coming clearly from someone that uses that project and wants to keep using it.

They would have a point if the comment was : "we plan to make multiple jemallocs, one for each arch" or something that would address the pain points of the users of a project - even not as one would expect, and you then claiming that "I am Hector Martin and demand things to happen my way", That would have been a different story.

I exaggerate to make the point more clear, I don't expect this from anyone that struggles to create things as a team. But it is obvious to me that this narrative path is not having a solution oriented or even discussion goal, most likely it is a vent that mistakenly brands a person as abusive because they made a point that was not aligned with the current decisions.

Unfortunately public projects have a lot of stress to maintain, that is understandable, but I think that there is nothing wrong to say something like "You may have a point, could we discuss this a bit more outside the scope of this PR ?" publicly, to remove the peer pressure from the discussion.

Collaboration is the while reason why public code exists.

@vbabka @marcan what dogpiling? there is one new comment today

@xsk @vbabka Note that the person who made the abuse claim is not a project contributor nor maintainer, as far as I know. It's a random drive-by.

@marcan I'm particularly interested in the case where it can be recompiled, but is distributed directly rather than via a distro (specifically, the astral.sh tools; I know some distros provide them, but they also have an installation script). Would building for 16k pages (the current solution: https://github.com/astral-sh/uv/blob/dd4b27bfbdacff33ebfcf9bccb971413240533b3/.github/workflows/build-binaries.yml#L291 ) allow it to work on Asahi? Or would 64k be necessary for that?

@mithicspirit That's 64K (1<<16) and it should work with all systems if built like that. The argument is the log2 of the page size.

@whitequark @vbabka I think my comment got a whole bunch of new thumbs up today and the abuse claim comment a bunch of new thumbs down, maybe that's dogpiling these days? :-)

@marcan @xsk @vbabka as always... Sadly