Can you come up with convincing *stats* on why backporting a huge pile of patches (say 800+, like in the case of #Linux 6.10.3) to stable #kernel series just days after being mainlined *during the merge window* and thus included in a mainline -rc1 is *worse* than quickly backporting changes mainlined *during the rest of the development cycle*?

Then tell me about it, because then I'll bring this up on the maintainers summit while talking about #regressions.

1/ FWIW, this is what I tried: I…

2/ …looked at the mainline changes backported to the 6.9.y series and then checked if they were referred to in a Fixes: tag found in another mainline change committed within 9 weeks (e.g. one cycle).

There the -rc1 commits did not stand out.

The only problem thus was the number of changes, which of course makes it more likely for users to be affected by regression.

Screenshot in the first toot of the thread if from https://lwn.net/Articles/863505/

@kernellogger What about counting reverts?

@penguin42

that's in fact an area where I still want to look closer; but some reverts should be covered by the tests I did (but I might not have noticed all of them, that is still on my todo list);

@kernellogger A priori, I would expect the rate of _fixes applicable to older kernels_ to be independent of where in the mainline cycle you are. Doing a quick grep on 5.15.y tells me that the number of commits from newer mainline -rc1s is about 8x the numbers compared to other -rcs. It would be interesting to know WHY. Are maintainers delaying fixes until the merge window? That seems wrong.

@vegard

Will think about that, but one thing now:

"[…] delaying fixes until the merge window? That seems wrong. […]".

Sometimes it is, but sometimes (often?) it's not. For example when the fix if fixing something the developer/maintainers knowns is not urgent or when it's fixing something in a kernel released more that a year ago, especially if there is a chance that the fix might cause regressions (see for example https://lore.kernel.org/all/CAHk-=wis_qQy4oDNynNKi5b7Qhosmxtoj1jxo5wmB6SRUwQUBQ@mail.gmail.com/ ).

@kernellogger Another possible explanation is that the "should this go into stable?" heuristic applied by the stable maintainers (*cough*autosel*cough*) has a certain probability of picking any given patch, i.e. it always picks the same _proportion_ of patches. So because -rc1 tends to be so big it translates into a higher number of patches from -rc1 overall. Which also seems wrong, somehow.

I'm sure Sasha has talked/written extensively about this in the past. More research needed, I think...

@vegard

hmmm, valid point, but I think that is an area I currently don't care about, as autosel only picks up patches after they have been in mainline for a while (4 weeks or something, can't remember). And then most regressions will have been found and reported already.

But most of those that made it into 6.10.3 are just backported because they have a Fixes: tag, so autosel is not involved here afaics -- and the commits did not get much testing in mainline yet.

@vegard @kernellogger my own experience is that after I add a new feature during the merge window, I'll get a bug report that was actually from an older change before the merge window. Why did this happen? Because the new feature started stressing code paths that were not stressed before and uncovered bugs that existed for a long time. Hence you may see more bug fixes for old bugs early in the rc releases.

@kernellogger Please do. This is reason why -stable is more experimental than -rc1. (At least in -rc1 changes are tested in right context. In -stable cherry-picked changes are applied with minimal review).

@pavel

sorry, but those are nice words and up to some point I agree, but those most likely won't convince Linus, Greg et. al.; I need some *stats* or something else that is able to do so.