@ptesarik @ljs

Hopefully, the rise of the physical world comes without the return-to-office :P

I should have been a photographer ?!

@a1ba

@ljs yay, this means I was weak and now I'm getting stronger!

@ljs And while on a diet, I can't eat like I used to before so I've been taking 3-4 teas a day like a Brit to calm down my stomach

@ljs It's really sore that I'm starting to worry lol

I didn't really work on my legs much before, but ever since I started PT, my legs hurt with every move I make

On diet day 3

@oleksandr
lol man I'd rather stay AI-illiterate (if possible)

@oleksandr and then misread LLM output!

Uh I don't want to misread code

@ljs Hail gym!

First day of PT, quite tired.
…now my trainer says only two sandwiches are allowed every day. Oh no!

For a while now, the kernel's configuration and build systems have been an area of concern for me. Almost nobody truly understands those complex subsystems, which were handled by a single maintainer.

That maintainer, Masahiro Yamada, has just stepped down after eight years on the job:

https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d6841d5cb20

Happily, Nathan Chancellor and Nicolas Schier have agreed to pick up the build system. The configuration system, instead, is now unmaintained. That ... seems less than optimal.

Thanks to Masahiro for doing this work all these years, and to Nathan and Nicolas for stepping up!

@ptesarik Thanks for the answer!
Yeah, that’s the tricky part, deciding to ignore (or not) when statistics don’t tell us anything about it.

By the way, if you're using mmtests for benchmarking, some results are marked significant (with asterisks) and some are not.

not significant: 17.74 ( -3.72%)
significant: 132.26 * -15.92%*

I assumed the asterisk would have some meaning, but didn't exactly know what it is until I read the code: "significant" means their (baseline and new) confidence intervals do not overlap.

When the variance is high and the number of observations is low, many results may not be marked as significant. However, that doesn’t necessarily mean the results can simply be ignored (?)

Still digesting it, and may be slightly incorrect, but a summary of what I've learned: properly using statistics to compare the performance of computer systems [1] [2].

0. When you get a sample and calculate sample mean, it is likely different from the population mean. Simply comparing two sample means from different computer systems may therefore lead to misleading conclusions, since the observed difference could be due to random sampling variation rather than a real performance difference (especially when the variance is high).

1. Central Limit Theorem indicates that, regardless of the underlying distribution, the sampling distribution of the mean tends to follow normal distribution when the sample size is sufficiently large (typically n >= 30). But to apply CLT, the observations should be independent and collected from the same distribution.

2. Based on the CLT, we can estimate how close the sample mean (to be precise, any population parameter) is likely to be to the population mean. A confidence interval [x, y] with a confidence level of p% means that if we repeated the sampling process many times under the same conditions, about p% of those intervals would contain the population mean.

3. To compare two distributions, confidence intervals can help determine whether the difference between mean values is statistically significant. This can be done by:

- Checking whether the confidence intervals of the two samples do not overlap, or
- Examining the confidence interval for the mean of the differences to check if it does not include 0.

4. Caveat: The CLT assumes that the data is collected independently, meaning one observation does not affect another. In computer systems, this assumption usually does not hold. Caches, memory layout, scheduling decisions etc. can introduce some degree of dependencies between observations.

This can be mitigated by 1) reducing dependence between experiments as much as possible or 2) by applying the bootstrapping method [3].

[1] Jan Kara, Measuring performance regressions, https://youtu.be/HAHhW13ofrg?si=drgegMwXUDegHsQf
[2] Dev Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, https://www.amazon.com/Art-Computer-Systems-Performance-Analysis/dp/0471503363
[3] https://en.wikipedia.org/wiki/Bootstrapping_(statistics)

@vbabka @ljs @wagi

right, no matter how many transistors you throw into a cache.

@wagi @ljs

I have a bigger, scalable bookshelf and I ordered small one to use it like a cache for a larger bookshelf but turns out the cache was too small :P

@ljs it's so sad that the bookshelf isn't big enough to fit the best book!