Conversation
vegard@mastodon.social

Vegard Nossum 🥑

I benchmarked a few different versions of sphinx-build for 'make -j128 htmldocs' (with 128 cores) and got these results:

2.4.4 -- 7m37s
3.0 -- 94m43s
4.0 -- 70m49s
5.0 -- error
6.0 -- 97m12s
7.2.6 -- 17m18s

This machine can compile a full distro kernel in like 2 minutes.

I actually really like Sphinx but this is Not Good.

3
0
0
itaru@mstdn.party

Itaru Kitayama

Reply to @vegard@mastodon.social

@vegard surprising

0
0
0
kees@fosstodon.org

Kees Cook tux

Reply to @vegard@mastodon.social

@vegard yes, something is horribly wrong with Sphinx's parallelism. :(

0
0
0
corbet

Jonathan Corbet

Reply to @vegard@mastodon.social
@vegard Something weird is going on there to get numbers like that. Sphinx got a lot slower starting with 3.x, but not that much slower (about a factor of two for me).

As far as I can tell, Sphinx can only do the parsing and building of the internal tree in parallel; the HTML generation is single-threaded. So more cores only help so much. But I have no idea why you would spend over an hour waiting for something I can do in six minutes on a basic desktop machine.
0
0
2
vegard@mastodon.social

Vegard Nossum 🥑

Reply to

@jani @corbet For kernel_abi I think it should be fairly straightforward to change it to do what you're doing in kerneldoc.py?

But I really don't think what happens before we get to 'sphinx-build' is that much of a problem. Convoluted, maybe, but not a huge performance issue.

0
0
0
vegard@mastodon.social

Vegard Nossum 🥑

Reply to

@jani @corbet Hm, I was going to suggest 'make htmldocs V=1' but that doesn't actually seem to do the right thing for some reason -- I was expecting it to print out the exact sphinx-build invocation. Although I do see it printing "Running Sphinx v4.3.2" so I know that getting to sphinx-build is less than 2-3 seconds.

0
0
0
vegard@mastodon.social

Vegard Nossum 🥑

Reply to

@jani @corbet Luckily this is also fixable 🤓

1
0
0
vegard@mastodon.social

Vegard Nossum 🥑

Reply to @vegard@mastodon.social

@jani Do you want to claim this, or can I submit a patch?

0
0
0
vegard@mastodon.social

Vegard Nossum 🥑

Reply to

@jani @corbet It also leaks absolute paths into the build, see e.g.: https://docs.kernel.org/admin-guide/abi-obsolete.html#file-var-lib-git-docbuild-linux-obsolete-sysfs-bus-usb or https://static.lwn.net/kerneldoc/admin-guide/abi-obsolete.html#file-stuff-k-git-kernel-obsolete-o2cb

0
0
0
About social.kernel.org

Terms of service

  1. Please do not use this service in violation of the Linux Kernel Code of Conduct. Doing so will result in your account suspension with the referral of the matter to the CoC committee.

  2. "Repeating"/"boosting" someone else's status on this platform will be treated as endorsement and will fall under rule #1.

  3. You are encouraged to use this platform to promote your work on the Linux Kernel, but there is no restriction on permitted topics (with the exception of anything covered by #1 above).

  4. There is no requirement to post in English, but it should be considered the primary language of communication on this platform.

Privacy notice

The admins of this service have access to all posted statuses. They aren't looking, but if it's something they shouldn't know about, then you should not post it on this platform.

Please see the Linux Foundation Privacy Policy, which applies to this platform as well.

Getting your own account

If you would like an account on this instance, please check that the following applies to you:

  • You are listed in MAINTAINERS or CREDITS

  • OR: You have a kernel.org account or email address

  • OR: You have a long and established history of involvement with the Linux Kernel

If the above is true and you agree with the Terms of Service and Privacy Notice listed above, please use these instructions to request an account: