Conversation

make htmldocs is 22% faster with pypy

1
0
0

Tried profiling sphinx to figure out where all the time is going.

...I still have no idea.

2
0
1

@vegard going to make a wild guess of "string copies"

1
0
0

@dotstdy We're spending like 3 minutes in something called deepcopy().

1
0
0

@dotstdy over 12 minutes are spent in that resolve_xref() trying to look up C identifiers by name by traversing some kind of tree and comparing them one by one. It *must* be possible to simplify that with a hash lookup or two.

1
0
0

@dotstdy and there's an open issue for it: https://github.com/sphinx-doc/sphinx/issues/10966 ...with patches 🤩

0
0
0
@kees @vegard maybe they glanced at paulmck's book and decided to take the easy way out
0
0
3

@kees I explicitly disabled the jobserver-exec/parallel-wrapper.sh logic in the kernel's documentation Makefile for profiling/benchmarking purposes. As far as I can tell, parallel support (-j) is there in the sphinx master branch.

0
0
0
@kees @vegard Current sphinx parallelizes much of the build, but output phase seems to be serialized, alas.
0
0
3
@vegard The C-domain stuff spends a lot of time building an elaborate data structure that, as far as I can tell, it doesn't actually use. A couple of years or so ago I went in with a hatchet and hacked a lot of it out, with a build-time improvement of about 20%.

I ran out of time before I could go further with it. All that work must be there for *some* reason, and I'd need to figure it out and prepare a proper patch to even try to upstream that work, and that would take a while. It would be nice to get back to it...
1
0
2

@corbet You don't still have those patches lying around by any chance? 🤓

I *think* a lot of that stuff was written for C++ where you may actually need to care more about name lookups since the same name can refer to different things depending on context (namespaces, member functions, etc.). But I could be completely wrong.

1
0
1
@vegard I stashed that stuff aside somewhere, would have to look. Not sure I can post it and ever show my face in public again, though...

C++ was my thought too, but I'm not convinced of that. I was wondering if somebody had been looking forward to features they never implemented.
0
0
2