Re: [PATCH v3] meson: Pass -j option to sphinx

[Fabiano Rosas]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Sat, Apr 29, 2023 at 02:33:17PM +0200, Paolo Bonzini wrote:
>> On 4/28/23 17:01, Fabiano Rosas wrote:
>> > Also make sure our plugins support parallelism and report it properly
>> > to sphinx. Particularly, implement the merge_domaindata method in
>> > DBusDomain that is used to merge in data from other subprocesses.
>> > 
>> > before:
>> >    $ time make man html
>> >    ...
>> >    [1/2] Generating docs/QEMU manual with a custom command
>> >    [2/2] Generating docs/QEMU man pages with a custom command
>> > 
>> >    real    0m43.157s
>> >    user    0m42.642s
>> >    sys     0m0.576s
>> > 
>> > after:
>> >    $ time make man html
>> >    ...
>> >    [1/2] Generating docs/QEMU manual with a custom command
>> >    [2/2] Generating docs/QEMU man pages with a custom command
>> > 
>> >    real    0m25.014s
>> >    user    0m51.288s
>> >    sys     0m2.085s
>> 
>> The 'nproc' fallback will potentially cause twice #CPUs processes to be
>> active, since sphinx will run in parallel with everything else.
>> 
>> Is this result with "-j auto", and if so with which computer?  If the
>> speedup is only 2x as it seems to be from the "time" above, I'd rather have
>> "-j 2" only so that sphinx doesn't risk killing the machine...

Tested with -j auto and -j16 on my 16 cpu i7-11850H.

>
> Why would it kill the machine ? If there are two sphinx processes
> concurrent, thus overcomitting available CPUs, the scheduler will
> just end up giving them shorter timeslice OS.

One other thing I noticed is that we're not actually running the two
documentation targets in parallel. The man pages have a dependency on
the html pages. I'm not sure if that is legitimate.

  sphinxmans += custom_target('QEMU man pages',
                              build_by_default: build_docs,
                              output: these_man_pages,
HERE -->                      input: this_manual,
                              install: build_docs,
                              install_dir: install_dirs,
                              command: [SPHINX_ARGS, '-b', 'man', '-d', 
private_dir,
                                        input_dir, meson.current_build_dir()])

Removing that line gains us about 10 more seconds. However there are
some annoying differences in the man pages produced such as quotation
marks using a different character. I'm still investigating.