Re: [PATCH] build: Speed up .go compilation.

[Taylan Ulrich Bayırlı/Kammer]

Mark H Weaver <address@hidden> writes:

> I'm sorry I haven't been following this discussion closely, but let me
> first say that the performance gains you've been able to achieve are
> very exciting.  Thanks for working on this!

Hi Mark, thanks for the kind words. :-)

> Unfortunately, I have concerns:
>
> A few people mentioned on IRC that when doing these concurrent
> compilations within a single process, they sometimes see warnings like
> this:
>
> ;;; WARNING (module #<module () 262a990> not in submodules table)
>
> I haven't yet investigated, but my strong suspicion is that this is due
> to the fact that Guile's module system is not thread safe.  More
> specifically, when a new module is created, it mutates the global
> directory of modules in a way that is not thread safe.
>
> New modules are created by 'compile-file', both for the module being
> compiled and for any imported modules that haven't been previously
> loaded.  Unfortunately, this means that this approach of compiling files
> in multiple threads within a single guile process is not safe.  There
> are likely to be random crashes and corruptions.
>
> I suggest that for now, the best way can do safely is to adopt an
> approach of running multiple Guile processes, where each one compiles
> multiple files within a single thread.

The latest version of the patch loads all needed modules first, in the
main thread, and then compiles the files corresponding to the modules in
parallel.  Does that still lead to mutation in the directory of modules
in the parallelized segment of the code?

>     Regards,
>       Mark

Taylan