bug#36630: closed ([PATCH] guix: parallelize building the manual-database)

[GNU bug Tracking System]

Your message dated Tue, 31 Mar 2020 15:02:47 +0200
with message-id <address@hidden>
and subject line Re: [bug#36630] [PATCH] guix: parallelize building the 
manual-database
has caused the debbugs.gnu.org bug report #36630,
regarding [PATCH] guix: parallelize building the manual-database
to be marked as done.

(If you believe you have received this mail in error, please contact
address@hidden.)


-- 
36630: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=36630
GNU Bug Tracking System
Contact address@hidden with problems
> --- Begin Message ---
>
>
>
> Subject: 
>
> [PATCH] guix: parallelize building the manual-database
>
>
>
>
> Date: 
>
> Fri, 12 Jul 2019 23:42:45 +0200
>
>
>
>
> * guix/profiles.scm (manual-database): par-map over the entries.  This
> distributes the load roughly equally over all cores and avoids blocking on
> I/O.  The order of the entries stays the same since write-mandb-database sorts
> them.
> ---
>  guix/profiles.scm | 20 ++++++++++++--------
>  1 file changed, 12 insertions(+), 8 deletions(-)
>
> diff --git a/guix/profiles.scm b/guix/profiles.scm
> index f5c863945c..374f0f8a90 100644
> --- a/guix/profiles.scm
> +++ b/guix/profiles.scm
> @@ -1312,15 +1312,11 @@ the entries in MANIFEST."
>          #~(begin
>              (use-modules (guix man-db)
>                           (guix build utils)
> +                         (ice-9 threads)
>                           (srfi srfi-1)
>                           (srfi srfi-19))
>
> -            (define (compute-entries)
> -              ;; This is the most expensive part (I/O and CPU, due to
> -              ;; decompression), so report progress as we traverse INPUTS.
> -              (let* ((inputs '#$(manifest-inputs manifest))
> -                     (total  (length inputs)))
> -                (append-map (lambda (directory count)
> +            (define (compute-entry directory count total)
>                                (format #t "\r[~3d/~3d] building list of \
>  man-db entries..."
>                                        count total)
> @@ -1330,8 +1326,16 @@ man-db entries..."
>                                  (if (directory-exists? man)
>                                      (mandb-entries man)
>                                      '())))
> -                            inputs
> -                            (iota total 1))))
> +
> +            (define (compute-entries)
> +              ;; This is the most expensive part (I/O and CPU, due to
> +              ;; decompression), so report progress as we traverse INPUTS.
> +              (let* ((inputs '#$(manifest-inputs manifest))
> +                     (total  (length inputs)))
> +                (apply append (par-map compute-entry
> +                                       inputs
> +                                       (iota total 1)
> +                                       (make-list total total)))))
>
>              (define man-directory
>                (string-append #$output "/share/man"))
> --
> 2.22.0
>
>
>
>
> --- End Message ---
> --- Begin Message ---
>
>
>
> Subject: 
>
> Re: [bug#36630] [PATCH] guix: parallelize building the manual-database
>
>
>
>
> Date: 
>
> Tue, 31 Mar 2020 15:02:47 +0200
>
>
>
>
> User-agent: 
>
> Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux)
>
>
>
>
> Hi Arne,
>
> Arne Babenhauserheide <address@hidden> skribis:
>
> > I now reduced the thread count to exactly 2 (to avoid running into
> > resource troubles; I hope that two should be safe) and added a mutex for
> > status messages to ensure that writes don’t overlap.
>
> It’s been 9 months but I finally committed a slightly modified variant
> as ef4b5f2fed3ca13a0e15a821ba7e561cd4395aa6.  It turns out that the
> mutex was unnecessary as ports are thread-safe.
>
> As noted in the log, I see a 36% speedup on my SSD laptop with 4 cores
> (slightly less with 2 cores).  It’s not great, but still an improvement!
>
> Thanks,
> Ludo’.
>
>
> --- End Message ---