[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Reducing memory usage of the linker and assembler

From: Ludovic Courtès
Subject: Reducing memory usage of the linker and assembler
Date: Mon, 09 Jan 2023 11:46:02 +0100
User-agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)

Hello Guilers!

The ‘wip-linker-assembler-memory-consumption’ branch I recently pushed
aims to reduce the allocation rate and memory consumption of the linker
and assembler, as well as the number of bytevector copies.

In 3.0.8, the linker and assembler keep roughly two copies of the final
ELF file in memory: each <linker-object> contains a ‘bv’ field with its
wire representation, and then ‘link-elf’ allocates a bytevector to
contain the whole ELF file and copies each <linker-object> bytevector
back into that one.  When you’re running ‘guild compile’, that big
bytevector is immediately passed to ‘put-bytevector’ to write it to

The branch replaces the ‘bv’ field of <linker-object> with ‘size’ and
‘writer’, the latter being a procedure that takes a bytevector and
writes to it.

At this point, ‘link-elf’ still allocates one bytevector for each linker
object.  Eventually we could rewrite those “writer” procedures to use
binary I/O primitives instead of expecting a bytevector to write to, and
that way we wouldn’t need those temporary bytevectors.  It’s a bit
tedious to do though.

One big source of memory consumption that remains is the assembler: its
‘buf’ field contains an in-memory copy of the ‘.rtl-text’ section, which
is often a significant portion of the ELF file.  Addressing that would
be more difficult.

On the compilation at -O1 of a big file (‘gnu/packages/python-xyz.scm’
in Guix), I see a 5% reduction of the heap size as measured at the end
of the whole compilation process.  There are other factors before the
linking step that contribute to the peak heap size, which probably
explains why the impact is not higher.

The current set of commits looks like this:

  cdf9a0ac4 * DRAFT linker: Do not store entire ELF in memory when writing to a 
  048651993 * linker: Linker object writer takes a single argument.
  2ac4e5a3d * linker, assembler: Avoid intermediate bytevectors.
  8c8d4bd37 * DRAFT Add 'bytevector-slice'.
  805275882 * linker: Separate effectful part of 'add-elf-objects'.
  8194e8a6e * assembler: Separate effectful part of 'link-docstrs'.
  0a5768add * assembler: Separate effectful part of 'link-frame-maps'.
  f6fb95db8 * assembler: Separate effectful part of 'link-procprops'.
  099cf8f30 * assembler: Separate effectful part of 'link-dynamic-section'.
  43dd2f184 * assembler: Separate effectful part of 'link-symtab'.
  ad4487091 * assembler: Separate 'process-relocs' from 'patch-relocs!'.

There’s no regression.  The main issue worth discussing is the last
commit, which changes ‘link-elf’ to optionally return a procedure
instead of a bytevector.  We also need to discuss ‘bytevector-slice’,
but that can be done separately.

Andy, what do you think of this approach?


reply via email to

[Prev in Thread] Current Thread [Next in Thread]