bug#14752: sort fails to fork() + execlp(compress_program) if overcommit limit is reached

[Petros Aggelatos]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/01/2013 12:52 PM, Pádraig Brady wrote:
> On 07/01/2013 08:16 AM, Bernhard Voelker wrote: Petros, for
> completeness, what kernel were you using, and was SELinux enabled?
I'm running kernel 3.9.7-1-ARCH and SELinux is disabled.

> Note tmpfs is backed by RAM and swap, so depending on the swapiness
> settings for the kernel, it will auto migrate to the swap device(s)
> under RAM pressure.
I have swap disabled on my system as 16GB or RAM seem more than
enough. The problem is irrelevant whether it is using tmpfs or not, it
just happened to trigger the error on my machine as it grew. But one
can reproduce by disabling overcommit and running a sort with
compression and -S60%.

> BTW, -S40% may be the root of the issue. Petros have you tried
> smaller buffers, which would probably avoid the issue on fork(),
> but also may take advantage of cache locality.
I tried using smaller buffers but I needed more temp space which I
didn't have. If the initial cutting of the file results in more than
NMERGE chunks then in the worst case it will need:
  N + NMERGE * P * N / (NMERGE + 1)
where N is the input size and P is the number of merge-sorts running
in parallel. But by using -S40% the input file would be split in less
than 16 chunks so it would be just a single 16-way merge and require
just N size of temp space.

> vfork() might be an option here.  One can't rely on it being
> different to fork(), and it blocks the parent until the exec() in
> the child, and there are various restrictions on the child, but
> that might be fine? But I think posix_spawn() is the new
> standardised equivalent, and I notice the spawn-pipe gnulib module
> which might be leverged here?
I did some further research on this after the initial report and
posix_spawn() seems to me that is the standard way too. I could write
a patch and test it.

Regards,
Petros
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJR0VrnAAoJEHebBSKmldDLOasH/3iSJzO2AMU2LxKPKviSuMTz
5519trFD83e38C9I0o1ta830hofPXcpjoEHcinbc5VXibcLyvpfigSrcXoJa9FDZ
pHvdOVJELFq4Rd4bXfDoVGoFZ79oNAUAZyR/tg2lapL0TKAiLF6hLOdG2WTtbdmM
A1bOFIfjOWjC3Ro2aN7IPs4n1QTPYwLlmKCc4zCRWyl2B3m6EeiLN+qdKWNLpf/3
g8XoGv4yHRQENzk2s2o1aBcPyJ9FLtyQy/HlheNHThc9k+ReJe8JJ89h2UXhFKiT
xgE62oNpPi3PL0w8SGuziTQgIKOgRbalQEZafUbeW2HB0ZiBKjZiY9o2PWh3UzM=
=J1rs
-----END PGP SIGNATURE-----