bug#10064: sort: sometimes segfault with certain limits (using prlimit)

[Bernhard Voelker]

On 11/16/2011 07:32 PM, Paul Eggert wrote:
> On 11/16/11 09:25, Bernhard Voelker wrote:
>
> >    prlimit --stack=1000 --memlock=100000 --rss=1000 sort -u /dev/zero
> >
> > Well, mostly (4/5 times) sort got correctly "Killed",
> > but a segfault smells.
>
> Sorry, I don't have prlimit and don't know what it does.
> Can you reproduce the problem with a standard shell
> and ulimit?
>
> Or failing that, can you give a backtrace?

prlimit is a new command from util-linux around the prlimit(2) system call.

Comparing the prlimit values with that of ulimit shed some light on it.
Used to the kilobyte unit of ulimit (but prlimit using bytes),
I gave it extreme low limits. The prlimit syscall even rounded
my --rss and --stack values down to 0, see man prlimit(2):

       RLIMIT_MEMLOCK
              The maximum number of bytes of memory that may be locked 
into RAM.
              In effect this limit is rounded down to the nearest 
multiple  of
              the  system page size.

$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31578
max locked memory       (kbytes, -l) 9
max memory size         (kbytes, -m) 0
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 0
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31578
virtual memory          (kbytes, -v) 4927920
file locks                      (-x) unlimited

It seems that "sort" isn't even involved yet:
(gdb) bt
#0  0x00007fb58936f712 in _dl_start () from /lib64/ld-linux-x86-64.so.2
#1  0x00007fb58936baf8 in _start () from /lib64/ld-linux-x86-64.so.2
#2  0x0000000000000003 in ?? ()
#3  0x00007fff49b2b469 in ?? ()
#4  0x00007fff49b2b482 in ?? ()
#5  0x00007fff49b2b485 in ?? ()
#6  0x0000000000000000 in ?? ()

So this is not a coreutils bug.

However, now I'm really confused about ulimit vs. prlimit(2) values:

$ bash -c 'ulimit -m'
3447088
$ prlimit --rss --rss=1000 --rss bash -c 'ulimit -m'
RESOURCE DESCRIPTION                 SOFT      HARD UNITS
RSS      max resident set size 3529818112 unlimited pages
RSS      max resident set size       1000      1000 pages
0
$ prlimit --rss --rss=1000 --rss prlimit --rss
RESOURCE DESCRIPTION                 SOFT      HARD UNITS
RSS      max resident set size 3529818112 unlimited pages
RSS      max resident set size       1000      1000 pages
RESOURCE DESCRIPTION           SOFT HARD UNITS
RSS      max resident set size 1000 1000 pages

It seems the kernel stores the bytes unit which can be read by
prlimit(2) again, while ulimit - using getrlimit(2) - displays
the value in kilobytes:

$ prlimit --rss=1000 --rss strace  -f -e trace=getrlimit bash -c 'ulimit 
-m'
RESOURCE DESCRIPTION           SOFT HARD UNITS
RSS      max resident set size 1000 1000 pages
getrlimit(RLIMIT_NPROC, {rlim_cur=31578, rlim_max=31578}) = 0
getrlimit(RLIMIT_RSS, {rlim_cur=1000, rlim_max=1000}) = 0
0

That means that the manpage of prlimit(2) is incorrect, isn't it?

Thank you & have a nice day,
Berny