Re: [coreutils] over aggressive threads in sort

[Jim Meyering]

Jim Meyering wrote:
> Pádraig Brady wrote:
>> FYI https://bugzilla.redhat.com/show_bug.cgi?id=655096
>
> Ouch!  Thanks for forwarding that.
>
> Here's one way to test for the bug:
>
>   seq 100000 > in
>   mkfifo fifo
>   (for i in $(seq 12); do read line; echo $i; sleep .1; done
>     cat > /dev/null) < fifo &
>   (ulimit -t 1; ./sort in > fifo \
>     || echo killed via $(env kill -l $(expr $? - 128)))
>
> It fails on multi-core systems:
>
>     1
>     2
>     3
>     4
>     5
>     6
>     7
>     8
>     9
>     killed via XCPU
>     10
>     11
>     12
>
> But succeeds with the sort from coreutils-8.5.

Here's the beginning of a test suite addition for that bug.
I've added it to the XFAIL list, for now.
As I'm posting this I realized it will fail (i.e, pass)
on uniprocessor systems, or if you set OMP_NUM_THREADS=1 in
your environment, which led to another patch, one that ensures
an OMP_NUM_THREADS setting does not perturb tests.  That's below.
I'll probably push the OMP_NUM_THREADS-adding test shortly.

>From fbb47bf42457f8126cc128ff2c4590ad34d5e730 Mon Sep 17 00:00:00 2001
From: Jim Meyering <address@hidden>
Date: Sat, 27 Nov 2010 15:50:43 +0100
Subject: [PATCH 1/2] tests: test for parallel sort spinlock abuse

* tests/misc/sort-spinlock-abuse: New file.
* tests/Makefile.am (TESTS): Add it.
(XFAIL_TESTS): Mark this as an expected-to-fail (for now) test.
---
 tests/Makefile.am              |    4 ++++
 tests/misc/sort-spinlock-abuse |   37 +++++++++++++++++++++++++++++++++++++
 2 files changed, 41 insertions(+), 0 deletions(-)
 create mode 100755 tests/misc/sort-spinlock-abuse

diff --git a/tests/Makefile.am b/tests/Makefile.am
index 2e33063..b3be4df 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -237,6 +237,7 @@ TESTS =                                             \
   misc/sort-merge-fdlimit                      \
   misc/sort-month                              \
   misc/sort-rand                               \
+  misc/sort-spinlock-abuse                     \
   misc/sort-unique                             \
   misc/sort-version                            \
   misc/split-a                                 \
@@ -654,4 +655,7 @@ pr_data =                                   \
   pr/ttb3-FF                                   \
   pr/w72l24f-ll

+XFAIL_TESTS =                                  \
+  misc/sort-spinlock-abuse
+
 include $(srcdir)/check.mk
diff --git a/tests/misc/sort-spinlock-abuse b/tests/misc/sort-spinlock-abuse
new file mode 100755
index 0000000..974e3d5
--- /dev/null
+++ b/tests/misc/sort-spinlock-abuse
@@ -0,0 +1,37 @@
+#!/bin/sh
+# trigger a bug that would make parallel sort use 100% of one or more
+# CPU while blocked on output.
+
+# Copyright (C) 2010 Free Software Foundation, Inc.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. "${srcdir=.}/init.sh"; path_prepend_ ../src
+print_ver_ sort
+
+test "$(nproc)" = 1 && skip_ "requires a multi-core system"
+
+seq 100000 > in || framework_failure_
+mkfifo fifo || framework_failure_
+
+# Arrange for sort to require 1.2+ seconds of wall-clock time,
+# while actually using far less than 1 second of CPU time.
+(for i in $(seq 12); do read line; echo $i; sleep .1; done
+  cat > /dev/null) < fifo &
+
+# Limit CPU time to 1 second
+ulimit -t 1
+sort --parallel=2 in > fifo || fail=1
+
+Exit $fail
--
1.7.3.2.846.gf4b062


>From 43d1112d01c0251076b5ec61605e45b101ab3e12 Mon Sep 17 00:00:00 2001
From: Jim Meyering <address@hidden>
Date: Sat, 27 Nov 2010 15:58:51 +0100
Subject: [PATCH 2/2] tests: don't let the OMP_NUM_THREADS envvar affect sort 
tests

* tests/envvar-check (vars): Add OMP_NUM_THREADS.
---
 tests/envvar-check |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/tests/envvar-check b/tests/envvar-check
index 1dee9af..ac53ee4 100644
--- a/tests/envvar-check
+++ b/tests/envvar-check
@@ -34,6 +34,7 @@ vars='
   LANGUAGE
   LS_BLOCK_SIZE
   LS_COLORS
+  OMP_NUM_THREADS
   POSIXLY_CORRECT
   QUOTING_STYLE
   SIMPLE_BACKUP_SUFFIX
--
1.7.3.2.846.gf4b062