Re: parallel stops working for no obvious reason

[Thomas Sattler]

>> This happend last week with v20120222, I'll give it another
>> try with v20120322 today.

_creating_ these files seems to work with v20120322. (I created
nearly 4,400,000 files by now.) But now deleting them is still
an issue. Luckily it's much easier to trigger this way. :-)

> As you probably can imagine that is hard to reproduce. See if
> you can make smaller example fail - preferably something that
> can run on smaller machines.

I wrote a small script that shows the problem. It completes
in less than 10 seconds on my desktop (two cores), but hangs
(read: "does not complete within hours") on two other
machines (8/32 cores).

The problem seems to be within load handling on multicore
machines. (All machines run successfully when removing
"--load 100%".)

Here's a sample output:

--- 8< ---------------------------------------------------------------

$ time pissue
PARALLEL=--load 100%
cores on dragon: 2

 16:26:50 up 9 days, 39 min,  2 users,  load average: 0.15, 0.11, 0.14
4 files: creating (0s) deleting (0s)

 16:26:50 up 9 days, 39 min,  2 users,  load average: 0.15, 0.11, 0.14
9 files: creating (0s) deleting (0s)

 16:26:50 up 9 days, 39 min,  2 users,  load average: 0.15, 0.11, 0.14
16 files: creating (1s) deleting (0s)

 16:26:51 up 9 days, 39 min,  2 users,  load average: 0.15, 0.11, 0.14
25 files: creating (0s) deleting (1s)

 16:26:52 up 9 days, 39 min,  2 users,  load average: 0.15, 0.11, 0.14
36 files: creating (0s) deleting (0s)

 16:26:52 up 9 days, 39 min,  2 users,  load average: 0.15, 0.11, 0.14
49 files: creating (1s) deleting (0s)

 16:26:53 up 9 days, 39 min,  2 users,  load average: 0.15, 0.11, 0.14
64 files: creating (0s) deleting (1s)

 16:26:54 up 9 days, 39 min,  2 users,  load average: 0.15, 0.11, 0.14
81 files: creating (0s) deleting (1s)

 16:26:55 up 9 days, 39 min,  2 users,  load average: 0.30, 0.14, 0.15
100 files: creating (0s) deleting (1s)


real    0m6.223s
user    0m2.873s
sys     0m0.813s

--------------------------------------------------------------- >8 ---

Thomas

pissue
Description: Text document