Re: parallel: This should not happen. You have found a bug.

[Ole Tange]

On Mon, Aug 12, 2013 at 1:05 AM, Linda Walsh <gnu@tlinx.org> wrote:

>> When run in --semaphore mode GNU Parallel uses hard links to create a
>> counting semaphore. If your filesystem does not support hard links the
>> behaviour is undefined - and that is likely what you see.
>
> NTFS supports hard links.  However, file accesses are infinitely slower than
> semaphores in memory.
>
> Ideally, the program could test to see if the system has
> semaphores (they work on cygwin and linux and any posix compatible
> using the POSIX::RT::Semaphore package.

It has a heavy downside: It seems not to be installed by default on:

  aix centos debian dragonfly freebsd hpux hurd irix mandriva minix miros
  netbsd openbsd openindiana qnx raspberrypi redhat scosysv solaris suse
  tru64 ubuntu ultrix unixware

GNU Parallel has a design goal to not require extra packages
installed. Installation should even on older systems simply be copy
the file 'parallel' to the path.

sem does 20 read accesses and 24 write access to do the semaphore
using the filesystem (fewer if it is not the only sem running).

sem takes 0.3 seconds to run on my machine. It is hardly a speed devil
as it is. So comparing it to xargs (or even parallel without
--semaphore) will always go against sem.

GNU Parallel is designed to work on multiple computers at the same
time if $HOME/.parallel is shared between the computers. So sem should
block on computerA if computerB has the semaphore (e.g. if they both
try to access the same file on an NFS based dir). A semaphore that is
only respected by computerA is thus not backwards compatible with the
current design.

Compared to the other 24 platforms Windows in general has more quirks
and has always been supported in GNU Parallel as a nice-to-have
platform - not a need-to-have. Not having ssh access to a
Windows/Cygwin environment also makes development for that platform
harder.

Compromising the design goal on at least 24 platforms is not worth it
to me to get --semaphore to work better on Windows.

If we can find a way that will work fast on Windows/Cygwin without
compromising the 1-file-install design goal and the shared $HOME
semaphore, I will definitely be interested.


/Ole