Re: parallel: This should not happen. You have found a bug.

[Linda Walsh]

> When run in --semaphore mode GNU Parallel uses hard links to create a
> counting semaphore. If your filesystem does not support hard links the
> behaviour is undefined - and that is likely what you see.
>   
NTFS supports hard links.  However, file accesses are infinitely slower 
than
semaphores in memory.

Ideally, the program could test to see if the system has
semaphores (they work on cygwin and linux and any posix compatible
using the POSIX::RT::Semaphore package.

I wrote the below blob to make it slightly easier to call:
(from the current program I am working on, so I just
copied it from the edit buffer...
)
############################################################
{    package POSIX::Semaphores;          #{{{
   use POSIX::RT::Semaphore;
   use mem;    #on CPAN
   our (@EXPORT, @EXPORT_OK, @ISA);

   sub sem_open ($;$$$) { POSIX::RT::Semaphore->open(@_) }
   sub sem_close($) { $_[0]->close }
   sub sem_unlink($) { POSIX::RT::Semaphore::unlink($_[0]) }

   sub sem_post($) { $_[0]->post }
   sub sem_wait($) { $_[0]->wait }
   sub sem_getvalue($) { $_[0]->getvalue }

   sub sem_timedwait($$) { $_[0]->timedwait }
   sub sem_trywait($) { $_[0]->trywait }
   sub sem_destroy($) { $_[0]->destroy }
   sub sem_name($) { $_[0]->name }

   use mem (@EXPORT = qw(sem_open sem_close sem_unlink
                                               sem_post sem_wait 
sem_getvalue sem_name ),
                       @EXPORT_OK = qw(sem_timedwait sem_trywait 
sem_destroy ),
                       @ISA    = qw(Exporter));
   use Exporter;
}           
------------------

Just made them easier to call but other than that there's not much to 
the above.

>   
> > > What OS is Perl running on and what file system is /home/law/.parallel
> > > hosted on?
> > >       
> > CYGWIN_NT-6.1-WOW64 / NTFS
> >
> > (thats 32bit cygwin on 64-bit win7-sp1)...
> >     
>
> I have no idea whether that configuration supports hard links or not.
>   
It does -- it had to support the POSIX subsystem which required it.

Just that NTFS file accesses are slow.... 20 minutes for ~400 in 
parallel using NTFS hardlinks
vs. <2 seconds for ~200  for the xargs...

Using real semaphores or something else in memory is worth it with that type
of performance discrepancy...