RE: Job Processing Was RE: Parallel Merge

[Nathan Watson-Haigh]

> 

 
Nathan Watson-Haigh
Senior Bioinformatician | The Australian Wine Research Institute
Waite Precinct, Hartley Grove cnr Paratoo Road, Urrbrae (Adelaide) SA 5064 | 
http://www.awri.com.au/contact/map.asp
PO Box 197, Glen Osmond SA 5064, Australia
T: +61 8 83136836 (direct) | T: +61 8 83136600 | F: +61 8 83136601
8 www: http://www.awri.com.au/ | http://www.awri.com.au/events/calendar/

This communication, including attachments, is intended only for the 
addressee(s) and contains information which might be confidential and/or the 
copyright of The Australian Wine Research Institute (AWRI) or a third party. If 
you are not the intended recipient of this communication please immediately 
delete and destroy all copies and contact the sender. If you are the intended 
recipient of this communication you should not copy, disclose or distribute any 
of the information contained herein without the consent of the AWRI and the 
sender. Any views expressed in this communication are those of the individual 
sender except where the sender specifically states them to be the views of the 
AWRI. No representation is made that this communication, including attachments, 
is free of viruses. Virus scanning is recommended and is the responsibility of 
the recipient.
-----Original Message-----
> From: ole.tange@gmail.com [mailto:ole.tange@gmail.com] On Behalf Of
Ole
> Tange
> Sent: Tuesday, 23 August 2011 6:06 PM
> To: Nathan Watson-Haigh
> Cc: parallel@gnu.org
> Subject: Re: Job Processing Was RE: Parallel Merge
> 
> On Tue, Aug 23, 2011 at 8:38 AM, Nathan Watson-Haigh
> <nathan.watson-haigh@awri.com.au> wrote:
> > Hi Ole,
> >
> > I'm in the middle of optimising the processing of such file. While
> I'm at it I have a quick question:
> >
> > I'm processing my input file with -N 2500 as it seems to give me the
> best processing time. My CPU usage and disk IO are well below their
> maximum capacity
> 
> I often face this problem aswell. Starting more processes normally
> solves the problem for me. I believe the issue is that while the disk
> I/O is below maximum capacity on average, there are spikes where it is
> over capacity (e.g. every time it has to seek). During theses spikes
> the CPU is waiting for data to process. By having more processes
> running than processors the "extra" processes can buffer up some input
> which can then be processed when there is idle CPU time.
> 
> So: Try starting twice as many processes (-j 200%).

I'm about to try this out, but you comments raised a couple of questions
for me:

Does this introduces the concept of a queue length vs the number of
processes that can be run at any one time? Are there options for
specifying these separately so I could specify only a subset of
available cores to be used, but still have parallel buffer up the extra
input in the way that you say?

> 
> Low throughput on disks can also be due to disk seeks.  Until recently
> I did not know of a tool that could detect disk seeks, but I have now
> found iostat. The '%util' column is very useful to see how busy the
> disk is:

This could be the issue as my raid array is only running at approx 10%
throughput while parallel is running.

> 
>   iostat -xd 1
> 

This isn't showing anything on my system at the moment, but I'll have a
look into it.

> Using -N indicates that your processing program can take more than one
> argument. If the startup time for the processing program is large, try
> using -X to distribute all the arguments amongst the processes.
> 
> > and wondered how GNU parallel processes/submits new jobs as others
> are completed? I'm thinking that GNU parallel is somehow stalling my
> pipeline. Could you provide some information on this aspect of GNU
> parallel?
> 
> GNU Parallel's main loop (drain_job_queue) is basically one big sleep.
> It gets woken up by a child dying. When a child dies GNU Parallel
> prints out stdout and stderr for that child and a new child is
> spawned. Then GNU Parallel sleeps again.
> 
> 
> /Ole