parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Any tips about parallel and sem when using intensive I/O operations?


From: ningyi shao
Subject: Any tips about parallel and sem when using intensive I/O operations?
Date: Thu, 15 Mar 2012 14:10:05 -0400

Now I am using parallel (in fact, sem) to run samtools and other next generation sequencing analysis.
Some things are quite similar as this blog described:
http://zvfak.blogspot.com/2012/02/samtools-in-parallel.html
But I like use sem in such way:

export PRO="${HOME}/projects/2012-03-09_H3K4me3"
export RESULT="${PRO}/result/ngs.plot/2012-03-15"
export DATA="">mkdir -p ${RESULT}

INPUTS=("Sample_H" "Sample_G")

# setup tagdirectory of inputs
for INPUT in ${INPUTS[@]};do
    sem -j4 samtools rmdup -s ${DATA}/${INPUT}.bam ${RESULT}/${INPUT}_rmdup.bam
done

TREATS=("Sample_D" "Sample_E" "Sample_F")
for TREAT in ${TREATS[@]};do
    sem -j4 samtools rmdup -s ${DATA}/${TREAT}.bam ${RESULT}/${TREAT}_rmdup.bam
done
sem -w

But I met some problems as when the load of the server heavy, then the output of the sem sometimes will lose output randomly. At the step there is no error report in the log files. The further processing then report and I feel quite trouble to trace back the problem because I didn't get clear clue about it. For example, a line of perfect bed file may should be:
chr1 1000 2000 tag1 256 +
and the result I found the line that was the output of sem subthread is :
chr

I know I'd better provide clear steps and example to repeat such problem, but it is quite randomly and what I only know is that it highly relates to high I/O intensive operations, especially when I use pipe. Does any body also meet such problem? Or I am the only one met such problem?
Are there some tips that I could find the error early and trace the problem? I read the man page of sem, but no clear clues about such problem, and I also search the mail-archive.com about parallel, no clear solution for it.

Thank you for my trivial problem.

Best,

Ning-Yi SHAO

reply via email to

[Prev in Thread] Current Thread [Next in Thread]