Re: Properly setting $Global::max_file_length to avoid line 12170 error

[Ole Tange]

On Wed, Oct 5, 2022 at 7:55 AM Nagle, Michael F
<michael.nagle@oregonstate.edu> wrote:
>
> Correct. I have a few lines of bash code to automatically prepare the jobs 
> file I just sent. Immediately after that file is made, it is passed as the 
> $job_list_name variable to...
>
> parallel --results output -a $job_list_name
>
> Please let me know if any other information would be helpful, and thanks for 
> all the suggestions and thought into this...

I have the feeling your problem is that the filesystem you use has a
restriction on the length of a dir:

mkdir 
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456

fails on ext4 and tmpfs.

The \_ is  (as you guessed) by design to keep the value in a single
dir, so that GNU Parallel can assume a structure like:

            parallel --header : --results foo echo {a} {b} \
              ::: a I II ::: b III IIII

            foo/a/II/b/III/seq
            foo/a/II/b/III/stderr
            foo/a/II/b/III/stdout
            foo/a/II/b/IIII/seq
            foo/a/II/b/IIII/stderr
            foo/a/II/b/IIII/stdout
            foo/a/I/b/III/seq
            foo/a/I/b/III/stderr
            foo/a/I/b/III/stdout
            foo/a/I/b/IIII/seq
            foo/a/I/b/IIII/stderr
            foo/a/I/b/IIII/stdout

What you can do is to use a replacement string to shorten your value:

    parallel --results output/'{=s/gemma.*pheno_files//=}' -a $job_list_name

Or you could use GNU Parallel to generate the commands:

   doit() {
       file="$1"
       outname=$(echo $file | sed -e "s/\\.noheader\\.pheno//g")-ResidOverPhase
       outname=$(basename $outname)
       gemma -bfile ./1323_cohort_maf01_geno10.snp.pass \
         -p ./pheno_files/$file \
         -k ./1323_cohort_maf01_geno10.cXX.txt \
         -lmm 1 \
         -n 3 \
         -miss 0.1 \
         -o "$outname"
   }
   export -f doit
   ls ./pheno_files | grep -i "noheader" |
       parallel --results output/ doit

/Ole