Re: big file and quoting

[Jay Hacker]

That's a doozy!  Quoting can be a bit tricky with parallel.  I
couldn't figure it out after a few minutes of playing with it either.
You may just want to throw your awk program into a file and have
parallel run that.



On Thu, Sep 8, 2011 at 11:34 AM, LU Zen <zen.lu@roslin.ed.ac.uk> wrote:
> I’m trying to process a big messy csv file with the --pipe option but I keep
> getting errors such as unexpected EOF while looking for matching `"' or awk:
> cmd. line:1:  ^ unexpected newline or end of string. I suspect I’m not using
> the quoting correctly. 2 of the commands I’ve tried:
>
>
>
> $ cat big.csv | parallel --pipe --files 'awk -v FS="\",\"" '{print $1, $3,
> $4, $5, $9, $14}' | grep -v "#" | sed -e '1d' -e 's/\"//g' -e
> 's/\/\/\//\t/g' | cut -f1-6,11 | sed -e 's/\/\//\t/g' -e 's/ /\t/g'' |
> parallel -Xj1 sort -k1 {} ';' rm {} > big_modified_parallel.csv
>
>
>
> $ cat big.csv | parallel --pipe --files 'awk -v FS="\",\"" '{OFS="\\t\"\;
> print $1, $3, $4, $5, $9, $14}'  | grep -v "#" | sed -e '1d' -e 's/\"//g' -e
> 's/\/\/\//\t/g' | cut -f1-6,11 | sed -e 's/\/\//\t/g' -e 's/ /\t/g'' |
> parallel -Xj1 sort -k1 {} ';' rm {} > big_modified_parallel.csv
>
>
>
> Can someone point out where I might have made a mistake?
>
>
>
> Thank you.
>
>
>
>
>
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>