[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

improve performance of a script

From: xeon Mailinglist
Subject: improve performance of a script
Date: Tue, 25 Mar 2014 07:12:16 -0700 (PDT)
User-agent: G2/1.0

For each file inside the directory $output, I do a cat to the file and generate 
a sha256 hash. This script takes 9 minutes to read 105 files, with the total 
data of 556MB and generate the digests. Is there a way to make this script 
faster? Maybe generate digests in parallel?

for path in $output
    # sha256sum
    digests[$count]=$( $HADOOP_HOME/bin/hdfs dfs -cat "$path" | sha256sum | awk 
'{ print $1 }')
    (( count ++ ))


reply via email to

[Prev in Thread] Current Thread [Next in Thread]