bug-bash
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: using mapfile is extreamly slow compared to oldfashinod ways to read


From: Lennart Schultz
Subject: Re: using mapfile is extreamly slow compared to oldfashinod ways to read files
Date: Sat, 28 Mar 2009 09:19:08 +0100

It seems that mapfile is OK for small numbers but for bigger numbers it
starts to compsume time.

I made a litle test:

rm Xyz; unset MAPFILE # clear
max=9999  # set limit
time for i in $(seq 0 $max); do echo 'Xyz' >> Xyz; done
real    0m0.490s
user    0m0.304s
sys     0m0.124s

 time mapfile < Xyz

real    0m0.005s
user    0m0.008s
sys     0m0.000s

time while read line; do echo $line > /dev/null; done < Xyz
real    0m1.124s
user    0m0.456s
sys     0m0.108s

time for i in $(seq 0 $max); do echo echo ${MAPFILE[$i]}> /dev/null; done

real    0m2.184s
user    0m0.976s
sys     0m0.104s

rm Xyz ;unset MAPFILE
max=99999

 time for i in $(seq 0 $max); do echo 'Xyz' >> Xyz; done

real    0m8.204s
user    0m3.264s
sys     0m1.188s

time mapfile < Xyz

real    0m0.062s
user    0m0.044s
sys     0m0.000s

time while read line; do echo $line > /dev/null; done < Xyz
real    0m11.328s
user    0m4.500s
sys     0m1.140s

time for i in $(seq 0 $max); do echo echo ${MAPFILE[$i]}> /dev/null; done

real    9m52.832s
user    5m38.305s
sys     0m3.636s


At the time of testing I had sufficient of free memory no swapping, and no
othe time compsuming programs.


2009/3/28 Chris F.A. Johnson <address@hidden>

> On Fri, 27 Mar 2009, Lennart Schultz wrote:
>
>  Chris,
>> I agree with you to use the right tool at the right time, and mapfile
>> seems
>> not to be the right tool for my problem, but I will just give you some
>> facts
>> of my observations:
>>
>> using a fast tool like egrep just to find a simple string in my datafile
>> gives the following times:
>>
>> time egrep '<pro' >/dev/null < dr.xml
>>
>> real    0m54.628s
>> user    0m27.310s
>> sys     0m0.036s
>>
>> My original bash script :
>>
>> time xml2e2-loadepg
>>
>> real    1m53.264s
>> user    1m22.145s
>> sys     0m30.674s
>>
>> While the questions seems to go on spawning subshells and the cost I have
>> checked my script
>> it is only calling one external command is date which in total is called a
>> little less than 20000 times. I have just for this test changed the call
>> of
>> date to an assignment of an constant. and now it looks:
>>
>> time xml2e2-loadepg
>>
>> real    1m3.826s
>> user    1m2.700s
>> sys     0m1.004s
>>
>> I also made the same change to the version of the program using mapfile,
>> and
>> changed  line=$(echo $i) to
>> line=${i##+([[:space:]])}
>> so the mainloop is absolulty without any sub shell spawns:
>>
>> time xml2e2-loadepg.new
>>
>> real    65m2.378s
>> user    63m16.717s
>> sys     0m1.124s
>>
>
>   How much of that is taken by mapfile? Time the mapfile command and
>   the loop separately:
>
> time mapfile < file
> time for i in "address@hidden"
>
> --
>   Chris F.A. Johnson, webmaster         <http://woodbine-gerrard.com>
>   ===================================================================
>
>   Author:
>   Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
>


reply via email to

[Prev in Thread] Current Thread [Next in Thread]