lmi
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [lmi] [lmi-commits] master 9c510ad 16/22: Measure elapsed time for M


From: Vadim Zeitlin
Subject: Re: [lmi] [lmi-commits] master 9c510ad 16/22: Measure elapsed time for MD5 data-file validation
Date: Mon, 30 Mar 2020 17:34:53 +0200

On Mon, 30 Mar 2020 13:26:42 +0000 Greg Chicares <address@hidden> wrote:

GC> On 2020-03-30 00:34, Greg Chicares wrote:
GC> > On 2020-03-29 17:58, Vadim Zeitlin wrote:
GC> [...]
GC> >>  In fact, the main reason for writing this reply is that I'm surprised by
GC> >> how long it takes to compute the hash. Using "openssl speed md5", I get
GC> >> ~300MB/s for 64 byte blocks and while I don't know how big the files 
we're
GC> >> verifying are exactly, I'm pretty sure that they're nowhere close to 30MB
GC> >> in size. So I wonder if it could be useful to profile the code doing this
GC> >> calculation to check if we're not doing something stupid?
GC> > 
GC> > Sounds worthwhile.
GC> 
GC> Here's evidence that suggests our self-built standalone md5sum program
GC> is as fast as the one in gnu coreutils.

 Yes, I can confirm this. I didn't have time to check if it can be made
faster however.

GC> Summary:
GC> 
GC>   clock
GC>    time  program used with "maximal" dataset
GC>   ----- ------------------------------------
GC>   0.194 wine lmi_md5sum.exe [i686]
GC>   0.190 wine lmi_md5sum.exe [x86_64]
GC>   0.200 wine md5sum.exe [i686 binary from the following URL]
GC>     
https://github.com/vadz/lmi/releases/download/new-cygwin-makefiles/md5sum.exe
GC>   0.085 debian native md5sum

 FWIW, under native MSW 7 eraseme_lmi_md5sum.exe in the archive you
provided is slightly but noticeably faster than Cygwin md5sum.exe: the zsh
command line

        % time (repeat 10 $MD5SUM *.xyz | md5sum)

takes 1.2s with MD5SUM=./eraseme_lmi_md5sum.exe compared to 1.7s with
MD5SUM=md5sum. Surprisingly, even though it's a faster machine with an SSD,
it's still much slower than under slower Linux machine where it takes 0.61s
with both standard md5sum and lmi_md5sum (built with -O2, it's much slower
without optimizations, of course). This might say more about the
comparative speed of Linux and MSW IO subsystems than anything else
however.

 I thought we could decrease the time further by running several processes
in parallel, but this doesn't help -- it looks like the overhead of
launching a process is too high for such small tasks, even under Linux, and
even if I use just 8 (== number of cores) processes in total. I'd like to
explore using several threads for executing this in parallel inside a
single process, normally this should result in a noticeable gain and this
is supposed to be simple to do in modern C++, in theory (but things have an
annoying tendency to work somewhat differently in practice, so we'll see).

 Regards,
VZ

Attachment: pgp0xg1u3HTyL.pgp
Description: PGP signature


reply via email to

[Prev in Thread] Current Thread [Next in Thread]