[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 2.13.40 regtests

From: Phil Holmes
Subject: Re: 2.13.40 regtests
Date: Fri, 26 Nov 2010 10:02:51 -0000

"Graham Percival" <address@hidden> wrote in message news:address@hidden
On Fri, Nov 26, 2010 at 12:14:17AM +0000, Neil Puttock wrote:
On 26 November 2010 00:00, Graham Percival <address@hidden> wrote:

> Hmm. It shouldn't take a huge amount of time to compare each pair
> of regtest images -- they're named, so you'd be comparing
> something like 500 pairs of .png images. (Neil: were you thinking
> of something else?)

I think this would be very slow unless it does something similar to
ouput-distance.py to reduce the number: you don't want to be comparing
every single pair unless absolutely necessary.

Hmm.  Average png is, what, 400x50 pixels?  20kb?  So the program
needs to load 40kb from disk, then do 20,000 comparisons between
pixels.  We have roughly 500 regtests, so that gives us 10 million
pixel comparisons.

I wouldn't have thought that this would be a problem for a modern
computer, but I must admit that I've never done any graphics
processing.  But if the pixels are in RGB format, then my first
guess would be to compare each color format, and see if the
numbers differ by more than a certain amount.  Something like
 int changed = 0;
 for (int c=0; c<2; c++) {
   if (abs(first.color[c] - second.color[c]) > 0.0001) {
     changed = 1;

Running that code 30 million times shouldn't take too long.  I
think?  I might be off by an order of magnitude, but my violin
physical modeling can generate 30 seconds of audio (that's 1.3e6
samples) in less than 2 seconds of wall-clock, and computing each
audio sample is a lot more complicated than the above code.
Granted, the violin stuff is C++ code, so I'd expect python or C#
to be slower.

My main concern is still with false positives.

While test bugfixes, I've occasionally produced regression checks
which have affected lots of tests, and the time for `make check' in
such cases rises drastically (from about five minutes to over half an
hour), so unless Phil's pixel-by-pixel comparison is much faster than
`compare', it's still going to be too slow.

Oh, I was only thinking about this being used for special cases
and in the release-build process.  I don't think we need to add
this to the usual "make check".

- Graham

On my machine, a 2.3 GHz quad core, single threaded (i.e. just one core active) a check of the full sized PNGs takes about 2 seconds CPU time. Generating the image with LilyPond takes 1.2 seconds. This is with full-sized PNGs of 835x1181, so an intelligent re-size of the images would speed it up lots. Even without that, and 500 images to check, we're only talking 250 seconds to check the lot if we using multi-threading. Don't think that's a real problem per release.

I'm not too worried about false positives - the colour coding will let a person see the diffs really quickly, so a little shift will be obvious.

I think the best bet is that I'll knock something up and run it over .41 when it arrives. We can then see whether it's of use before working out whether and how to progress it.

Phil Holmes
Bug Squad

reply via email to

[Prev in Thread] Current Thread [Next in Thread]