Re: [gnugo-devel] Timing data

[Arend Bayer]

A few remarks:

> All regressions, current CVS:
> 
> alt and exp conn and exp influence: 29108 seconds
> alt and exp conn                  : 27911 seconds
> 
> The penalty for the experimental influence seems to be about 4.2%

I have just finished a different timing test of --experimental-influence vs
--standard-influence via a replay of games. Since the persistent caching
is of no use in the regression test setting, I might have expected a
different result. However, I also got exactly 4.2% time cost for
experimental mode, varying between 1.2% and 6.7% for different games
(details below).

> I have more specific timing information for strategy.tst alone. 
> Inge pointed out that I should really use time to get the exact
> CPU time but since comment came after these experiments
> were already underway and I didn't want to switch my methodology
> in the middle. Next time I'll do it Inge's way.

Incidentally, I had used a script that reported both the 'time' output
and measured the time itself by reporting the date before and after the
test. In my setting (abolutely minimalistic Linux - no X, hardly any
drivers, just one bash), I did not get any differences. In theory, one
could use 'time' to do performance tests with GNU Go running in
background. However, I personnaly would not safely assume that this works
very reliably for a program like GNU Go.

Btw, I also did all replays twice, to measure the accuracy of this test
method. The variance I got was of magnitude ~0.2%, very reasonable.

> 
> And perhaps we should think about releasing 3.2 soon.
> The question is, if we put out a version which is 45%
> slower than 3.0, is that acceptable?

The speed of PCs still gets doubled every 15-18 months. I'd say
we could allow GNU Go to get slower at the same rate. Then 45% would not
be too far off.

OTOH, I think there are still some speed-ups we can do in the short term.


Arend


Resulst:

stand infl      exper. infl     diff abs.       diff in %

911s            937s            26s             2.8%
651s            689s            37s             5.8%
1038s           1062            24s             2.3%
951s            1007s           56s             5.9%
2363s           2467s           104s            4.4%
1613s           1718s           105s            6.5%
2938s           2975s           37s             1.2%
1572s           1678s           106s            6.7%
752s            796s            44s             5.8%

Sum:
12789s          13329s                          4.2%