Re: [Bug-gnubg] Snowie 4 vs. GNU 0.13

[Joern Thyssen]

On Mon, Jun 09, 2003 at 01:06:40PM -0300, Albert Silver wrote
> 
> > The average of the luck adjustment values (for all 100 matches) is
> > 0.500, i.e., the bots are equally good.
> 
> Is that official or are you just saying this out of argument? 

It's official.

> If it IS offical could you post some numbers (such as how this was
> reached) so that I might share this with GOL and RGB readers?

For each match I analysed the luck at 0-ply:

set priority idle
import mat ...
set analysis cube off
set analysis moves off
analyse match
show stat match

I extracted the total luck rate from the match statistics

Player                          Snowie4                 gnubg
Luck rate (total)               -1.501 (-49.717%)       -0.341 ( -1.415%)

In the example above  gnubg won, so the luck adjusted result is

100% - 1.415% - ( -49.717% ) = 51.698%

I averaged all the luck adjusted results: 49.9947% with standard
deviation 13.5%, hence the 95% confidence interval is 50.0% +/-2.7%
(13.5% * 1.96/sqrt(100) = 2.7%).

I've attached a file which has the following entries:

game number, actual result, gnubg luck, snowie luck, luck adjusted result

For example, for game 100 (my example above):

100 1 -.01415 -.49717 .51698

Some of the games could be very interesting to inspect carefully. For
bots of similar strength we expect luck adjusted results around 50%.
However, this is not always true in the 100 match sample you've sent me:

Examples:

8 0 .19603 .19853 .00250
39 1 .07856 .00087 .92231

Either gnubg's luck analysis is totally wrong or snowie (gnubg)
played very bad in game 39 (game 8). 

Match 39 re-analysed:

0-ply:   1 .07856 .00087 .92231
1-ply:   1 .1150 -.0151  .87449
2-ply:   painfully slow; I gave up

The result is changed by 5%, but we're still far from a luck adjusted
result of 50%. I can't explain this...

Jørn