[gnugo-devel] Level 15

[bump]

Since Stefan Mertin informed us that at Level 15 GNU Go 3.2
seems to him greatly stronger than at level 10, I ran the
regressions with the current CVS. It took 16 hours, very
much slower than the current CVS. More precisely, it took
53843 seconds whereas a typical current time is about 8000
seconds.

The results are posted below. I marked a few results with (*)
which are the same as the current CVS.

After correcting for these, there are exact 69 PASS and 69 FAIL.

Needless to say, this raises more questions than it answers.
Who is right, Stefan or the regressions? Does raising the
level improve the strength or not?

Probably the truth is that it does improve the strength,
but whether Stefan's results are the result of that or
statistical fluke remains unclear.

Certainly GNU Go is tuned to do well on these regressions at
level 10. This probably helps keep the program strong but it
be that the regressions are a very poor measure of actually
strength.

Dan

./regress.sh . reading.tst 
65 unexpected PASS!
134 unexpected PASS!
138 unexpected PASS!
154 unexpected PASS!
165 unexpected PASS!
178 unexpected FAIL: Correct '0', got '1 G9'
./regress.sh . owl.tst 
29 unexpected FAIL: Correct '1 C7', got '1 E9'
31 unexpected FAIL: Correct '1 C7', got '1 E9'
80 unexpected PASS!
109 unexpected PASS!
132 unexpected FAIL: Correct '(1|2) R4', got '0'
150 unexpected FAIL: Correct '0', got '1 G13'
166 unexpected PASS!
179 unexpected FAIL: Correct '3 R19', got '0'
262 unexpected PASS!
./regress.sh . owl_rot.tst 
./regress.sh . ld_owl.tst 
./regress.sh . optics.tst 
1201 unexpected FAIL: Correct '1 2 T8 T8', got '0 0'
./regress.sh . filllib.tst 
./regress.sh . atari_atari.tst 
./regress.sh . connection.tst 
52 unexpected PASS!
62 unexpected FAIL: Correct '1 T15|T17', got '1 S15'
./regress.sh . blunder.tst 
./regress.sh . trevora.tst 
200 unexpected FAIL: Correct 'E5', got 'F7'
390 unexpected PASS!
530 unexpected PASS!
531 unexpected PASS!
./regress.sh . nngs1.tst 
16 unexpected FAIL: Correct 'M4', got 'D5'
34 unexpected PASS!
./regress.sh . strategy.tst 
34 unexpected FAIL: Correct 'E17', got 'N15'
./regress.sh . endgame.tst 
./regress.sh . heikki.tst 
./regress.sh . neurogo.tst 
14 unexpected FAIL: Correct 'Q4', got 'M8'
./regress.sh . arb.tst 
./regress.sh . rosebud.tst 
./regress.sh . golife.tst 
./regress.sh . arion.tst 
./regress.sh . viking.tst 
3 unexpected PASS!
./regress.sh . ego.tst 
./regress.sh . dniwog.tst 
6 unexpected PASS!
./regress.sh . lazarus.tst 
4 unexpected FAIL: Correct 'R12|Q12|M8', got 'O11'
9 unexpected FAIL: Correct 'D15|F15', got 'T5'
./regress.sh . trevorb.tst 
120 unexpected PASS!
130 unexpected PASS!
140 unexpected PASS!
440 unexpected PASS!
590 unexpected PASS!
740 unexpected PASS!
960 unexpected PASS!
./regress.sh . strategy2.tst 
55 unexpected PASS!
73 unexpected FAIL: Correct 'F7|R17', got 'P15'
74 unexpected FAIL: Correct 'F7|R17', got 'P15'
77 unexpected FAIL: Correct 'H3', got 'H15'
80 unexpected PASS!
./regress.sh . nicklas1.tst 
501 unexpected FAIL: Correct 'G7', got 'F6'
1216 unexpected PASS!
./regress.sh . nicklas2.tst 
701 unexpected PASS!
1402 unexpected FAIL: Correct 'J8|J6', got 'H8'
2401 unexpected FAIL: Correct 'G3|G2', got 'G6'
./regress.sh . nicklas3.tst 
1403 unexpected PASS!
./regress.sh . nicklas4.tst 
./regress.sh . nicklas5.tst 
./regress.sh . manyfaces.tst 
./regress.sh . niki.tst 
./regress.sh . trevor.tst 
9 unexpected PASS!
220 unexpected FAIL: Correct 'E8', got 'C3'
1060 unexpected PASS!
./regress.sh . tactics.tst 
./regress.sh . buzco.tst 
./regress.sh . nngs.tst 
480 unexpected FAIL: Correct 'G14', got 'F14'
490 unexpected FAIL: Correct 'J18', got 'G18'
720 unexpected PASS!
1020 unexpected PASS!
1140 unexpected FAIL: Correct '1 (A11|B11)', got '1 B10'
1280 unexpected PASS!
1955 unexpected FAIL: Correct 'D3', got 'B3'
./regress.sh . trevorc.tst 
400 unexpected FAIL: Correct '!J11', got 'J11'
910 unexpected FAIL: Correct 'H13', got 'E1'
1540 unexpected FAIL: Correct 'M10', got 'L10'
1650 unexpected PASS!
./regress.sh . strategy3.tst 
119 unexpected PASS!
136 unexpected FAIL: Correct 'E2', got 'B3'
./regress.sh . capture.tst 
4 unexpected PASS!
18 unexpected FAIL: Correct '1 F6', got '0'
./regress.sh . connect.tst 
70 unexpected PASS!
./regress.sh . global.tst 
4 unexpected FAIL: Correct 'Q6', got 'F5'
5 unexpected FAIL: Correct 'O4', got 'F5'
22 unexpected FAIL: Correct 'F2', got 'N11'
./regress.sh . vie.tst 
./regress.sh . arend.tst 
9 unexpected FAIL: Correct 'S17', got 'R15'
31 unexpected FAIL: Correct 'B13|C13', got 'D13'
./regress.sh . 13x13.tst 
20 unexpected FAIL: Correct 'M8', got 'L7'
39 unexpected FAIL: Correct 'H4|J4', got 'J5'
65 unexpected FAIL: Correct 'H4', got 'L2'
72 unexpected FAIL: Correct 'J10', got 'F5'
78 unexpected PASS!
./regress.sh . semeai.tst 
32 unexpected FAIL: Correct 'ALIVE_IN_SEKI ALIVE_IN_SEKI B6', got 'DEAD ALIVE 
PASS'
./regress.sh . trevord.tst 
780 unexpected FAIL: Correct 'B18', got 'D19'
./regress.sh . strategy4.tst 
155 unexpected FAIL: Correct 'D18', got 'P2'
179 unexpected PASS!
183 unexpected FAIL: Correct 'P10|H9', got 'O12'
197 unexpected FAIL: Correct 'K3|S18', got 'G14'
198 unexpected FAIL: Correct 'C10|S18', got 'G14'
199 unexpected FAIL: Correct 'N5|S18', got 'G14'
200 unexpected FAIL: Correct 'P6|P7|Q7|S18', got 'G14'
./regress.sh . owl1.tst 
266 unexpected FAIL: Correct '1 B17', got '1 C18'
283 unexpected PASS!
293 unexpected PASS!
./regress.sh . handtalk.tst 
7 unexpected FAIL: Correct 'R4', got 'P4'
10 unexpected FAIL: Correct 'E9|F8|D8', got 'C11'
./regress.sh . nngs2.tst 
140 unexpected PASS!
240 unexpected FAIL: Correct 'F17', got 'B17'
460 unexpected PASS!
510 unexpected PASS!
./regress.sh . nngs3.tst 
140 unexpected FAIL: Correct 'D9', got 'C10'
160 unexpected PASS!
220 unexpected PASS!
230 unexpected FAIL: Correct '!L12', got 'L12'
280 unexpected FAIL: Correct '!D15', got 'D15'
390 unexpected PASS!
470 unexpected PASS!
700 unexpected FAIL: Correct 'Q2', got 'O4'
710 unexpected FAIL: Correct 'Q2', got 'O4'
820 unexpected FAIL: Correct '!F2', got 'F2'
830 unexpected PASS!
850 unexpected FAIL: Correct 'F9', got 'F8'
(*) 1010 unexpected PASS!
./regress.sh . nngs4.tst 
100 unexpected PASS!
190 unexpected FAIL: Correct 'T8', got 'E2'
230 unexpected PASS!
250 unexpected PASS!
260 unexpected FAIL: Correct 'A4', got 'P8'
./regress.sh . strategy5.tst 
229 unexpected FAIL: Correct 'D7', got 'N10'
236 unexpected PASS!
./regress.sh . century2002.tst 
60 unexpected PASS!
260 unexpected PASS!
./regress.sh . auto01.tst 
13 unexpected FAIL: Correct '1 (Q12|Q13)', got '0'
./regress.sh . auto02.tst 
7 unexpected PASS!
./regress.sh . auto03.tst 
./regress.sh . auto04.tst 
./regress.sh . auto_handtalk.tst 
7 unexpected PASS!
./regress.sh . safety.tst 
./regress.sh . ninestones.tst 
260 unexpected PASS!
./regress.sh . tactics1.tst 
./regress.sh . manyfaces1.tst 
30 unexpected PASS!
33 unexpected FAIL: Correct 'S17|R3', got 'Q3'
36 unexpected PASS!
60 unexpected PASS!
70 unexpected PASS!
90 unexpected FAIL: Correct '!A10', got 'A10'
./regress.sh . gunnar.tst 
19 unexpected PASS!
./regress.sh . arend2.tst 
50 unexpected FAIL: Correct 'S14', got 'T13'
./regress.sh . nando.tst 
2 unexpected PASS!
(*) 18 unexpected PASS! 
(*) 19 unexpected FAIL: Correct '0', got '1'
(*) 20 unexpected FAIL: Correct 'S8|S9|T12', got 'T16'
(*) 21 unexpected FAIL: Correct 'S9|T12', got 'T16'
(*) 22 unexpected PASS!
(*) 23 unexpected FAIL: Correct '0', got '1'
(*) 24 unexpected FAIL: Correct '1 M13', got '0'
(*) 25 unexpected FAIL: Correct '!P16', got 'P16'
26: PASS (unexpected FAIL in current CVS)
(*) 27 unexpected FAIL: Correct 'D6', got 'G5'
(*) 28 unexpected PASS!
(*) 29 unexpected FAIL: Correct '0', got '1'
(*) 30 unexpected FAIL: Correct '1 T6', got '0'
(*) 31 unexpected FAIL: Correct '0', got '1 F19'
150 unexpected PASS!
151 unexpected PASS!
./regress.sh . thrash.tst 
./regress.sh . 13x13b.tst 
1 unexpected FAIL: Correct 'M4', got 'J3'
3 unexpected FAIL: Correct 'M5', got 'PASS'
4 unexpected PASS!
32 unexpected FAIL: Correct 'H4', got 'G5'