gnugo-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [gnugo-devel] auto generated regression tests


From: Arend Bayer
Subject: Re: [gnugo-devel] auto generated regression tests
Date: Wed, 14 Aug 2002 16:36:57 +0200 (CEST)

On Sat, 3 Aug 2002, Evan Berggren Daniel wrote:

> attached are a collection of games and regression tests.  They all are
> based on matcher_status inconsistencies from the matcher_check.pl script.
> Expected passes are marked relative to gnugo 3.2, however gnugo 3.3.5
> gains at least one test.  Some of these I have analyzed and put in checks
> on owl_attack, etc, but most are just based on dragon_status.  I'm not
> sure all the tests are correct (I may have read things wrong), so someone
> else should probably look them over.
>
> Anyway, hope these help.

I have looked through the tests and I think they are pretty useful.

As a general comment, "dragon_status" takes a lot more time than
"owl_defend/attack". I am all in favour of adding a lot more of
these semi-automatic tests to the regression, but we don't want the
regressions run to take unnecessariyl longer, so I'd suggest to use the
latter. (I.e., instead of
X dragon_status XX
#? [!dead]

we can use

X owl_defend XX
#? [!0]

or even better

X owl_defend XX
#? [1 (list of working defense moves)]

There were two or three tests where I came to a different conclusion.
These changes, plus the conversion away from using dragon_status, are
in the attached .tgz-file as well as in the patch below.

As you have already pointed out in another e-mail, this type of self-play
catches a particular type of owl mistake. I would think that analyzing
games against humans in the same manner would be extremely useful, too,
and possibly catch a bigger variety of mistakes.

> Also, has anyone tried running the matcher_status script?  does gnugo
> finish games completely when you do?

Did GNU Go play the passes at the "end" of auto010.sgf?? That would be
very weird, and it doesn't happen when I let GNU Go generate a move
there. Would be quite a bug.

Arend



diff -u gnugo-3.2/regression/auto01.tst autotests/regression/auto01.tst
--- gnugo-3.2/regression/auto01.tst     Fri Aug  2 19:15:23 2002
+++ autotests/regression/auto01.tst     Wed Aug 14 16:11:01 2002
@@ -1,59 +1,63 @@
 #auto generated owl inconsistencies
 loadsgf games/auto_gen/auto001.sgf 136
-1 dragon_status p11
-#? [!dead]*
+1 owl_defend p11
+#? [1 Q11]*

 loadsgf games/auto_gen/auto001.sgf 139
-2 dragon_status r16
-#? [!dead]*
+2 owl_attack r16
+#? [1 (O11|M10)]*

 loadsgf games/auto_gen/auto001.sgf 140
-3 dragon_status r16
-#? [!dead]*
+3 owl_defend r16
+#? [1 (O11|T15|T13)]*

 loadsgf games/auto_gen/auto002.sgf 186
-4 dragon_status n8
-#? [!dead]*
+4 owl_defend n8
+#? [1 (R1|N3|N5|M5|N6)]*

 loadsgf games/auto_gen/auto003.sgf 52
-5 dragon_status s9
-#? [!dead]*
+5 owl_defend s9
+#? [1 Q10]*

 loadsgf games/auto_gen/auto004.sgf 9
-6 dragon_status f3
-#? [!dead]*
+6 dragon_defend f3
+#? [!0]*

-loadsgf games/auto_gen/auto005.sgf 29
-7 dragon_status f17
-#? [!dead]*
+# This is a duplication of the previous one. /ab
+# loadsgf games/auto_gen/auto005.sgf 29
+# 7 dragon_status f17
+# #? [!dead]*

 loadsgf games/auto_gen/auto006.sgf 25
-8 dragon_status g4
-#? [!dead]*
+8 owl_defend g4
+#? [1 (H5|J4|G5)]*

-loadsgf games/auto_gen/auto006.sgf 78
-9 dragon_status m17
-#? [critical]*
-
-loadsgf games/auto_gen/auto006.sgf 82
-10 dragon_status m17
-#? [critical]*
+# I think M17 is dead. After White H18, black can e.g. play L18. /ab
+# loadsgf games/auto_gen/auto006.sgf 78
+# 9 dragon_status m17
+# #? [critical]*
+
+# M17 lookd dead. /ab
+# loadsgf games/auto_gen/auto006.sgf 82
+# 10 dragon_status m17
+# #? [critical]*

 loadsgf games/auto_gen/auto006.sgf 81
 11 owl_attack m17
-#? [j18]*
+#? [1 J18]*

 loadsgf games/auto_gen/auto006.sgf 94
-12 dragon_status r12
-#? [!alive]*
+# This is duplicated by the next test.
+# 12 dragon_status r12
+# #? [!alive]*

+# P13 probably does not work. (P13-Q12-P12-Q9-Q11-Q7 and black should win
+# the semeai.) /ab
 13 owl_attack s11
-#? [q12|q13|p13]*
-
-loadsgf games/auto_gen/auto006.sgf 121
-14 dragon_status s9
-#? [!alive]*
+#? [1 (Q12|Q13)]*

-15 owl_attack s9
-#? [!0]*
+# This is a difficult semeai problem.
+loadsgf games/auto_gen/auto006.sgf 99
+15 owl_analyze_semeai S9 S10
+#? [ALIVE DEAD S6]*

diff -u gnugo-3.2/regression/auto02.tst autotests/regression/auto02.tst
--- gnugo-3.2/regression/auto02.tst     Sat Aug  3 00:40:49 2002
+++ autotests/regression/auto02.tst     Wed Aug 14 16:14:37 2002
@@ -1,43 +1,45 @@
 loadsgf games/auto_gen/auto007.sgf 20
-1 dragon_status q13
-#? [!dead]*
+1 owl_defend q13
+#? [1 (Q11|P12)]*

 loadsgf games/auto_gen/auto007.sgf 42
-2 dragon_status q13
-#? [!dead]*
+2 owl_defend q13
+#? [!0]*

+# C16 is probably best. /ab
 loadsgf games/auto_gen/auto008.sgf 58
-3 dragon_status d16
-#? [!dead]*
+3 owl_defend d16
+#? [1 (C16|D17|E15)]*

 loadsgf games/auto_gen/auto008.sgf 150
-4 dragon_status c12
-#? [!alive]*
+4 owl_attack c12
+#? [1 (A14|A12)]*

+# Good problem for connection-related owl tuning.
 loadsgf games/auto_gen/auto008.sgf 180
-5 dragon_status a15
-#? [!dead]*
+5 owl_defend a15
+#? [1 (B9|D9)]*

 loadsgf games/auto_gen/auto009.sgf 94
-6 dragon_status f3
-#? [!dead]*
+6 owl_defend f3
+#? [!0]*

 loadsgf games/auto_gen/auto010.sgf 66
-7 dragon_status q17
-#? [!alive]*
+7 owl_attack q17
+#? [1 (S15|R15)]*

 loadsgf games/auto_gen/auto010.sgf 64
-8 dragon_status q17
-#? [!alive]*
+8 owl_attack q17
+#? [!0]*

 loadsgf games/auto_gen/auto010.sgf 77
-9 dragon_status c18
-#? [!alive]*
+9 owl_attack c18
+#? [D18]*

 loadsgf games/auto_gen/auto011.sgf 43
-10 dragon_status d9
-#? [!dead]*
+10 owl_defend d9
+#? [!0]*

 loadsgf games/auto_gen/auto012.sgf 168
-11 dragon_status h15
-#? [!dead]*
+11 owl_attack h15
+#? [0]*
diff -u gnugo-3.2/regression/auto03.tst autotests/regression/auto03.tst
--- gnugo-3.2/regression/auto03.tst     Sat Aug  3 04:59:24 2002
+++ autotests/regression/auto03.tst     Wed Aug 14 15:45:25 2002
@@ -1,47 +1,50 @@
 loadsgf games/auto_gen/auto013.sgf 47
-1 dragon_status h16
-#? [!dead]*
+1 owl_defend h16
+#? [!0]*

 loadsgf games/auto_gen/auto013.sgf 51
-2 dragon_status h16
-#? [!dead]*
+2 owl_defend h16
+#? [!0]*

 loadsgf games/auto_gen/auto013.sgf 53
-3 dragon_status l17
-#? [!alive]*
+3 owl_attack l17
+#? [K17]*

+# This is a connection problem. /ab
 loadsgf games/auto_gen/auto014.sgf 106
-6 dragon_status l14
-#? [!unknown]*
+6 owl_attack l14
+#? [N12]*

 loadsgf games/auto_gen/auto014.sgf 107
 7 owl_defend l14
 #? [n12]*

 loadsgf games/auto_gen/auto015.sgf 145
-8 dragon_status t7
-#? [!alive]*
+8 attack t7
+#? [T5]*

 loadsgf games/auto_gen/auto016.sgf 37
-9 dragon_status j15
-#? [!dead]*
+9 owl_defend j15
+#? [!0]*

 loadsgf games/auto_gen/auto016.sgf 48
-10 dragon_status p16
-#? [!dead]*
+10 owl_defend p16
+#? [O16|P15]*

 loadsgf games/auto_gen/auto016.sgf 53
-11 dragon_status q4
-#? [!dead]*
+11 owl_defend q4
+#? [P4|P3|Q3|R4]*

 loadsgf games/auto_gen/auto016.sgf 55
-12 dragon_status n5
-#? [!dead]*
+12 owl_defend n5
+#? [!0]*

-loadsgf games/auto_gen/auto017.sgf 15
-13 dragon_status c13
-#? [!alive]*
+# I don't think black can kill C13. Of course, white should still answer
+# the black attachment, hence this might be worth a genmove test instead. /ab
+# loadsgf games/auto_gen/auto017.sgf 15
+# 13 dragon_status c13
+# #? [!alive]*

 loadsgf games/auto_gen/auto017.sgf 45
-14 dragon_status q4
-#? [!dead]*
+14 owl_defend q4
+#? [!0]*
diff -u gnugo-3.2/regression/auto04.tst autotests/regression/auto04.tst
--- gnugo-3.2/regression/auto04.tst     Sat Aug  3 20:11:45 2002
+++ autotests/regression/auto04.tst     Wed Aug 14 15:55:05 2002
@@ -1,45 +1,27 @@
 loadsgf games/auto_gen/auto018.sgf 103
-1 dragon_status r14
-#? [!alive]*
+1 owl_attack r14
+#? [!0]*

 loadsgf games/auto_gen/auto019.sgf 76
-2 dragon_status e18
-#? [!dead]*
+2 owl_defend e18
+#? [!0]*

 loadsgf games/auto_gen/auto020.sgf 67
-3 dragon_status n4
-#? [!dead]*
-
-4 owl_defend n4
-#? [m5]*
+3 owl_defend n4
+#? [M5|L4|N5|L5]*

 loadsgf games/auto_gen/auto021.sgf 72
-5 dragon_status g15
-#? [dead]
-
-6 owl_defend g15
-#? [!j14]
-
-7 owl_does_defend j14 g15
+7 owl_defend j14
 #? [0]*

 loadsgf games/auto_gen/auto021.sgf 71
-8 dragon_status g15
-#? [critical]*
-
-9 owl_does_attack k15 g14
-#? [1]*
+8 owl_attack g15
+#? [K15]*

 loadsgf games/auto_gen/auto022.sgf 37
-10 dragon_status e18
-#? [!dead]*
-
 11 owl_defend e18
-#? [d17]*
+#? [D17|F12|G14|G18|H18]*

 loadsgf games/auto_gen/auto022.sgf 136
 12 owl_attack g4
-#? [d2]*
-
-13 dragon_status g4
-#? [critical]*
+#? [D2]*
Common subdirectories: gnugo-3.2/regression/games and autotests/regression/games

Attachment: auto_tests.tgz
Description: GNU Unix tar archive


reply via email to

[Prev in Thread] Current Thread [Next in Thread]