[gnugo-devel] 9x9 opening stats against humans

[Douglas Ridgway]

Since self play doesn't seem like a good way to test move quality, I took
a look at play against people. I downloaded all May 2004 games from all
the KGS bots running various versions of GnuGo [1], and selected 9x9 games
with no handicap and komi between 5.5 and 8.0. I did no selection based on
opponent. I looked at win frequency by opening pattern, and also looked
for opening patterns played by pros [2] which were not played at all.

It's much more common for the human to be black than white in even games,
no doubt due to the GTP client setting up the game offer. There are 1276
games with GnuGo playing white, compared to 61 with black. This means more
data is available on play as white.

GnuGo as Black won 54% overall. By opening move this breaks down as

5-5: 12 - 9, 57%
3-3: 10 - 7, 59%
3-4: 4 - 9, 31%
4-4: 4 - 2, 67%
4-5: 3 - 1, 75%
3-5: missing

The differences by opening move look substantial, but there aren't enough
games to be statistically significant. Still, the relative success with
3-3 and failure with 3-4 seems surprising to me -- I'd have thought 3-3 to
be mediocre, and 3-4 solid.

GnuGo as white won 60% overall, doing best against a 3-3 opening, worst
against 4-4. The differences between black opening moves is significant, 
but the differences in responses, though sometimes substantial (why a 3-3 
invasion of 4-4 opening on 9x9?), do not reach significance.

5-5: 69 - 49, 58%
  best: 3-3 (67%)
  worst: 5-4 (38%)

3-3: 373 - 116, 76%
  best: 7-7 (85%)
  worst: 5-5 (64%)
  missing: 7-3
  weird: 3-4 (71%)

3-4: 176 - 152, 54%
  best: 7-6 (65%)
  worst: 5-4 (40%)
  missing: 6-5 (Fuseki33, most pop among pros), 5-6, 7-5, 3-6, 4-5

4-4: 145 - 196, 43%
  best: 5-5 (64%)
  worst: 3-3 (25%)
  successful experiment: 7-4 (100%, 1/1)
  missing: 5-7

Comments:

  * Pros do indeed play moves that the amateurs in the original training 
set played rarely or never.

  * It will take a lot of data to prove mediocre moves as definitively 
worse.

  * The move weighting, based on popularity in the original training set,
seems problematic. First, since a deweighted move is played more rarely,
it takes more games to figure out if it's better or worse than the higher
rated moves. Second, the more moves are weighted, the more predictable 
they become. And finally, it looks like some of the best moves are played 
fairly rarely, so changing the weights could improve average play.

I hope to figure out how to add the pro game patterns to the existing set. 
Would anyone mind if I changed the weights at the same time? Also, does 
anyone have the original human-human 9x9 training set around?

I also have lots of 19x19 games if anyone wants them. Let me know.

doug.

[1] http://senseis.xmp.net/?KGSBots
[2] http://senseis.xmp.net/?9x9Openings