RE: [Bug-gnubg] Computer Olympiad

[motiv4u]

1. Report of the 12th Computer Olympiade

2. Matches played
   
-----------------

1. Report
=========

Participants:

MCGammon by François van Lishout and Guillaume Chaslot (Université de Liège,
Belgium)
BgBlitz by Frank Berger
GNU Backgammon by GNU Backgammon development team

a. MCGammon
-----------

MCGammon is based on a Monte Carlo tree, choosing playable moves at random,
playing games from then on, finally choosing the move winning the most
games.
The program is written in C++ and once compiled, plays over 1,000 games a
second.

They only started this project a couple of months ago, and the 12th
Olympiade merely was a testcase for the program itself, not for the strength
of the program.
The implementation was a simple: If I can, I do.

Version 1.0 had problems with combined bearin and bearoff. It choose either
bearin either bearoff, not "seeing" the possibility to bearin ánd bearoff
with the same move.

Version 1.1 could play combined bearin-bearoff moves.

Since the program only looked at possible moves without any theory or
principles or tactics, it played weak. The games played - based on the same
weakness - did not offer a trustable base for decisions.
It sometimes played illegal moves.
The cube was never offered (by design)

In the future, François is thinking of using a neural net to get a list of
possible (strong) candidates, and afterwards doing the 'at random' Monte
Carlo tree. So if I understand well, the Monte Carlo tree will be used to
play games with those candidates and decide wich move to play with the one
winning the most games.

b. BgBlitz
----------

This year, Frank Berger used the new trained TachiAI for both matches.
Compared with the previous version, this one gained aproximately 1.5 %. He
used TD lambda for the training, broadening the nodes to 200.

c. GNU Backgammon (gnubg)
-------------------------

Adrian Wright and Nardy Pillards represented gnubg.
The 20070427 Windows build was used (0.16 devel).

The first matches against MCGammon and BgBlitz (gnubg respectively played by
Adrian and myself) were set to supremo for both chequer play and cube. The
G11 MET was used. The default bearoff databases loaded.

The second match against BgBlitz, I used the two-sided 6x11 bearoff database
and set chequerplay to grandmaster, cube to supremo (world class). Not that
I didn't trusted gnubg's 2-ply play or the embedded bearoff databases, but
after the 4-15 loss in the first match, I thought: "Let's do whatever we
can".

d. Dice generator
-----------------

For the first match with MCGammon, both players (Adrian and François) threw
the dice manually (real dice, real board) and the match was also represented
on a real backgammon board.
The second day - Adrian being caught by a nasty cold - I used the Mersenne
twister of gnubg to throw the dice.

With BgBlitz, Frank's BgBlitz generated the dice (the seed was given by
Nardy) since BgBlitz sometimes is troublesome with given dice.

e. Sitenote
------------

People showed more interest for the real board representation than they did
for the World Class match on screen. Perhaps a thought to keep in mind for
future attendances?


2. Matches played
=================

The matches attached are not analysed. The analysed versions take over 1 MB,
too big to put in the mailing list.
When you do a simple 2-ply analysis, you'll see MCGammon scores porely and
BgBlitz comes out with a 2030+ rating.

One thing stroke my attention: it's remarkable to see that almost all of
BgBlitz's moves marked erronous by gnubg turn out to be the best move after
a rollout!!
Two exceptions: the 9th game of the 2nd match, BgBlitz stayed too long on
the 10 point(look at 0,052 error move 12: a 5-3 roll). And BgBlitz sometimes
does cube too early.

GNU Bacgammon - MCGammon Match 1
--------------------------------
Settings: chequerplay supremo, cubeplay supremo (world class)

MCGammon encountered a problem entering debugmodule, thus MC resigns that
match to allow the MC team to fix it.
(Adrian has the records of that match)

GNU Backgammon - BgBlitz Match 1
--------------------------------
Settings: checkerplay supremo, cube supremo (world class)
Default bearoff databases
15 pt match

0       2       move 11 and 12 of BgBlitz: very lucky
0       4       move 18: a miraculeous 4-2 roll from the bar
0       6       move 13: very unlucky 5-3 for gnubg followed by a cube
0       10      
4       10      a very funny 5-5 for BgBlitz, cost: a gammon (last move)
4       12      opening 5-2: do a rollout and see: best move!!
                last move of gnubg: not even playing off the 1
4       15      the 4-1 opening remains an error after rollout

2033 rating for BgBlitz

First game: errorfree play by BgBlitz, except a 5-2 opening.

GNU Backgammon - BgBlitz Match 2
--------------------------------
Settings: checkerplay grandmaster, cube supremo (world class)
Two Sided 6x11 bearoff database
15 pt match

2       0       a very unlucky 4-4 for BgBlitz (move 15)
2       2       move 3: cube error (0,063)
2       3       move 15: a beautifull 6-6 roll, getting the game
2       5
gnubg crashed at the end of the 4th game. I have no idea why, since this is
the first crash of the 20070423 while playing.
The game got restored from the saved match (not a bad idea to click 'Save'
from time to time)

2       7       
3       7       
5       7       Move 21, 22: unlucky 6-4 and lucky 2-1
7       7       
7       9       BgBlitz errors 6-4 and 5-3 (move 11, 12)
11      9       move 4: a great 2-2 roll consolidated in win
11      10      move 7: joker roll for BgBlitz
                move 17: a 6-6 for gnubg saving the gammon!
11      12      miraculous 2-5 followed by a whiny 6-2 :-(
11      13      BgBlitz plays the blitz
11      15      move 8: 2-1 from the bar, gnubg dances with 6-1
                move 25: and gnubg dances again...

GNU Bacgammon - MCGammon Match 2
--------------------------------
Settings: chequerplay supremo, cubeplay supremo (world class)
15 pt match

4       0       
6       0       
8       0       
12      0       
15      0       the cube was offered to finish the match

Although MCGammon does not cube, François offered the cube in the last game
to finish the match.


The first match played against MCGammon was rated (if I recall correctly)
982. Adrian will correct me here if I'm wrong.

MCGammon got some overnight adaptions, putting more weight to basic
principles, the computer used was three times faster, so in stead of playing
80000 games, MCGammon played 200000 games.
But still the rating stayed below 1,000


Nardy 


No virus found in this outgoing message.
Checked by AVG Free Edition. 
Version: 7.5.472 / Virus Database: 269.9.0/853 - Release Date: 18/6/2007
15:02

olympiade_2007-gnubg-MCGammon_15p_2007-06-16_01.sgf
Description: Binary data

olympiade_2007-gnubg-BgBitz_15p_2007-06-15_01.sgf
Description: Binary data

olympiade_2007-gnubg-BgBitz_15p_2007-06-15_02.sgf
Description: Binary data