Re: JSD question [WAS : [Bug-gnubg] GNU Question]

[Jim Segrave]

On Tue 02 Sep 2003 (10:47 +0200), address@hidden wrote:
> 
> >>J.s.d. gives the standard error of the difference between two normal
> >>distributions and is calculated as the square root of the sum of the
> >>squares of the standard errors of the two options being compared.
> >
> >Ok, this is exactly the stderr of the difference distribution (assuming
> >the two are normal distributions).
> 
> Just another detail : if you compute the jsd as you said, you implicitly
> assume the the two equity distributions (assumed normal) are independent.
> In such a case Var(X-Y) = Var(X)+Var(Y), and the computation of the stderr
> described above is fine.

Yes the assumption being made is that you run enough trials that the
result approximates a normal distribution and that the results of the
trials are independant.

In a sense I have simply automated what many users of backgammon
programs already do, which is to see if the equities given by the
rollout are far enough apart that even with the standard error being
reported the result is still probably correct.

For any given rollout, it's hard to say how many trials will be
required (if it ever even occurs) such that the result will be a
normal distribution. I think that the rollouts of different moves are
independant with the main problem area being if gnubg has systematic
errors handling some particular feature of a rollout.

> But I'm not sure this independency assumption is reasonable : I would
> compute the estimate of the stderr of the difference of the two equity
> by it's definition (just like you compute the estimates of the stderr
> of the equities, but on the equity difference). The estimate would then
> include eventual correlation terms and would be equivalent (for a number
> of samples large enough) to the computation whenever the distributions
> were really independent.

Are you suggesting treating a rollout of two different moves for n trials
as n pairs of results (or n occurrences of a value for the equity
difference) which would then be treated as an evaluation on it's own? 

This could be done, although for a rollout of more than two options,
you'd want to be keeping the equity differences for every pair of
options, as you don't know which one will be the 'best' which will be
used as the norm for all the comparisions (few people will care about
the relative ranking of the 3rd and 4th best plays, only the ranking
vs. the best play).
 
> It will be interesting to :
> 
> - show this to one of the gurus of the GNUbg team (feel free to froward
>   this message to anybody may be interested)

CCed to bug-gnubg for general discussion. 

> - eventually (if anyone else shares my doubts), run some test

I think it's not to hard to do the above (assuming that's what you're
suggesting) the hardest part is probably finding a way to display the
results.

-- 
Jim Segrave           address@hidden