RE: [Bug-gnubg] Confidence intervals from rollouts

[David Montgomery]

Doug Zare wrote:
> Ok, I think it would be useful to clarify the definitions we are
> encountering
> here. I see at least three variances. First, there is an actual
> variance of the
> estimated equity of a rollout scheme. Second, and we can estimate
> this by the
> variance of a variable uniformly distributed among the trials of a single
> rollout, with or without the n/(n-1) adjustment. Third, we can
> talk about the
> average value of the latter quantity through some rollout scheme,
> as opposed to
> the actual output of the estimate in one rollout. (It's possible
> that another
> measure of the second quantity makes more sense, like the average of the
> squareroots.)
>
> I'm going to call these the real variance, the observed variance,
> and the ideal
> sample variance. Please feel free to override these with better names; I
> research probability (among other things) but I'm not very familiar with
> statistics, and certainly not statistical conventions.

I can think of at least two other variances, so I'm going to
describe them and suggest an alternate naming scheme.

We can also repeat a rollout of n games many times, and
estimate the real variance from that.  And we can talk about
the expected value of this variance (or the squared expected
value of the square root, whatever.)  These two cases are analogous
to Doug's last two cases, except that instead of looking *within*
-- considering the individual trials -- to estimate the variance,
we look outside, to multiple runs of the same experiment.

Repeating an n-game rollout many times to calculate the
variance certainly feels like an "observed" variance to
me, so I'll suggest

- real variance
- sample internal variance (above's observed variance)
- expected internal variance (above's ideal sample variance)
- sample external variance
- expected external variance

Hmmm... I guess maybe we don't want 5, because "expected
external variance" *is* the real variance.

"Sample external variance" is an unbiased estimate of the
real variance, regardless of the variance reduction techniques
we used.

But both "internal" variances can be biased wrt to the real
variance, depending on what techniques we use.

David