[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Bug-gnubg] Re: Rollout jsd, statsig etc. [LONG]
From: |
Timothy Y. Chow |
Subject: |
Re: [Bug-gnubg] Re: Rollout jsd, statsig etc. [LONG] |
Date: |
Tue, 17 Nov 2009 23:58:39 -0500 (EST) |
On Tue, 17 Nov 2009, Massimiliano Maini wrote:
> If the above is true, on top of assuming a uniform prior, you assume
> that the real pdf are the ones you have on your last step.
> This is not far from assuming that the real equities are the estimated
> ones. Duh, I must be missing something ...
Maybe a simplified example will help.
I fill a bag with 1000 marbles, some white and some black, and hand the
bag to a group of four people: Martha the Mathematician, Bob the Bayesian,
Helen the Hypothesis tester, and Stan the Statistics Student. I invite
them to sample 10 marbles at random from the bag. They do so; 8 are white
and 2 are black. I ask, "What is the probability that I put more black
marbles than white marbles in the bag?"
Martha complains, "Unless we make some further assumptions, your question
is unanswerable. Either you put more black marbles than white marbles in
the bag or you didn't. There's no probability calculation to do here."
Bob says, "You're right, Martha; we need to assume something. Let's
assume that Tim picked a number from 0 to 1000 uniformly at random, and
put that many white marbles in the bag, filling the rest of the bag with
black marbles. Given that assumption, and given that our random sample
has produced 8 white marbles and 2 black marbles, can you now calculate,
for each i from 0 to 1000, the probability P(i) that Tim put i white
marbles in the bag?" Martha brightens. "Yes!" she says, and she happily
cranks out a probability distribution P(i). "Good," says Bob. "Let's now
compute the sum of P(i) from i = 0 to 499; that's our answer to Tim's
question."
Helen objects, "Bob, I don't like your assumption; it seems unjustified to
me. For all we know, there are just as many white marbles as black
marbles in the bag, and we got a skewed sample by random chance." Bob
replies, "That doesn't seem very likely to me." Martha, eager to have
something to do, starts to crank away. She says, "In fact, the
probability that, assuming there are 500 white marbles and 500 black
marbles, we would get at least 8 white marbles in our random sample is..."
and she states a figure. Helen concedes, "Well, that's a pretty small
number. So we can rather confidently reject my suggestion that there are
equally many white and black marbles." I ask, "So what's your answer to
my question?" Helen says, "I can't answer your question without making
assumptions that I'm not prepared to make. I'll stick to what I can say
for sure: the probability that we would see as many as 8 white marbles out
of random sample of 10 marbles is very small, if there were equally many
white and black marbles."
Stan the student now interjects, "Helen, I don't see why you can't answer
Tim's question. We got 8 white marbles and 2 black marbles. So our best
guess is that there are 800 white marbles and 200 black marbles in the
bag. Why don't we just assume that Tim filled the bag by repeating the
following procedure 1000 times: with probability 0.8, he threw a white
marble into the bag, and with probability 0.2, he threw a black marble
into the bag. Then we can calculate the probability that he would end up
with more black marbles than white marbles in the bag. Wouldn't that
number answer Tim's question?" Martha says, "I don't know if it would
answer Tim's question, but I can compute your number. It works out to
be..." and she states a number. "That's pretty small," says Stan. "And
that's my answer to Tim's question."
Now what do I say after hearing all this discussion? I would say that
Martha, Bob, and Helen have all taken defensible positions. They have
stated what assumptions they're prepared to make what assumptions they're
not prepared to make, and have reasoned accordingly. The one position I
don't find to be defensible is Stan's. His reasoning doesn't make sense
to me. Why should we think that the result of his calculation is a
sensible answer to my question? Even if Stan's number turns out to be
close to Bob's number, does that do anything to legitimize Stan's
reasoning?
Tim