Re: [Heartlogic-dev] comments on story rating thing

[William L. Jarrold]

On Fri, 13 May 2005, Joshua N Pritikin wrote:

> On Fri, 2005-05-13 at 02:29 -0500, William L. Jarrold wrote:
> > Instead of saying...
> >
> > Story
> >
> > Jack and Jill went up the hill
> >
> > to fetch a pale of water
> >
> > Understood As
> >
> > Jack wants something about a pale of water for himself.
> >
> > ....say....
> >
> > Story
> >
> > Jack and Jill went up the hill to fetch a pale of water
> >
> > Understood As
> >
> > Jack wants something about a pale of water for himself.
>
> Yah, I had that on my TODO list.  Thanks for the nudge.

u r welcome.

> > Also don't say "Most experts agree."  Say most "humans" (too nerdy or
> > nerdy chic??) or "of our research participants" or "of our
> > volunteers."
>
> Changed.

a w e s o m e .

> > Also don't say...
> >
> > The statistics above represent the combined ratings of everyone who
> > has rated this appraisal.  Following standard APA notation, N is the
> > number of samples, M is the average on a scale of -1.0 (highly
> > unbelievable) to 1.0 (highly believable), and SD is the standard
> > deviation.
> >
> > ...first of all no need to say this is APA standard notation.
>
> Removed.

looks nice.

it might be a teency bit better if you do it on several lines.

e.g. insteaed of...

You judged this statement as highly believable. We have collected 3 
ratings so far (including yours). The average believability rating is 
0.67 on a scale of -1.0 (highly unbelievable) to 1.0 (highly believable). 
The standard deviation is 0.58.

...can you format  it as..

You judged this statement as highly believable. 
We have collected 3 ratings so far (including yours). 
The average believability rating is 4.67 on a scale of 1 (highly unbelievable) to 5 (highly believable).

> >>   second,
> > do it however we do it for "Vienna is wet." and friends (i.e. the
> > rumination story).
>
> OK, maybe, but I want to let them diverge for a while.  It's more
> creative.  ;-)

yarhg.  whatever.  let's keep going...BUT, maybe we agree to the
following principle: we can ask our study-ignorant friends to look at
it and ask them to imagine doing it the other way.  and then say,
"which way would you prefer."

i mean to apply this principle to any thing on which we disagree.

> > ...also don't say...
> >
> > Understood As
> >
> > Jill is indifferent about a pale of water for herself.
> >
> > ...say....
> >
> > After reading this story HAL thinks that:
> >
> >         Jill is indifferent about a pale of water for herself.
> >
> > How correct is HAL?
> >
> > ...and have not believability ratings but correctness ratings, a la...
> >
> >         Completely Incorrect
> >         Slighly Incorrect
> >         So So
> >         Slightly Correct
> >         Completely Correct
> >
> > [except presented horizontally]
>
> Wow, uh, seems like a good idea.  Give me a day or two to digest it.

okay.

> > ...ALSO THIS SEEMS LIKE AN IMPORTANT SUGGESTIONS this kinda makes more
> > sense if the participant saw them grouped meaningfully.  E.g. like
> > so...
> >
> >         Jill is indifferent about a pale of water for herself.  1 2 3 4 5
> >         Jack is indifferent about a pale of water for herself.  1 2 3 4 5
> >         Jill thinks that Jack is indifferent about a pale of water for herself. 
> > 1 2 3 4 5
> >         Jack thinks that Jill is indifferent about a pale of water for herself. 
> > 1 2 3 4 5
> >
> > ....Whuddyasay to that?
>
> I don't think that works as well as shuffling all the questions
> together.  Here are some reasons off the top of my head:
>
> * What if an impatient web surfer only answers half of the questions?

that is fine.  what would be wrong with that?  would we loose there data?
that seems like a problem which can be overcome.

tim chklovski does it the  way i am suggesting.

when i first saw it, i kinda went, huh?????....and then as i did a few
it all started to make sense...an impatient person working for free
who did not have their entire ego invested in this (-; would have
given up long ago.  so, what i think you should do is this:

for the beginning few, let them see them ALL clustered together.  then
they will get a sense that we will be circulating thhrough each
charcter AND we will be cicrculating through wants to avoid/not avoid.
then for the next ones they should see them one at a time.

> * It gets monotonous.

huh?  it gets MORE monotonous when you have to re-read each screen in
its entirety.

i can ask my study ignorant friends and see what they say.

> * Reliability might suffer because the order that the questions are
> presented might have some affect on the ratings.

yes.  this is a very valid point.

so the solutions are two fold.

1) you could very the order in which they are presented en masse.

OR

2) like i said above they first get a sense of the overall variability
and then later the next items are presented one by one.

> > One more suggestion...
> >
> > For the rumination Vienna is wet study, the type in boxes where we ask
> > them to comment, can you pretty please with sugar on top put it
> > *below* the radio buttons?  Also ...
>
> Done, but I want our team of useability testers to squint at this before
> I am convinced that it is the one true best way.

okay.

yes, a problem is that thhey might thi8nk that they can make a rating anbd then
add a comment.  but this is not the case.  thhey must type in commment BEFORE
rating.

to the wiki page, i would like to add "comment box before or after
ratings" as a list of open issues thhat we must decide before we go
live with baby pilot.

> > Do you have any comments about the truth of this statement
> >
> > ...would be better as this....
> >
> > Feel free to add any comments about the believability of this
> > statement or about your rating right here.
> >
> > ...and maybe just "any comments" after the first 3 items.
>
> Done.

Nice.  I just verified this.  But can you make it say
"Any comments?" instead of "Any comments:"

> > See this would be another great place for a wiki.
>
> I'm not convinced.

THink of it as a way to collect the list of decisions we must make
before we go live.  Having a shared place away from email clutter will
help us see the big picture of the things that must be done bbefore we
go live.  maybe there are patterns/common themes, interrelationships
between tthese issues.  any more convinced?  like i said before, maybe
we just try it for a little while and see if it is convincing via
experience rather than by abstract argumentation.

bill


> --
> If you are an American then support http://fairtax.org
> (Permanently replace 50,000+ pages of tax law with about 200 pages.)