[Heartlogic-dev] rating comments from last year (was RE: OHL v2 alpha test)

[Joshua N Pritikin]

On Sun, 2005-03-27 at 21:38 +0530, Joshua N Pritikin wrote: 
> I haven't gone back and studied the rating comments we got last year
> from the old OHL site, but I bet that some of the comments can be traced
> back to the underspecified goal status of the parents.  Granted, it's
> just a gut feeling.  I'll have to investigate at some point.

I spent a few hours studying the rating comments.

There were two categories of comments which suggest that the WLJ 2003
study worked as planned and collected appropriate data.  These
categories are AFFECT REVERSAL with 54 comments and NO AFFECT REVERSAL
with 30 comments.  Here are some examples of the two types:

'she is happy because god hears her desire .and answers it .. maybe not
with the same object...'
'what needs to be explained?'
'Sad contradicts the original statement'
'His wish was satisfied, and so he should be pleased.'

However, this is not the end of the story.  There were 53 comments which
either point out the need for a supposition or added some suppositions.
Here are some examples:

'Hot choc doesn\'t substitute for OJ.'
'We don\'t know if Toby got what he wanted or not.  He might have wanted
OJ earlier in the day and might have got it then or not.  Daddy\'s
turning on the TV at bedtime (presumably _Toby\'s_ bedtime) has no clear
bearing on Toby\'s desire for OJ or whether he got any OJ or not.  OTOH
kids tend to not like bedtime so -- for that reason -- it\'s unlikely
thast Toby would be happy then.'

I imagine that such creative appraising adds a good deal of noise to the
statistics.  There are three smaller categories of comments which may
offer more insight into the sources of ambiguity: PROBLEM WITH COGNITIVE
PART OF APPRAISAL with 5 comments, SEPARATE THE REASON AND AFFECT with 8
comments, and AFFECT INFORMS REASON with 12 comments.  Here are two
examples of each:

'I agree with the affect, but the reason seems strange.'
'I think Toby wille be happy, but not for the reason given. Toby will be
happy because he gets what he wants'
'The affect and reason are mixed up.'
'The affect is unlikely.  The reason given is a true statement, but does
not support the unlikely affect.  (This is another example of why the
affect and reason questions need to be separated.  It\'s very
unsatisfying to be forced to give one rating for both questions
together.)'
'Good attitude ... Eric is a patient and sensible person'
'Sadistic .... Happy because of unhappiness'

Of course all these categories are my own invention.  You are welcome to
study the raw data yourself and form your own opinion.  I admit my
analysis was done quickly without taking too much care to minimize
error, however, the results seem clear enough that I am ready to
send this email.

Anyway, based on my analysis, I think most of these sources of noise can
be reduced with a few minor changes to methodology.  You will likely
notice that I have argued for the same methodology changes before.  Past
arguments were based on my thinking about the problem.  Now it seems
that data corroborates, to some extent, what I was imagining.

SEPARATE THE REASON AND AFFECT -- Not much to add here.
AFFECT INFORMS REASON -- We can ask for a rating of the reason first and
then a rating of the affect afterward.
PROBLEM WITH COGNITIVE PART OF APPRAISAL -- If the reason is
unbelievable then why ask for a rating of the affect?

As for the suppositions, I don't think the computer model has to care
one way or another.  It seems easiest to leave it up to the participants
of the study themselves.  For example,

'Hot choc doesn\'t substitute for OJ.'
'Apples and bananas are quite different even though both are fruit.  No
basis to extrapolate one way or the other.'

Why not _ask_ whether Hot choc can substitute for OJ?  Or whether
bananas can substitute for apples?  The fewer covert suppositions the
participants make, the cleaner the statistics can be.

Did you examine the comments collected in your U of Texas studies?  I
don't recall reading anything about the comments in your dissertation
except that you solicited them.

There were also 4 comments which were rather unclear.  I mention them
for completeness.  Two examples:

'because we can not exactly understand human mind...he couldn t get his
want but he still can find something to be happy'
'This one is almost rated \"neutral.\"'

signature.asc
Description: This is a digitally signed message part