Re: [Heartlogic-dev] the plan

[William L. Jarrold]

On Tue, 12 Apr 2005, Joshua N Pritikin wrote:

> On Tue, 2005-04-12 at 02:14 -0500, William L. Jarrold wrote:
> > On Tue, 12 Apr 2005, Joshua N Pritikin wrote:
> > > On Tue, 2005-04-12 at 00:27 -0500, William L. Jarrold wrote:
> > > > > 1. Rating the believability of goal status (Goal, Anti-goal, NoGoal) of
> > > > > various participants.  This corresponds to level 'g'.
> > >
> > > Without going too deep, the hypothesis is that a pair of goal statuses
> > > (same topic, same mindreader, different appraisers) characterize the
> > > situation in an affectively meaningful way.  We can take this hypothesis
> > > in lots different directions.
> >
> > I confused.  I thought the goal statuses were Goal, AntiGoal and NoGoal.
>
> Oh, yes, correct.
>
> In simpler terms, what I mean is to ask about the goal status of Toby
> and Toby's dad.  So there are two goal statuses (a pair).  The
> hypothesis is that a pair of goal statuses is an affectively meaningful
> way to characterize a situation.

I'm still feeling very unclear about what this is all about.

What is the structure of each item?  Is there a scenario cue
(i.e. an explicitly overriding goal plus an outcome)?  Is their
an emotion?

> I am not in a hurry to turn this goal-status thing into an empirical
> study.

Okay, well, I've got plenty to do for OHL and 15 to 30 minutes per night 
(on good days) to put into it so if you want to back burner this it is 
okay with you.  On the other hand, I value your contribution, your 
perspective and want to work as a team.

> However, it seems like a useful thing to start exploring and we
> can do it in a Hot-or-Not style format without any registration
> requirement.

Okay fine.  Well, if you want to keep going on this, can I see like 3 to 5 
proposed items...But if it is an informal, pilot kind of thing, I might
want to run, oh, 20 - 30 subjects on it.  Not hundreds.  Hundreds of 
subjects are valuable things to be used on well thought out research 
studies.

Maybe I am loosing the fun spirit of Josh White's turning challenge idea. 
But we need a working AI model before we do that.  Making the AI model
live again is a top priority.  BTW, I want to try to get Peter Yeh's 
matcher in on the action too.

> > > Are you hinting at a tree or flow-chart style extension of WLJ 2003?  If
> > > so, sounds good.
> >
> > I'm not sure.  Reimplimenting WLJ 2003 would not involve a flowchart.
> > Doing your study might.
> >
> > Here's what I'm thinking.  JoeUser logins in.  A computer (or a
> > ROBOT!!!!) flips a coin.  If it's heads, JoeUser gets stuck with WLJ
> > 2003.  JoeUser will be required to complete about 10 items for his data
> > to be used.  If it's tails, JoeUser gets to do your study.  Hopefully,
> > the amount of effort required by JoeUser would be roughly the same
> > regardless of whether he gets to do yours versus gets stuck with mine.
>
> Hrm.
>
> Don't we need a valid email address to allow participation in a study?
> The requirement for an email address (and informed consent) has been my
> assumption.  Is it wrong?

I don't know.

I think we'd really want an email addresss to content them latter.  E.g. 
"Dude!!! Thanks so much for your participation in our study.  We are 
getting ready to party hard on the beach now that Joshua and Bill are all 
millionaires.   You're invited."  No, seriously, we could thank them and
debrief them (a ha!  debriefing subjects is an important, probably a 
necessary component of what the psych biz calls "informed consent") and 
maybe recruit them for other studies.

One of us, (hint hint, nudge nudge, (-;) should search the web, maybe the 
apa.org website for guidelines RE doing human subjects work on the WWW.
I remember you at least sorta already did this.

> > > Can you propose a specific list of things you want ready before we go
> > > live?  It seems like your list is longer than mine.
>
> > Compare currently collected data to WLJ 2003.  If data appears sane
> > then continuing marching forward with replication of WLJ 2003.  Maybe talk
> > through the project with UT statistical services to make sure everything
> > is sane.  Make sure I can run the data and test the hypotheses on a
> > moments notice using SAS.  Make sure I know precisely what constitutes
> > replication.  Give Joshua study 3 items from 2 or 3 more groups.  Make
> > sure website can randomly decide which group each new subject goes into
> > yet keep the groups balanced.  Review ethical issues associated
> > with doing WWW study and implement/modify informed consent/etc as
> > necessary.  Publicisize the website a little to see what kinds of
> > response we get for what kind of publicity.   If response is
> > suprisingly  big, then we will want to be careful to not blow all our
> > e.g. 1000's of subjects unnecessarily.  Instead, have other studies lined
> > up.  If response is tepid, then this is less of a worry.  Once completed,
> > distill all remaining study 3 items for the of the 12 groups and send off
> > to Joshua to be put up on website.  Publicize as necessary to get
> > sufficient subjects.
>
> OK, good.
>
> > As for your deal...
> >
> > Get clear on the terms/constructs, research questions and hypotheses.
> > Develop the experimental method.  Probably write it up to prove we know
> > what we are doing.  Plan the statistical analysis -- e.g. seek input from
> > UT Statistical Services as necessary.  Pilot he items on 1 - 20 friends
> > and tweak based on their feedback.  Run study.  Analyze results.  Write
> > up.  Publish.
>
> OK, and I want to do something like WLJ 2003 modified into a tree /
> flowchart.

Okay, again, planning this will be a major undertaking.  As I have said 
before, it would be *very* interesting to use the flowchart technique to 
address some of the puzzling bits of the data.  I.e. the items that did 
not live up to the hypothesis.

> > Once both of the above are done... Become famous and watch the $$$ roll
> > in as we sip margaritas in Goa or somewhere!!!!! (-;
>
> Yah.  :-)
>
> To review, our front page is:
>
> 1. Hot-or-not style single-click goal status

Yes.

> After registration & informed consent then we offer a choice of:

Well, to be sure, my druthers is that this is the computer's random 
choice.   Poor Joe User will have no idea that a choice is even being 
made.  Else, choice that Joe User makes is an independent variable that
needs to be part of the research design.

> 2. WLJ 2003 replication
> 3. WLJ 2003 modified into a flow-chart format
> 4. Adding cues and exploring the statistical results of #1

By the way, the other day, I tried out your ohl.nirmalvihar.info website
and it was just after midnight here and it worked.  I thought you had 
regular power outages 12 to 3, no?  Maybe you've got those generators
hooked up to the school playground equipment!!?!!??!!

Bill