Re: Proposal for an Emacs User Survey

[Adrien Brochard]

> > > As a start, lose all of the multiple choices.
> > > Just have open questions, letting users say
> > > what they do/use/prefer, etc.
> > >
> > > The multiple choices, even with an "other",
> > > bias the results.
> >
> > How does it bias results exactly?
>
> Suggestions of what is possible.
>
> My suggestion is to just leave the questions open,
> and let people express themselves however they
> think best, rather than being incited to choose
> among suggested alternatives.
>
> Just a suggestion.

I understand what you are suggesting, but I do not understand the "bias
the results" you are referring to.

> Higher completion rate and specific answers can
> mean little, compared to thoughtful responses.
> Emacs has always preferred the latter.
>
> And explicitly solicit reasons - everywhere.
> Reasons can be important for guiding decisions here.

How about having a "why" optional free-text section after each
non-factual multiple choice question? Like "which package manager do you
use? and why?" or "What are some of your favorite packages and why?"

I also could argue that if you present a paragraph input to users, only
those with the time and energy will make the effort to write a
comprehensive answer, which might create a real bias. On the other hand,
if more people can emit their preference for a particular option, you do
not know why, but you know where to start looking for why.

You do point out something very real: the survey questions are mixing
facts with opinions. We are asking factual things, along with general
opinions on the state of things. There's obviously a trade-off to make
between the two. Factual gives us quick results but without reason. Open
ended decreases the response rate and increases the analysis time, but
opens for more meaningful answers.