Re: Emacs Survey: Toolbars

[Jean Louis]

* Gregory Heytings via "Emacs development discussions. <emacs-devel@gnu.org> 
[2020-12-22 13:59]:
> 
> Christopher Dimech:
> > 
> > Shitty data obtained without care, attention, and skill is useless data.
> > Forget the weighting!
> > 
> 
> As the saying goes: "Criticism is easy, and art is difficult."  Could you
> please create and conduct a "good" survey, according to your criteria of
> "good"?  Or at least enlighten us, poor mortals, and explain what should
> have been done, and how?

Data may be obtained with care, attention and skill and still be
misleading.

That there are good intentions with the survey, I have no doubts.

On how results were presented and how survey has been conducted I have
no doubt that author lacks the skill on the specific subject of making
an opinion poll. 

I have no doubts that author of the survey did conduct the survey,
connected websites together and made some marketing for people to
place their polls. And I have no doubt on his other skills,
programming, using mathematics, and I did not review the way of
counting, but I would not doubt on the manner of counting statistics
at this moment.

Emacs survey has a lot of misleading information that was already
mentioned here.

Major misleading information is the difference between general survey
and specific survey that relates to a subset of people from specific
groups of people.

Then we come here to discuss the toolbar that only a subset of people
from specific groups of people said something about that. I do not
agree that "any data is better than no data". The information may be
useful but due to its presentation it may be even more misleading than
useful.

When one says it is "Emacs Survey" I do not see it and do not
understand it by such general term.

As one may see on this picture of the survey in question this survey
represents 54.1% of people comming exclusively from Reddit and Hacker
News, then comes 7.8% Twitter. Majority of people 61.9% answering the
survey are groups coming from Reddit, Hacker news and Twitter users.
https://emacssurvey.org/2020/how-did-you-hear-about-this-survey.svg

Better would be to say that it is Emacs Survey that represents almost
62% of Emacs users coming from Reddit, Hacker News and Twitter.

One may see here that 1.4% of users answered the survey by email:
https://emacssurvey.org/2020/submissions-medium.svg

There are 7344 total responses.

1.4% of 7344 is 102+ users (more than 102 due to floating
point).

Maybe those users did not want to submit the online form as they did
not want to use non-free Javascript? But maybe those users did not
want to submit as they wanted to express themselves better by using
the Org file sent by email? We do not know why, but we can assume that
1.4% did not want to submit the form online.

We may compare data, we could take the Debian popularity contest:
https://qa.debian.org/popcon.php?package=popularity-contest
where recent data says that 200504 people installed the package
`popularity-contest' that counts which other packages are installed on
the system.

Then we may look into the subset of those 200504 Debian users who
installed Emacs: https://qa.debian.org/popcon.php?package=emacs

To me it looks like about 14,000 users installed Emacs on their Debian
GNU/Linux system. That amounts to some 6.98% of 200504 users who use
the Debian popularity contest package.

As Ubuntu comes from Debian GNU/Linux some vague information says
there are 20 million Ubuntu users.

If we would follow the trend of 6.98% let us truncated it to 6% of
people using Emacs on Debian derivative distribution, that would
amount already to (* 20000000 0.06) 1,200,000 people using Emacs in
Ubuntu. https://ubuntu.com/blog/ubuntu-is-everywhere maybe this number
is larger or bigger, but it does give some traces on how to conclude
how many people are using Emacs.

This page says there must be 100+ millions of GNU/Linux users on
Desktop (not counting Android):
https://www.quora.com/How-many-Linux-users-are-there-in-the-world/answer/Juha-Kaskiharju

but I cannot know where and how information has been obtained:
https://www.netmarketshare.com/linux-market-share

If we do assume there are 100+ millions GNU/Linux users who use their
browser as that is how companies detect such users and that 6.98%
trend as established from Debian popularity contest uses Emacs than
that may amount to 6-7 millions of Emacs users.

If we now come back to those 1.4% that did not want to answer the
survey by using non-free Javascript, maybe the real amount of those
people, who did not see the survey, but would not answer the WWW
survey for some of not unknown reasons, is already (* 6980000 0.014)
97720 users.

Emacs survey of 62% of Emacs users coming from Reddit, Hacker News and
Twitter has shown that that subset of users being about 4340 users
would gladly answer the survey over WWW form, with or without
considering if they use free or non-free Javascript.

I am personally, not by mathematical estimation, thinking that 90,000
of Emacs users and more than that, would not answer the survey only
because of the non-free Javascript or because how survey was
conducted.

Now let us say we discuss of toolbars, we do not discuss of toolbars
for Emacs users. We discuss of the fact that 85.1% of users using
Reddit, Stackexchange and Twitter who answered the Emacs survey that
relates to 62% to those groups are disabling the toolbar.

MacOS users from the Emacs survey are 26.6% plus Windos 7.4%, total
being 34% among all Emacs survey users or 34% from 7000 amounting to
some 2380 users.

When considering then if toolbar shall be disabled, shall we also
consider are we disabling it for majority of GNU/Linux users, or for
majority of proprietary system users, or for those Emacs users who
never answered the survey due to not unknown reasons which may amount
to almost 100,000 but which user habits we do not know.

In general the survey results would be much more useful would we have
intersections of people by various characteristics. We do not have
intersections, though it could be possible to make such intersections
with the data at hand.

Then survey did not give the result if users would like by default the
toolbar to be disabled. As when user disables the toolbar it does not
mean user would like it by default to be disabled upon first
installation.

The Emacs survey we speak about is not general Emacs survey.

That is in my opinion major misconception that people will get as it
all sounds so much official with nice and shiny charts.

It is Emacs survey that represents for 62% those users coming from
Reddit, Stackexchange, Twitter and we do not know many of intersection
results as such are not presented on the website.

If one only looks at one chart, like on chart among Emacs Survey
representing 62% users from Reddit, Stackexchange, Twitter, then one
may draw conclusions which are counter useful to the real number of
Emacs users.

What would be useful to know for development could be interesections
of:

- users using Windows, using Emacs with daemon mode
- users using MacOS, using Emacs with daemon mode
- users using GNU/Linux, using Emacs with daemon mode

- users using Windows, using Emacs as GUI or terminal
- users using MacOS, using Emacs as GUI or terminal
- users using GNU/Linux, using Emacs as GUI or terminal

- users using Windows, disabling toolbar
- users using MacOS, disabling toolbar
- users using GNU/Linux, disabling toolbar

then again intersections for menu bar, splash screen, etc.

Taking just one chart and looking how 85.1% users disabled toolbar is
incorrect interpretation that there is something to be handled as that
data comes from specific groups of people.

One cannot even ask experienced user if the tool bar should be
disabled as that is misleading. One could only ask a very new user who
has never seen Emacs, by placing that user on the chair and showing
him Emacs interface and then asking him if the toolbar is useful or
not useful.

Make that a first question to 10 people who never used Emacs and then
see the result and compare it. I guess that 10 people would be enough,
Nielssen says 5 people are enough, and I understand his methodology.

It is thus in my opinion more productive and more useful to take those
5 people or make 3 groups or 5 groups of 5 people and place them in
front of Emacs to tell us if they find toolbar useful.

It is counter productive to make assumptions to disable toolbar
because users who already used Emacs probably for years disabled the
toolbar. They are not new users.

While they may tell that disabled toolbar for them is good, they
cannot tell that for new users for which toolbar would be disabled by
default, they are not qualified for that as they are not new users any
more.

Jean