Re: Shrinking the C core

[Arthur Miller]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Arthur.
>
> On Wed, Sep 06, 2023 at 07:04:43 +0200, Arthur Miller wrote:
>> Richard Stallman <rms@gnu.org> writes:
>
>> > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
>> > [[[ whether defending the US Constitution against all enemies,     ]]]
>> > [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>> >   > Would you please stop arguing for rewriting Emacs in Common Lisp?  It
>> >   > is a non-starter.
>
>> >   > It would be an enormouse job -- including rewriting the Emacs Lisp
>> >   > Referance Manual.
>
>> > Also, there are aspects of Common Lisp which i rejected as clumsy and
>
>> With all respect to you, but sound to me like an emotional argument, not
>> a rational one.
>
> It's a rational argument expressed in emotional terms for simplicity.

What you basicaly say: it somehow is a summarization of some more
advanced/complicated reasons which can't be explained, but have to be
expressed in the term "I don't like it" for the simplicity?

I don't think it is a rational argument, but I explicitly didn't want to
speculate why RMS feels so, but asked him to clarify for pure respect
for him as a person. You can try to teach me why is it a rational
argument and why it has to be expressed in emotional terms for
simplicity, and I promise I'll try to understand it as best as I can,
but I have hard time to see such an explanation, because the argument is
truly just an opinion.

CL for sure has things that in retrospect can be debated about, but I
don't think keyword arguments are one of those.

Also note that keyword arguments, or at least similar principle is used
in other parts of Emacs, not just cl-lib; for example look at
define-minor-mode. While define-minor-mode macro does not use "&key" as
an explicit keyword, the arguments to it are a plist which basicaly
gives you the very same effect. The "new" defvar-keymap is another
example.

Keyword arguments are undeniably useful because they let us omit
arguments we are not interested in, and type only the ones we are
interested in. I don't see how they complicate a Lisp by much; we have
&rest and &optional so what is problem having &key?

In my personal opinion it is by far more clumsy to have two defun
macros, and to prefix one with cl- for no good reason at all; and than
have to explain for users why they are two different versions, how they
differ, when they should use each, why would they like to use one or
another etc. If ordinary "defun" was simply upgraded to act as cl-defun,
potential user dilemas and explanations would be avoided.

>> I don't know why you reject them as clumsy, but why does it matter to us
>> if a tool includes features we don't use?
>
> It matters a very great deal.  In practice you cannot avoid "using" these
> features if you have to understand or debug somebody else's code.

In practice most of code will use some common part of the language, and
relatively little code will use some specialized parts. In a big
language that targets many different kind of developers, there will
always be features that are needed by some developers but not by
all. Falacy in your, by now very tired and wornout argument, is the
assumption that everyone will use everything in the language all the
time, which certainly isn't the case.

Interestingly, yesterday why commuting home after the work, I have heard
a talk by Stroustrup, quite recent just a few week old:

https://www.youtube.com/watch?v=eo-4ZSLn3jc

The talk is more or less about that very argument: why is C++ so big and
why are there so many features in c++. (If the link gets removed search
on Stroustrup C++ 2023). I think that was one of better Stroustrup's
talks, definitely worth watching.

There is also an incredibly inspirational talk by Steel (search on
"Growing the Language"); I have seen it just a few weeks ago myself:

https://www.youtube.com/watch?v=lw6TaiXzHAE

Observe that Steel gave us the "most minimal" and "elegant" Lisp as
Schemer's like to think of Scheme, which is probably important to
remember in the context of this discussion. If you don't watch it, the
essence is in this summary, and in my opinion is really a clever
insight:

"Over the last quarter-century Guy Steele has been convinced that trying
to design a complete and perfect programming language is the worst thing
you can do. A programming language (including its associated libraries)
must grow over time as its user community and its development community
grow. This is a different situation from 25 years ago, when all such
communities were relatively small. The difference is a problem of
scale. As a result, programming language design now and in the future is
necessarily as much a matter of social engineering as technical
engineering and must rely more on a set of general principles than on a
set of specific technical decisions."

>> I don't know if I understand it correctly, but CL was made so other
>> Lisps can be abstracted on top of it; so we use those features to
>> abstract stuff on top of those lengthy verbose functions? In other
>> words we can hide those keyword params behind elispy abstractions if we
>> don't like them?
>
> That's complexity and bloat.  Far better not to have them in the first
> place.

Complexity and bloat by which metric?

Emacs has long time ago exceeded CL in numbers of "core functions". By
the time I have extracted all Lisp defuns from the C core, a couple of
months ago or so, it was ~1800 functions in C core. That is by far many
more than what CL standard defines. I know that some of Emacs devs are
lurking on Reddit behind pseudonims, I don't know if you are one of
them, but there are also quite many users who consider Emacs to be quite
bloated and would like many of some older parts to be expunded out of
the core.

Anyway, by the nature of Lisp, there are very few actual language
concepts, and there are quite few additional concepts that CL offers
that need support of the system that are not found in Emacs; but most of
additions to the language are actually looking exactly like the language
itself. The nice thing is that, when you write a defun in Lisp, it
becomes part of the language itself, unlike languages like Java, C, C++
etc where users can not extend the language itself since the syntax has
to be encoded in the compiler. In other words, it means, that even if
you don't want it, people can extend it the way they like it, and cl.el
was a proof.

>> > best avoided.
>
>> Why are they best avoided? Is there some technical reason or is it
>> psychological?
>
> It's because they are not needed.  Bear in mind, Common Lisp is a massive

I don't know if you agree with me or not, but in order to drive from
Nuremberg to Berlin you certainly don't need AC in a car; but on a hot
summer day, with 40 degrees (celsius) it is very nice to have AC in the
car, isn't it? Sure, keyword arguments are not needed; but they are
handy from time to time. Optional or rest arguments are not needed
either, but they too are nice to have sometimes, aren't they? What
makes keyword arguments more "clumsy" then? 

To be honest to you, I really didn't want to answer this mail, because I
see just very same arguments I have seen many, many times before. I had
very much, very same sentiment about C++ as you, and I didn't even know
much of CL at all; I just thought it was big and bloated and didn't want
to use it, untill not so long time ago. Untill perhaps a year or two
ago, when I invested some time in learning more about CL and tried to
understand why things are as they are, not just in CL, but in general;
C++ very much included.

> It's because they are not needed.  Bear in mind, Common Lisp is a massiveI
> language, much like C++ or PL/1 or Algol-68.  With such a language,
> nobody uses all of it (it's just too big), but everybody has her own
> personal subset.  This creates difficulty when somebody else has to
> understand that first hacker's code.

I totally understand you; but I believe this is a bit misguided. For the
first, for the same argument as above, nobody uses the entire language
because they don't need it; not because it is too big. Some parts target
certain specialized domains which are needed by some groups and not by
everyone. I think that is more true for C++ and less for CL, because CL
is not that massive as you are trying to make it. CL is also a child of its
time, just like Emacs, and some parts are probably in need of a
revision, but it certainly stands the proof of time when it comes to
core language principles.

If we look at this:

(defun directory-files (directory &optional full match nosort count)
...)

When you call this function in the code; how easy is to remember the
order and number of arguments? Was it 5 or 6 arguments? I had fortune to
contribute the idea for the 5th argument, and yet I typed by a misstake
the wrong number:

CL-USER> (directory-files "./" nil nil nil nil 5)
; Debugger entered on #<SB-INT:SIMPLE-PROGRAM-ERROR "invalid number of 
arguments: ~S" {1001E92C63}>
[1] CL-USER> 
; Evaluation aborted on #<SB-INT:SIMPLE-PROGRAM-ERROR "invalid number of 
arguments: ~S" {1001E92C63}>
CL-USER> (directory-files "./" nil nil nil 5)
("alloc.lisp" "buffer.lisp" "callint.lisp" "callproc.lisp" "casefiddle.lisp")
CL-USER>

If I only change &optional to &key:

(defun directory-files (directory &key full match nosort count) ... )

I can now type:

CL-USER> (directory-files "./" :count 5)
("#dired.lisp#" ".#dired.lisp" "alloc.lisp" "buffer.lisp" "callint.lisp")
CL-USER> 

How "clumsy" is that? Honestly, I am not trying to be PITA or devil's
advocate or anything like that; but I think that everyone will agree
that the second one is both nicer, less error prone to type and puts
less cognitive load on users to remember or lookup the information.

Just as a remark: by just having it in CL I was able to just go to the
code, and change "optional" for "key", C-x C-e in Sly and I changed the
implementation of "directory-files", nice? For the record if you want to
try it (just a fast hack for the demonstration purpose, needs cl-ppcre lib):

(defun directory-files (directory &key full match nosort count)
  (let ((files
          (if full
              (mapcar #'namestring (uiop:directory-files directory))
              (mapcar #'file-namestring (uiop:directory-files directory)))))
    (when match
      (let ((scanner (ppcre:create-scanner match))
            matches)
        (dolist (filename files)
          (when (ppcre:scan scanner filename)
            (push filename matches)))
        (setf files matches)))
    (unless nosort
      (setq files (sort files #'string-lessp)))
    (when (and (numberp count) (> count 0))
      (when (> count (length files))
        (setf count (length files)))
         (setf files (subseq files 0 count)))
    files))

Observe also that I agree with you; keyword arguments are not needed,
but are very nice to have.

> CL is a niche language; it has not captured the hacker mindset.  I think
> it is just too big and too unwieldy.
>
>> > best avoided.  All those keyword arguments!  I intentionally excluded
>> > tham from Emacs and I am not going to let them in.
>
>> I don't want to be impolite, but they are already in; via cl-lib.el.
>
> Yes.  There was a time not so long ago when cl.el was banned from use in
> our Lisp code, except for at compile time.  Our Emacs Lisp was small,
> simple to understand, and easy to learn.  Now things in cl-lib.el get
> used as if they are just a normal part of Emacs Lisp.  Our language is
> thus MUCH more difficult to understand, perhaps by a factor of somewhere
> between 3 and 10.  When perusing even established parts of Emacs I groan
> inwardly every time I encounter one of these needless cl-lib features.
> It stops me dead, forcing me to consult doc strings (which are often
> missing and often inadequate even when they are present) or even manuals.

"MUCH more difficult" for whom in which sense?

What you are telling is that "good old times were so much better" which
is just an emotional argument. You are actually just reinforcing the
same sentiment that RMS expressed. I shared that sentiment myself, but I
think it is misguided.

We can certainly speak about "old ways", let us take the or-idiom or how
should I call it: initialization of the default value for optional arguments:

(defun foo (&optional who-am-I)
  (let ((who-am-I (or who-am-I "foo")))
    (message "I am %s." who-am-I)))

Is that really better than typing:

(cl-defun foo (&optional (who-am-I "foo"))
  (message "I am %s." who-am-I))

The user has to learn the idiom, which uses operator "or" to perform
something that visually has nothing to do with the intention of the
code, and also has to type the additional let-form each and every
time. Than the users who are not familiar with the idiom will perhaps
come up with their own version, using setq or some other thing, and you
will really have to think what the user wanted to say with their code if
you had to debug it. Is it better than seing an initialiser and knowing
directly what is the default value and everyone using uniform syntax?

There was a discussion, perhaps a couple of years ago, I think on Emacs
help list, I don't remember the detials, but I think something about
car, cdr, cdar, cddr & co, and using "nth" instead of them. Monnier said a
clever thing there: using those old functions is like writing in
assembler, but without the benefit of additional speed (something in
that sense). I think he was more than correct there. It is much more
clear to say (nth N list), than to chain cars and cdrs together.

A bit further, that really is about abstractions. We can for sure do
lots of stuff manually, without higher-level abstractions; but higher
level abstractions, like using initializer in a defun arguments, help us
convey the meaning more clearly and concize, we can rationalize about
abstractions, and they are probably easier to remember than teaching
people how to use some common pattern or idiom.

I also happened to read the CMUCL manual just yesterday. They have a
nice writing about using structures instead of lists in CL:

"Even if structures weren't more efficient than other representations,
structure use would still be attractive because programs that use
structures in appropriate ways are much more maintainable and robust
than programs written using only lists. For example:
     

    (rplaca (caddr (cadddr x)) (caddr y))

could have been written using structures in this way:
     

    (setf (beverage-flavor (astronaut-beverage x)) (beverage-flavor y))

The second version is more maintainable because it is easier to
understand what it is doing."

https://cmucl.org/downloads/doc/cmu-user-2010-05-03/compiler-hint.html#toc189

> Were we to rewrite Emacs in Common Lisp, these things would get worse
> fairly quickly.

I think you are very, very wrong about that one; on the contrary we
would get some tools that Emacs is lacking to manage the code
complexity, but I don't think they are actually applicable on the
existing code, so they don't matter much. But things certainly wouldn't
get worse. That is just my opinion of course, you can think I am an
idiot and I can be wrong, but at least give me a reason to change my
mind that isn't based on subjective opinion.