bug#32728: bug#32729: Xemacs 23 times as fast as GNU Emacs

[Lars Ingebrigtsen]

The example is a bit convoluted, but it's an interesting question -- is
Emacs' handling of process output fast or not?

As a baseline, let's read a GB of zeroes and output to /dev/null:

larsi@marnie:~/src/emacs/trunk$ time dd if=/dev/zero bs=1000 count=1000000 > 
/dev/null
1000000+0 records in
1000000+0 records out
1000000000 bytes (1.0 GB, 954 MiB) copied, 2.04672 s, 489 MB/s

real    0m2.064s
user    0m0.173s
sys     0m1.684s

(benchmark-run 1 (call-process "dd" nil nil nil "if=/dev/zero" "bs=1000" 
"count=1000000"))
=> (0.665899839 0 0.0)

So that's better in Emacs than in the shell, but we're just setting
stdout to /dev/zero here, so we're not actually seeing the data at all.

But let's insert the data somewhere:

(benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") nil 
"if=/dev/zero" "bs=1000" "count=1000000"))
=> (4.703641145 0 0.0)

(Note: Don't visit the " *zeroes*" buffer after this, because that will
hang Emacs totally.  I guess the long-line display problem hasn't been
fixed after all?)

4.7s isn't horrible, but it's not good, either.  But most of that time
is taken up in coding system conversion, so:

(let ((coding-system-for-read 'binary))
  (benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") nil 
"if=/dev/zero" "bs=1000" "count=1000000")))
=> (1.750549617 0 0.0)

Which is faster than writing to a file:

larsi@marnie:~/src/emacs/trunk$ time dd if=/dev/zero bs=1000 count=1000000 
of=/tmp/foo
1000000+0 records in
1000000+0 records out
1000000000 bytes (1.0 GB, 954 MiB) copied, 2.21987 s, 450 MB/s

real    0m2.325s
user    0m0.168s
sys     0m1.957s

So that's OK.  But what happens when we add a process filter?

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
                               "if=/dev/zero" "bs=1000" "count=1000000")))
      (set-process-filter proc (lambda (proc string)
                                 ))
      (while (and (process-live-p proc)
                  (accept-process-output proc 0.01))))))
=> (16.878995199 993 12.469541476)

That's slow, and we're just discarding the data.  If we output the data,
it's even slower, but not a surprising amount:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
                               "if=/dev/zero" "bs=1000" "count=1000000")))
      (set-process-filter proc (lambda (proc string)
                                 (with-current-buffer
                                     (get-buffer-create " *zeroes*")
                                   (goto-char (point-max))
                                   (insert string))))
      (while (and (process-live-p proc)
                  (accept-process-output proc 0.01))))))
=> (19.801399562 1000 12.700370797000001)

Byte-compiling the function makes no difference.

So it would seem that the Emacs filter method is just unnecessarily
slow, which I've long suspected.  Creating the strings before calling
the filter is probably what's taking quite a bit of this time, but the
rest is taken up by garbage collecting as it spends 13 of these 20
seconds doing that.

And it's a real world problem: When reading data from any network
source, you have to use filters because the protocol is usually based on
parsing the output to find out when it's over, so you can't use
sentinels.

So for Emacs 28 I want to explore adding a new type of filter to
processes: One that doesn't take a string argument, but which just
inserts the data into the buffer, and then calls the filter with the
region positions of what was inserted, which is just as powerful, but
should allow streams to be 10x more efficient.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no