bug#53432: [PATCH] Avoid losing keyboard input when inotify is too busy [and 1 more messages]

[Ian Jackson]

Michael Albinus writes ("Re: bug#53432: [PATCH] Avoid losing keyboard input 
when inotify is too busy [and 1 more messages]"):
> Ian Jackson <ijackson@chiark.greenend.org.uk> writes:
> > I guess we probably want a "file notifications might have been missed"
> > callback.  Having it be a callback like a normal file event callback
> > will probably make it easiest to deal with.
> 
> Yep.

> > I don't think separating out the input buffers would remove that
> > requirement.  Ie, the ability to cope correctly with a flood of events
> > (of whatever kind) will remain necessary.
> 
> Yes. But if we divide the keyboard buffer (also good for mouse events
> and other events which do not harm) from the D-Bus and file notification
> events, such a check is needed at fewer places.

I don't follow.

Any buffer that we have can fill up.  So each thing that generates
events needs to be able to handle the buffer (that it is writing to)
becomeing full; and to be able to handle the buffer becoming
no-longer-full, so that it can restart.

So each event-generator has a "stop" hook and a "start" hook, each of
which has one call site, in the code that handles the buffer for that
event generator.  The number of these calls is the same either way.

The difference is just whether there are two buffers, which call
disjoint sets of start/stop hooks, or one buffer, which calls all the
start/stop hooks.

[reordered slightly:]
> The advantage of splitting into "keyboard" and "other things" buffers
> would be, that the keyboard buffer doesn't overrun, whatever burst of
> D-Bus or file notification events arrives.

The two sets of code would seem very similar.  It would probably be
sensible to actually use the very same code and data formats and
everything.  The advantage is precisely that we can prioritise them
differently: the keyboard buffer could keep working, even while the
other buffer is full.  In practice I don't think this is likely to be
particularly important since we don't usually expect emacs to be
overloaded in this way.  And when it is, there is an argument that
processing events in order is likely to be better in the sense that it
is less confusing to the user.


> > I think every one of these other event sources ought to be able to
> > partake of this scheme (adding their own hooks like I proposed for
> > inotify) without undue difficulty.
> 
> Yes. But I believe we don't need to handle lost events. If we have a
> mean to inform the upper libraries, filenotify.el and dbus.el, about
> this case, it would be sufficient. A simple check whether the event
> buffer is full.

I think we are just having a semantic difference here.  To my mind
"handling" lost events *consists of* notifying the next layer up that
"some events may have been lost".  Eventually this informaton will get
to somewhere that can do something sensible with it.


> > The result would be that if any of the buffers overflowed, we would
> > stop processing inotify events altogether until emacs has caught up
> > with the user's keyboard input, and then (via the file notification
> > "maybe lost" callback, do one check on every file which we might want
> > to revert).
> 
> I don't believe that the native backends, like inotify.c, deserve too
> much intelligence. They shall do stupid event receiving and reporting,
> with a callback invoked in case of problems. This callback must be
> clever then.

If the buffer overflowed, it is quite easy to discard now-superfluous
events at the lower layer (thus coalescing events into the single
"some may have been lost" callback).  At higher levels the state
machine needed to do this becomes more complicated.

And discarding events at a lower layer helps performance because it
means they don't need to be transformed/analysed/etc.  The performance
of event processing is particularly important in the case when things
are stressed and overloaded.

But I don't think this optimisation, at this level, is inherently
*necessary* for correct operation.

> > Would it be OK to postpone splitting this out ?  I don't think
> > splitting this up is necessary to fix these lost notifications.
> 
> Not to fix lost events. But it helps to fix lost keyboard strokes.

It will help to fix lost keystrokes only if we don't plan to fix the
uncontrolled spew from the other event sources.  But it is easier to
fix the lost keystrokes:

I suggest introducing a version of kbd_buffer_store_event which is to
be called by anyone that doesn't do flow control properly.  It would
stop putting things in the buffer when the buffer is (say) 3/4 full.

The plan would be to replace all calls to this function eventually.

> > I guess the next thing I should do is go and read the file
> > notification lisp code to see how a "please do a one-off poll"
> > callback could be implemented.
> 
> Hmm. We are still in different camps about the approach ...

I am confused.

The file notification code must have a "check all files we are
polling" facility, for when file events are not available.  Perhaps
this is buried in a polling loop, but it could be made separately
callable.

I think that the correct response to discovering that file events have
been lost, is to do a one-off poll of every file.  Perhaps we would
want to rate limit this, but that should be done by *deferring* a
poll-all, not by *skipping* one, as skipping would introduce a race
that might lose a buffer revert.  But I don't think this is actually
necessary.  In any case no keyboard events will be lost.

An alternative would be to turn off inotify entirely and just switch
back to polling.  But we wouldn't want to do that permanently; when
things have calmed down we want emacs to back to quiescently waiting
for file events.

So there would want to be some kind of timer there, and the system
would have two modes.  It seems simpler to me to have only one mode.
Writing the code to switch modes and coordinate everything seems
error-prone to me.

Ian.

-- 
Ian Jackson <ijackson@chiark.greenend.org.uk>   These opinions are my own.  

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.