Re: Rethinking the design of xwidgets

[Arthur Miller]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Akira Kyle <ak@akirakyle.com>
>> Cc: emacs-devel@gnu.org
>> Date: Tue, 13 Oct 2020 14:37:31 -0600
>> 
>> > I'll need to read up on those two topics, but in general, as 
>> > someone
>> > already pointed out, the Emacs display of images supports
>> > interactivity by letting you call functions triggered by mouse
>> > gestures on the varipous spots on the image.  If such a function
>> > produces a zoomed-in/out image and displays it instead of the 
>> > original
>> > one, wouldn't that fulfill the need?  E.g., we already support
>> > animated GIF images and zoom-in/out by mouse scroll, so it is
>> > inaccurate to say that we can only display static images.
>> 
>> Can actions be triggered on a click and drag as one would do to 
>> pan within a plot?
>
> I didn't have time to try that, but just looking at the code, I don't
> see anything that makes the clocks on images special, so I'd expect
> the same modifiers and drag-detection code to work.  And if it
> doesn't, I seed no reason why we couldn't teach it to do so.
>
>> As far as I know there are really only two FLOSS PDF libraries out 
>> there: poppler and muPDF. Between them poppler supports more PDF 
>> features and seems to be more widely used. I agree this isn't 
>> really as compelling of a use case for xwidgets, but I seems like 
>> it would be easier to achieve better PDF rendering via xwidgets.
>
> I can see how that is tempting: the image handling is taken care of.
> What I'm saying is that this advantage should be carefully weighed
> against the disadvantages and difficulties in integrating such
> self-managing objects into the Emacs display code.
>
>> > E.g., the kludge in dispnew.c around line 4365.  It disables one
>> > of the most important redisplay optimizations in Emacs, once you
>> > build with xwidgets enabled.
>> 
>> Ah, I had briefly looked at that but I wasn't sure how important 
>> it was. I may be able to make some progress on that but it's hard 
>> to understand the context for that function. It seems like that 
>> function is responsible for an optimization that avoids redrawing 
>> glyphs if the window is being scrolled by copying them
>
> Not only if the window is scrolled, but also if the previous display
> can be converted into the current display by scrolling some of the
> stuff that is already on the glass.  the algorithm is a basic
> comparison of the current and desired display, similar to what the
> Diff utility does.
>
>> however I couldn't find where or how the glyphs are actually copied?
>
> See the calls to rif->scroll_run_hook.
>
>> > I'm not trying to scare you, mind you, just point out that the
>> > ideas you have should probably be talked through in mode detail,
>> > to see that they are workable.  For example, did you study the
>> > various window-redisplay optimizations, and if so, did you think
>> > how the widgets will adapt to them?
>> 
>> I'm just starting to dig my teeth into some of that. However I'm 
>> wondering if anyone has more recently tried to quantify the 
>> improvements the various redisplay optimizations make to both 
>> actual and perceived render time of Emacs. For example the comment 
>> on the scrolling_window function in dispnew.c that you pointed me 
>> too references that it is implementing an algorithm from 1978, 
>> perhaps it would be worth exploring if there are any algorithmic 
>> advances?
>
> I don't think that algorithm is less optimal nowadays than it was back
> then; the really interesting question is how much it saves us when
> compared to simply redisplaying all those lines.
>
>> It strikes me that perhaps some optimizations may no longer be as
>> necessary on modern hardware or on GUI displays as they were when
>> running emacs in a VT on 20 year hardware.
>
> Those are very good questions.  I'm not aware of any such
> investigations since the current redisplay optimizations were
> implemented around 20 years ago: at that time, Gerd Möllmann, who
> developed the current display engine, did add optimizations one by one
> until he got reasonable redisplay performance.

In case where there is a discret gfx card (i.e. Nvidia/AMD) it is
probably faster to send everything to GPU and ask it to render a 
giant texture and then use it as XWindow pixmap, or something similar
then to figure out on CPU all the stuff that should not be displayed.
But Emacs will maybe run on some slow devices (atmel Emacs anyone?), so
you probably don't want to ditch away all that disp stuff.

> Measuring the speedups from each of the several optimizations is an
> important job, but it's a large job.  For starters, one needs to come
> up with a large enough and representative enough sample of redisplay
> use cases, and that is not easy.  So I'd encourage you (or anyone
> else, actually) to do this important job, but be aware that it could
> take non-trivial time and effort.
>
>> Conversely there may be further optimization that could be done by
>> taking advantage of more recent hardware advances such as offloading
>> to a GPU or utilizing vectorized instructions.
>
> We try not to use machine-dependent code in Emacs, because that's a
> maintenance burden, what with today's fast pace of chip development
> and obsolescence.  Vectorization is generally left to optimizing
> compilers, and relying on special hardware, such as GPU, is not
> something we should depend on directly.  We should instead hope that
> the GUI toolkits and display systems we use will do that for us.
Would it be considered too machine-dependent if you rendered into
off-screen OpenGL surface as your rendering target? Should be avialable
on Nvidia/AMD/Intels gpus. With offscreen egl you would have relatively
OS independent code too; but that would require explicit linking to
proprietary libs. I mean X11/Win32 already does that when proprietary
drivers are installed, so Emacs (and any other software) "implicitly"
links to those, but I don't how it is viewed if you would to explicitly
when present.

You could maybe also expose egl-surface to elisp as an "image format"
and enable rendering to an image directly instead of going through
external process.