Re: vnc clipboard support

[Marc-André Lureau]

Hi

On Fri, Jan 29, 2021 at 3:24 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Fri, Jan 29, 2021 at 12:18:19AM +0400, Marc-André Lureau wrote:
> > Hi
> >
> > On Thu, Jan 28, 2021 at 9:14 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
> >
> > >   Hi folks,
> > >
> > > I'm looking for a good way to implement cut+paste support for vnc.
> > >
> > > The vnc core protocol has support for text/plain cut+paste, and there
> > > is an extension adding support for other formats.  That'll cover one
> > > part of the problem, exchanging cut+paste data between vnc client and
> > > qemu vnc server.
> > >
> > > The tricky part is the second: the guest <=> qemu communication.
> > > I see basically two possible approaches here:
> > >
> > >   (1) Have some guest agent (spice does it that way).
> > >       Advantage: more flexible, allows more features.
> > >       Disadvantage: requires agent in the guest.
> > >
> > >   (2) Send text as key events.
> > >       Advantage: no guest agent needed.
> > >       Disadvantage: is translated by guests keyboard map, so qemu
> > >       needs to know the map for proper char -> key event translation.
> > >       Only works for text/plain and only for chars you can easily
> > >       type, anything needing input methods (emoji for example)
> > >       isn't going to fly.
> > >
> > > I think that (1) is clearly the better way.  Given that the agent
> > > would need to run in user wayland/xorg session context the existing
> > > qemu-guest-agent will not work.  Also linking against some UI library
> > > like gtk3 for clipboard handling is not something we want for the
> > > qemu-guest-agent.  So we need another one, I'll name it
> > > qemu-clipboard-agent for the rest of this mail.  And we need a
> > > communication channel.
> > >
> > > I'd tend to model the qemu-clipboard-agent simliar to the
> > > qemu-guest-agent, i.e. have some stream as communication path and run
> > > some stream protocol over it.
> > >
> > > Stream options I see are (in order of personal preference):
> > >
> > >   (1) New virtio-serial port.  virtio-serial likely is there anyway
> > >       for the qemu-guest-agent ...
> > >
> > >   (2) Have qemu-clipboard-agent and qemu-guest-agent share the agent
> > >       channel, i.e. qemu-clipboard-agent will proxy everything through
> > >       qemu-guest-agent (spice does it that way).
> > >
> > > Protocol options I see are (not sure yet which to prefer, need to have
> > > a closer look at the candidates):
> > >
> > >   (1) Add clipboard commands to QMP and use these.
> > >
> > >   (2) Reuse the clipboard bits of the vnc protocol (forward
> > >       VNC_MSG_CLIENT_CUT_TEXT messages to the guest agent)
> > >
> > >   (3) Reuse the clipboard bits of the spice-agent protocol.
> > >
> > >   (4) Reuse the clipboard bits of the wayland protocol.
> > >
> > > Once we have sorted the qemu <-> guest communication path it should be
> > > possible to also hook up other UIs (specifically gtk) without too much
> > > effort.  Which probably makes (2) a rather poor choice.
> > >
> > > Comments?
> > > Suggestions?
> > > Other ideas?
> > >
> >
> >
> > I also had recently some thoughts about how to implement clipboard sharing
> > in a more general way for QEMU.
> >
> > I admit I like Christophe's suggestion ("it's somebody else problem"), but
> > it falls short to me as I don't know of a common open-source remoting
> > solution for various operating systems, and I don't see how it could
> > integrate well with our UI and remote protocols. Or look at reusing some
> > VirtualBox code perhaps?
> >
> > Some things I keep in mind:
> > - the spice protocol had a number of iterations to fix some races. It would
> > be great not to repeat the same mistakes, and I don't know if VNC have the
> > same flaws or not.
> > - the spice agent design isn't great: the system agent proxies messages to
> > the active session. It would be nice if the new solution didn't have such a
> > middle-man.
>
> I was going to say that's a limtation of the use of virtio-serial, and
> to some extent that is true, but it is also not true. There is no reason
> it had to limit itselt to 1 virtio-serial device. It would have been
> valid to have 1 virtio-serial device for the system level functionality,
> and then 1 virtio-serial device per guest seat. Any guest OS configured
> for multi-seat is already going to have extra devices added to QEMU
> for each seat, so adding a virtio-serial device per seat would have
> been easy enough and avoided the crazy proxying.
>
> > - with wayland, clipboard sharing isn't really possible. Or not in a
> > seamless way at least. Today it kinda works with some X11 compatibility
> > extensions, but this will eventually go away or change behaviour.
> > - the GNOME desktop is working on remoting using RDP, and they are
> > implementing a DBus interface for it (
> > https://gitlab.gnome.org/jadahl/mutter/-/commits/wip/remote-desktop-clipboard
> > )
> > - it's not just about clipboard. We would also want to have some kind of
> > drag and drop (even if limited to files like Spice atm). We may want some
> > windowing integration. We may also want to have access to some desktop
> > services: apps, documents etc.. And what's not.
>
> Ultimately that all says we need a "remote desktop agent", which is
> extensible to other desktop integration features. Essentially that
> is what the spice agent was aiming to provide.
>
> > That leads me to think that virtio-serial is not very well suited, as it
> > doesn't allow various services / processes to come and go. I see vsock as a
> > much better alternative. (I also wonder if it handles control flow any
> > better btw)
>
> virtio-serial does make it a bit easier to have the host backend side
> self-contained in QEMU though. I think it is nice that nothing else on
> the host ever sees the agent backend, as it is a private internal channel
> between SPICE & QEMU's virtio-serial impl via the special chardev type.
>
> vsock gets exposed to the host OS in general, and though you

Not necessarily with vhost-user-vsock though.

> can have QEMU open the vsock device internally, it feels a bit silly to
> have one part of QEMU writing to a vsock device, and then another part
> of QEMU reading back from the very same device. Especially because you
> have now introduced the extra problem of having to allocate unique
> vsock addresses for each instance and deal with possibility of external
> apps maliciously trying to interact with your vsock backend.
>
> As above though, I think the way spice used virtio-serial was suboptimal
> and it should have had one extra virtio-serial device per seat.

And per service? and libvirt to hotplug stuff? Sounds insane to me.
And what about services that need to handle several connections in the
guest. For example, the way spice-webdavd works is really a pain, it
has to multiplex guest connections over virtio-serial... All of this
would be so much simpler with a single vsock connection and some kind
of bus.

>
> > I think we shoud work on getting the free desktops our best-class support.
> > To me, this means we need to speak the lingua franca, which is DBus. The
> > great thing is that DBus is also equipped to handle services that come and
> > go, handling discovery, introspection etc. Various services are already
> > available. As mentioned earlier, that's what the GNOME desktop will offer
> > for clipboard sharing. There are good chances other desktops will follow if
> > that design works, as it should be easy for them to implement the same
> > service. That means good reuse of existing desktop code. Speaking DBus on
> > Windows, MacOS or Android isn't an issue. However, vsock support may be a
> > bit tricky atm.
>
> Speaking DBus is only useful though if the guest desktop actually implements
> the DBus interfaces that we're speaking. The remote desktop clipboard work
> shown above is only going to be available on modern Linux guest OS, and
> possibly only a subset of them. IOW on the guest side, we're going to have
> to have platform specific backends to do the integration in whatever way
> is best suited for each OS. I see windows, macos, android, Wayland and
> X11 as distinct backends for the guest side.

Sure, this adaptation to various OS is needed regardless of DBus as a
protocol or not.

But we are in a much better position if we actually shape the future
and put our requirements there rather than playing a catch up game the
way desktop agents usually do. Iow, I think we need deeper integration
in the desktop. The good news is that we have pretty much the same
requirements has remote desktop solutions.


-- 
Marc-André Lureau