Re: [Qemu-devel] [PATCH 5/8] virtio-serial-bus: Add support for buffering guest output, throttling guests

[Jamie Lokier]

Amit Shah wrote:
> On (Fri) Jan 08 2010 [01:12:31], Jamie Lokier wrote:
> > Amit Shah wrote:
> > > Guests send us one buffer at a time. Current guests send buffers sized
> > > 4K bytes. If guest userspace applications sent out > 4K bytes in one
> > > write() syscall, the write request actually sends out multiple buffers,
> > > each of 4K in size.
> > > 
> > > This usually isn't a problem but for some apps, like VNC, the entire
> > > data has to be sent in one go to make copy/paste work fine. So if an app
> > > on the guest sends out guest clipboard contents, it has to be sent to
> > > the vnc server in one go as the guest app sent it.
> > > 
> > > For this to be done, we need the guest to send us START and END markers
> > > for each write request so that we can find out complete buffers and send
> > > them off to ports.
> > 
> > That looks very dubious.  TCP/IP doesn't maintain write boundaries;
> > neither do pipes, unix domain sockets, pseudo-terminals, and almost
> > every other modern byte-oriented transport.
> > 
> > So how does VNC transmit the clipboard over TCP/IP to a VNC client,
> > without those boundaries, and why is it different with virtserialport?
> 
> TCP does this in its stack: it waits for the number of bytes written to
> be received and then notifies userspace of data availibility.
> 
> In this case, consider the case where the guest writes 10k of data. The
> guest gives us those 10k in 3 chunks: the first containing 4k (- header
> size), the 2nd containing the next 4k (- header size) and the 3rd chunk
> the remaining data.
> 
> I want to flush out this data only when I get all 10k.

No, TCP does not do that.  It does not maintain boundaries, or delay
delivery until a full write is transmitted.  Even if you use TCP_CORK
(Linux specific), that is just a performance hint.

If the sender writes 10k of data in a single write() over TCP, and it
is split into packets size 4k/4k/2k (assume just over 4k MSS :-), the
receiver will be notified of availability any time after the *first*
packet is received, and the read() call may indeed return less than
10k.  In fact it can be split at any byte position, depending on other
activity.

Applications handle this by using their own framing protocol on top of
the TCP byte stream.  For example a simple header saying "expect N
bytes" followed by N bytes, or line delimiters or escape characters.

Sometimes it looks like TCP is maintaining write boundaries, but it is
just an artifact of its behaviour on many systems, and is not reliable
even on those systems where it seems to happen most of the time.  Even
when connecting to localhost, you cannot rely on that.  I have seen
people write code assuming TCP keeps boundaries, and then some weeks
later they are very confused debugging their code because it is not
reliable...

Since VNC is clearly designed to work over TCP, and is written by
people who know this, I'm wondering why you think it needs to be
different for virtio-serial.

-- Jamie