Re: [PATCH v3 7/7] introduce simple linear scan rate limiting mechanism

[Andrey Gruzdev]

On 19.11.2020 23:02, Peter Xu wrote:
> On Thu, Nov 19, 2020 at 03:59:40PM +0300, Andrey Gruzdev wrote:
> > Since reading UFFD events and saving paged data are performed
> > from the same thread, write fault latencies are sensitive to
> > migration stream stalls. Limiting total page saving rate is a
> > method to reduce amount of noticiable fault resolution latencies.
> >
> > Migration bandwidth limiting is achieved via noticing cases of
> > out-of-threshold write fault latencies and temporarily disabling
> > (strictly speaking, severely throttling) saving non-faulting pages.
>
> Just curious: have you measured aver/max latency of wr-protected page requests,
> or better, even its distribution?
>
> I believe it should also be relevant to where the snapshot is stored, say, the
> backend disk of your tests.  Is that a file on some fs?
>
> I would expect the latency should be still good if e.g. the throughput of the
> backend file system is decent even without a patch like this, but I might have
> missed something..
>
> In all cases, it would be very nice if this patch can have the histogram or
> aver or max latency measured and compared before/after this patch applied.
>
> Thanks,
So far I have no objective latency measurements, that really 
interesting. For testing I commonly used SATA HDD, not too fresh one, 
1.5TB Seagate 7200.11 series, specially not to have large hardware cache 
or flash buffer. And yes, backend is a file on ext4.

I tried mostly with 'migrate exec:streamdump_utility', a very simple 
tool which writes stream to a file. It even doesn't use AIO - so reads 
from STDIN and file writes don't overlap. Made so intentionally to 
reduce throughput and give some random high-latency writes.

I think snapshotting performance may be severely degraded by I/O 
happening in parallel on the same storage media, that's the problem we 
need to consider.

And yes, to take latency histogram before/after the patch is nice idea!
Also I need to make stream dumping tool with AIO, to test with full 
storage throughput.

--
Andrey Gruzdev, Principal Engineer
Virtuozzo GmbH  +7-903-247-6397
                virtuzzo.com