dd: avoid swap and benchmark of the nocache option

[Francois]

Hi Coreutils

today we can already give the oflag "nocache" to dd in the hope that
POSIX_FADV_DONTNEED will have the kernel discard the cache. In today's
implementation each time dd writes a buffer, it calls fadvise for *this
buffer only*. Unless sync or dsync is used as well, this is not very
efficient and we could improve it by advising the *whole file* instead.



In practice, the memory written by dd is placed in writeback for a
kernel thread to write it asynchronously. The kernel cannot
really discard the memory. This thread
https://github.com/jborg/attic/issues/252 suggests that

| It is indeed the case that FADV_DONTNEED will purge the file from the
cache immediately if it is not dirty (and will do nothing if it is
dirty).

Even more in practice, I am experiencing slowdowns on my system, the
system becomes slow and unresponsive when copying big files. **Even with
the nocache** option, the system is barely usable. Xorg goes into
uninterruptible sleep doing page faults.

I wanted to benchmark the effectiveness of the nocache parameter on my
700MB RAM + 700MB swap virtual machine, running dd to overwrite a 2GB file
on my ext4 file system mounted with data=writeback, kernel
4.3.0.5-generic from Ubuntu:

   /usr/bin/time -v /bin/dd if=/dev/zero of=out conv=notrunc bs=4096
   count=600000  iflag=nocache oflag=nocache

I compared the performance of two versions of dd:
- dd delivered on coreutils 8.23-4ubuntu
- patched dd takes coreutils 8.23 and integrates a patch in this fashion:

=======
diff --git a/src/dd.c b/src/dd.c
index d5d01f3..7bed44b 100644
--- a/src/dd.c
+++ b/src/dd.c
@@ -1062,7 +1062,7 @@ invalidate_cache (int fd, off_t len)
           if (0 <= output_offset)
             {
 #if HAVE_POSIX_FADVISE
-              adv_ret = posix_fadvise (fd, output_offset, clen,
+              adv_ret = posix_fadvise (fd, 0, output_offset + clen,
                                        POSIX_FADV_DONTNEED);
 #else
               errno = ENOTSUP;

=======

This ensures that the advice runs on the full file already written,
including those dirty pages that were previously written but maybe
not yet processed by the journaling/writeback thread at the time.

I'm using perf to count page faults for the whole system and for Xorg,
time -v dd to count context switches and report system time, and vmstat
to count swapping.  I'm testing in this fashion:

- boot the vm
- start firefox and run "10 hours of nyancat"
- runs a script that last one minute exactly. It starts dd and wait for
  the minute to complete, then do that again 14 times. The script
  completes in 15 minutes exactly, and 15 dd have run.
- halt the vm

I do that 5 times:
- running dd without options (dd)
- dd oflag=sync (dds)
- dd oflag=sync,nocache (ddsn)
- dd oflag=nocache (ddn)
- dd oflag=nocache with the patch (pddn)

I have added the numbers for the 15 runs below

                          dd    dds   ddsn    ddn   pddn
system major-faults     2118   3402    252   2766     85
Xorg major-faults        413    426      1    826      0
dd cs                 544957 289999 226657 451857 370731
dd system time (s)      27.1   32.4   32.1   39.5  100.3

vmstat gives also interesting results. I put the median value in this
table:

-----spwd---free---buf---cach--swap--bi--bo---in----cs-us--sys--id--wa-st-
dd   56548 12804   9848 340400 7  8  58 40084 691  1976 6   9   65  20  0
ddn  57140 12000   9728 308360 12 9  46 40056 742  1856 6   9   65  21  0
dds  45736  8984  16188 348996 15 12 88 40680 985  2770 6   5   61  27  0
ddsn 13892 12884 116124 146096 0  4  6  40652 1045 2778 7   6   62  25  0
pddn  3996 83716   9692 221776 0  0  1  40035 744  1763 6   12  70  12  0

What I conclude:
- that dd oflag=sync,nocache might be what you want if you want to
  minimize swapping today. This should give you the best user experience
  if you're running graphical applications at the same time.
  I think we could add this information somewhere in the man page?
- that dd oflag=nocache does not "discard the cache"
- that there might be an issue with how Xorg is mlocking or not its
  memory pages, and that Linux distros swap defaults might be dangerous
- and maybe that we can do a better use of posix_advise. What would you
  think of a oflag=nofilecache flag option?

Any thoughts?
Francois