coreutils
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[PATCH] split: advise the kernel of sequential access pattern


From: Pádraig Brady
Subject: [PATCH] split: advise the kernel of sequential access pattern
Date: Mon, 8 May 2023 21:36:04 +0100

As split is often dealing with large files,
ensure we indicate to the kernel our sequential access pattern.
This was seen to operate 5% faster when reading from SSD,
as tested with:

dd bs=1M count=2K if=/dev/urandom of=big.in

for split in split.orig split; do
  # Ensure big file is not cached
  dd of=big.in oflag=nocache conv=notrunc,fdatasync count=0 status=none
  # Test read efficiency
  CWD=$PWD; (cd /dev/shm && time $CWD/src/$split -n2 $CWD/big.in)
done

real    0m9.039s
user    0m0.055s
sys     0m3.510s

real    0m8.568s
user    0m0.056s
sys     0m3.752s

* src/split.c (main): Use fdadvise to help the kernel
choose a more appropriate readahead buffer.
* NEWS: Mention the improvement.
---
 NEWS        | 5 +++++
 src/split.c | 4 ++++
 2 files changed, 9 insertions(+)

diff --git a/NEWS b/NEWS
index 9fad8a775..e07b10e31 100644
--- a/NEWS
+++ b/NEWS
@@ -27,6 +27,11 @@ GNU coreutils NEWS                                    -*- 
outline -*-
   due to -i, or -u.  Instead they only output this information with --debug.
   I.e., 'cp -u -v' etc. will have the same verbosity as before coreutils-9.3.
 
+** Improvements
+
+  split now uses more tuned access patterns for its potentially large input.
+  This was seen to improve throughput by 5% when reading from SSD.
+
 
 * Noteworthy changes in release 9.3 (2023-04-18) [stable]
 
diff --git a/src/split.c b/src/split.c
index d872ec56a..09209cc5a 100644
--- a/src/split.c
+++ b/src/split.c
@@ -32,6 +32,7 @@
 #include "alignalloc.h"
 #include "die.h"
 #include "error.h"
+#include "fadvise.h"
 #include "fd-reopen.h"
 #include "fcntl--.h"
 #include "full-write.h"
@@ -1621,6 +1622,9 @@ main (int argc, char **argv)
   /* Binary I/O is safer when byte counts are used.  */
   xset_binary_mode (STDIN_FILENO, O_BINARY);
 
+  /* Advise the kernel of our access pattern.  */
+  fdadvise (STDIN_FILENO, 0, 0, FADVISE_SEQUENTIAL);
+
   /* Get the optimal block size of input device and make a buffer.  */
 
   if (fstat (STDIN_FILENO, &in_stat_buf) != 0)
-- 
2.26.2




reply via email to

[Prev in Thread] Current Thread [Next in Thread]