[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH] dd: work around buffer length restrictions with oflag=direct

From: Jim Meyering
Subject: Re: [PATCH] dd: work around buffer length restrictions with oflag=direct (O_DIRECT)
Date: Fri, 07 Aug 2009 16:40:27 +0200

Eric Sandeen wrote:
> Jim Meyering wrote:
>> Eric Sandeen mentioned that dd's O_DIRECT-exposing code
>> didn't always work.  The problem is that the kernel imposes
>> draconian restrictions on the size of buffer that one may
>> write to an FD opened with O_DIRECT.  Currently, at least on
>> ext4 and with a recent linux kernel, that buffer size must
>> be a multiple of 512, which happens to be dd's default buffer size.
> (there are alignment restrictions too, just FWIW)
> ...

Yes, we should be ok on that front, with page-aligned buffers.

>> Note also that the code makes no attempt to determine the
>> appropriate block size.  So if you chose obs=42, you lose.
>> The adventurous user who experiments with oflags=direct
>> must also select a block size that works with O_DIRECT
>> and the destination file system.
> I think that's fine, docs can say "subject to size and alignment
> requirements of the kernel and filesystem" or something I think.

I mentioned size, because that can (and may have to) be changed via the
obs, ibs, bs options.  I didn't want to mention alignment because dd users
can't influence that.

> I suppose it would be nice to at least make that last buffered IO
> synchronous if possible, or sync the file afterwards.
> POSIX_FADV_DONTNEED for extra credit?  :)

Good idea.
What do you think of this?
It feels slightly excessive to make 3 additional syscalls just
to preserve some semantics of O_DIRECT for that final buffer.
...but I like it.

>From ff159a605e5bc10fe871109f66cba5ee410c9138 Mon Sep 17 00:00:00 2001
From: Jim Meyering <address@hidden>
Date: Fri, 7 Aug 2009 16:21:35 +0200
Subject: [PATCH] dd: preserve semantics of O_DIRECT even for final block

* src/dd.c: Include "ignore-value.h"
(iwrite): When disabling O_DIRECT, try to compensate
via POSIX_FADV_DONTNEED and fsync.
Suggested by Eric Sandeen.
 src/dd.c |   17 +++++++++++++++++
 1 files changed, 17 insertions(+), 0 deletions(-)

diff --git a/src/dd.c b/src/dd.c
index 43ad718..d9e4c85 100644
--- a/src/dd.c
+++ b/src/dd.c
@@ -29,6 +29,7 @@
 #include "fd-reopen.h"
 #include "gethrxtime.h"
 #include "human.h"
+#include "ignore-value.h"
 #include "long-options.h"
 #include "quote.h"
 #include "quotearg.h"
@@ -843,6 +844,22 @@ iwrite (int fd, char const *buf, size_t size)
       if (fcntl (STDOUT_FILENO, F_SETFL, old_flags & ~O_DIRECT) != 0)
         error (0, errno, _("failed to turn off O_DIRECT: %s"),
                quote (output_file));
+      /* Since we have just turned off O_DIRECT for the final write,
+         here we try to preserve some of its semantics.  First, use
+         posix_fadvise to tell the system not to pollute the buffer
+         cache with this data.  Don't bother to diagnose lseek or
+         posix_fadvise failure. */
+      off_t off = lseek (STDOUT_FILENO, 0, SEEK_CUR);
+      if (0 <= off)
+        ignore_value (posix_fadvise (STDOUT_FILENO,
+                                     off, 0, POSIX_FADV_DONTNEED));
+      /* Attempt to ensure that that final block is committed
+         to disk as quickly as possible.  */
+      conversions_mask |= C_FSYNC;

   while (total_written < size)

reply via email to

[Prev in Thread] Current Thread [Next in Thread]