[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH] dd: work around buffer length restrictions with oflag=direct
Re: [PATCH] dd: work around buffer length restrictions with oflag=direct (O_DIRECT)
Fri, 07 Aug 2009 16:40:27 +0200
Eric Sandeen wrote:
> Jim Meyering wrote:
>> Eric Sandeen mentioned that dd's O_DIRECT-exposing code
>> didn't always work. The problem is that the kernel imposes
>> draconian restrictions on the size of buffer that one may
>> write to an FD opened with O_DIRECT. Currently, at least on
>> ext4 and with a recent linux kernel, that buffer size must
>> be a multiple of 512, which happens to be dd's default buffer size.
> (there are alignment restrictions too, just FWIW)
Yes, we should be ok on that front, with page-aligned buffers.
>> Note also that the code makes no attempt to determine the
>> appropriate block size. So if you chose obs=42, you lose.
>> The adventurous user who experiments with oflags=direct
>> must also select a block size that works with O_DIRECT
>> and the destination file system.
> I think that's fine, docs can say "subject to size and alignment
> requirements of the kernel and filesystem" or something I think.
I mentioned size, because that can (and may have to) be changed via the
obs, ibs, bs options. I didn't want to mention alignment because dd users
can't influence that.
> I suppose it would be nice to at least make that last buffered IO
> synchronous if possible, or sync the file afterwards.
> POSIX_FADV_DONTNEED for extra credit? :)
What do you think of this?
It feels slightly excessive to make 3 additional syscalls just
to preserve some semantics of O_DIRECT for that final buffer.
...but I like it.
>From ff159a605e5bc10fe871109f66cba5ee410c9138 Mon Sep 17 00:00:00 2001
From: Jim Meyering <address@hidden>
Date: Fri, 7 Aug 2009 16:21:35 +0200
Subject: [PATCH] dd: preserve semantics of O_DIRECT even for final block
* src/dd.c: Include "ignore-value.h"
(iwrite): When disabling O_DIRECT, try to compensate
via POSIX_FADV_DONTNEED and fsync.
Suggested by Eric Sandeen.
src/dd.c | 17 +++++++++++++++++
1 files changed, 17 insertions(+), 0 deletions(-)
diff --git a/src/dd.c b/src/dd.c
index 43ad718..d9e4c85 100644
@@ -29,6 +29,7 @@
@@ -843,6 +844,22 @@ iwrite (int fd, char const *buf, size_t size)
if (fcntl (STDOUT_FILENO, F_SETFL, old_flags & ~O_DIRECT) != 0)
error (0, errno, _("failed to turn off O_DIRECT: %s"),
+ /* Since we have just turned off O_DIRECT for the final write,
+ here we try to preserve some of its semantics. First, use
+ posix_fadvise to tell the system not to pollute the buffer
+ cache with this data. Don't bother to diagnose lseek or
+ posix_fadvise failure. */
+ off_t off = lseek (STDOUT_FILENO, 0, SEEK_CUR);
+ if (0 <= off)
+ ignore_value (posix_fadvise (STDOUT_FILENO,
+ off, 0, POSIX_FADV_DONTNEED));
+ /* Attempt to ensure that that final block is committed
+ to disk as quickly as possible. */
+ conversions_mask |= C_FSYNC;
while (total_written < size)