Re: dd PATCH: add conv=direct

[Paul Eggert]

Bryce <address@hidden> writes:

> there's a sub annoyance that I was trying to handle regarding RHAT's
> AS2.1 kernels which is a kernel limitation regarding reads and FS types
> that did or did not support O_DIRECT.

In this case, couldn't users specify a set of ibs= and iflags= flags
that avoid the limitation?  If so, it's probably not worth bothering
GNU dd about this (particularly if these are older kernels).

> The reasoning behind md5sum was,.. the boss wanted it.
> Actually it's because in automated scripting if you write a large file
> using O_DIRECT, it may not be fully committed to disk after the write
> process has finished. The upshot of this is that you will not be able to
> access the file until it's fully committed

This sounds like a bug in the kernel.  O_DIRECT should merely be a
performance hint: it should not affect the semantics of the file
system.  If one process writes to a file with O_DIRECT, and another
one later reads from the same file without O_DIRECT, it should work
correctly.  One way to do this is for the kernel to arrange that
everybody uses O_DIRECT behavior if anybody is accessing the file with
O_DIRECT.

> and md5sum could suddenly encounter a 'permission denied' condition.

Wouldn't it be better to work around this bug by using dd
iflags=odirect to read the file, and then piping dd's output to
md5sum?  This idea will work for all commands (not just md5sum).

> As for why should cp/mv have these options? I'll direct your attention
> to the following chart
> http://oss.oracle.com/~bryce/cp.gif

Sorry, I don't understand the rows and columns of that chart.
Could you please explain them?  Is there some documentation
about this stuff somewhere?

> my convolution of embedding the status in the arguments
> is expressed by your iflags/oflags options which is better to be honest

OK, it sounds like that part is done, then.

> OK, next step, what would you like from me?

Assuming Jim Meyering like the idea of adding O_DIRECT support to
cp/mv, the next thing is to see how that might be done.

Before hacking on it, though, we should regularize the usage.  Perhaps
cp and mv should have --iflag and --oflag options, that work like GNU
dd's iflag= and oflag= options?  That might be the easiest to explain
to people.

Another related project is port to non-Linux operating systems.  Are
you familiar with directio on Solaris?  If so, what is the difference
between it and O_DIRECT?  Please see the following Usenet thread:

http://groups.google.com/groups?selm=7whdvno6vn.fsf%40sic.twinsun.com
http://groups.google.com/groups?selm=107otsm4bl6p3a5%40corp.supernews.com

I'd like to modify GNU dd to do this on Solaris:

  1. Inspect the current status of the directio flag.
  2. If it's currently off, call directio(fd, DIRECTIO_ON).
  3. Do the I/O.
  4. When done, call directio(fd, DIRECTIO_OFF) if the flag was originally
     off, to restore its original state.

However, I see no way to do (1).  Too bad.

Also, step (4) can't be done if "dd" is aborted with kill -9 or
something like that.

Further: if some other program does (1) and (2) while we're doing (3),
and then does its (4) after we do (4), then the state won't be
restored correctly.  This is unfortunate too.

Ouch.  It's kind of awkward all around.  Perhaps the Solaris port
isn't worth doing.

Also, with respect to the Solaris port: Solaris doesn't have
O_NOFOLLOW, but it does have the other open options.  (Solaris has
O_NDELAY but this is so close to O_NONBLOCK that I'm not sure it's
worth worrying about.  And Solaris's O_PRIV doesn't seem to be
documented.)