Re: du --files-from feature request

[Bob Proulx]

Aaron Peterson wrote:
> I had no idea that this specifically was a security risk.

It isn't this specifically.  It is this generally. :-)

> I do remember one of my friends explaining how other characters could
> be used as seporators, 

There are two characters which are invalid within a filename.  Those
are the zero character ('\0') and the slash character ('/').  The zero
byte character because that is used to terminate the string.  The
slash character because that is used by the filesystem to seperate
directory paths.  All other characters are valid.

> and that if I have files with dashes in them and do a * that the
> shell expands that it tries to interpret it as an option...

Yes, but that is a completely different topic.  We were talking about
file names and this is about how programs interpret options from their
program argument lists.

> It looks like the documentation has been made more n00b friendly...
> so this should help doc readers very much.

The problem is that there are an infinite number of ways to have
problems.  It is impossible to enumerate them.  Trying to do so causes
more problems than simply trying to document the rules by which the
computer operates.  From the rules the behavior can be known.  Can you
imagine how long the documentation would be if all possible incorrect
uses of every command were listed?  How many people would read this
documentation if it were possible to produce it?

> Padraig had this solution to one of the problems I found on the debian site;
> 
> >     # wc could not be used to count NUL-separated items
> >     # (another more complex program would be needed)

For reference:
  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=444067

> tr -d '\n' | tr '\0' ''\n' | wc -l
> which works,  but really ----
> How many system admins are going to know to nuke the  \n's could be
> part of a filename,

All of them should IMNHO.  It is a valid filename character.  But that
doesn't mean that it is encouraged.  Traditionally whitespace in
filenames is not used because it is more difficult to handle and
therefore many programs do not get it right.  Like the old joke where
the patient says, "Doctor it hurts when I do this." and the doctor
replies, "Then don't do that."

> or care if they are just putting together a quick script. (that
> always seems to end up going into production because of pointy
> haired bosses) If they had no prior knoweldge and do just the tr
> from \n to \0 for du, they still get the security vulnerability.

Yes they do.  Don't do that.  :-)

> (because the hacker makes files with \n as characters in the really
> big file and the du command doesn't count it, so the hacker manages
> to fill the disk triggering some sort of unhanded exception..)

Well...  Less likely a problem with du but more likely a problem with
writing files and removing files.

> Is there already standard that is more end user friendly than posix
> for general usage, one where "advanced" knowledge isn't required to
> maintain security?

Two thoughts occurred to me while reading that questions.

1. For years and years we used paper and pencil.  It still works.  But
   most people using electronic methods do so because they prefer them
   in spite of the problems.

2. Winston Churchill's famous statement, "Many forms of Government
   have been tried, and will be tried in this world of sin and woe.
   No one pretends that democracy is perfect or all-wise.  Indeed, it
   has been said that democracy is the worst form of Government except
   all those other forms that have been tried from time to time."
   (from a House of Commons speech on Nov. 11, 1947)

People keep trying to produce better systems.  But so far they have
all been worse.

> Example, I can live without newline being an allowed filename
> character...

It is 30 years too late to change this standard though.

> and I also tend to find case sensitivity annoying.

Ew...  I am not going to touch that one.

> Ultimately, I find it annoying that more options are needed than this:
>    find "*want*" | du
> For the utilities to work properly they would need to know their
> context and I believe that everybody agrees that it's not the
> utilities job to get that complicated.

Yes that is annoying.  Because most Unix utilities were designed to
operate on text files and text files are line terminated with a
newline.  Then on the same system newlines are allowed in filenames
making this not a direct transference.  It is too bad about that.

Bob