[PATCH] Correct mistakes in formulation and structuring of the document.

[Sergiu Ivanov]

---

Hello,

On Fri, Jul 17, 2009 at 05:08:28AM +0200, olafBuddenhagen@gmx.net wrote:
> On Sat, Jun 13, 2009 at 09:23:23AM +0300, Sergiu Ivanov wrote:
> 
> > I'm sending in my attempt to compile a unionfs documentation. It is
> > formatted as a stand-alone Texinfo file for now, so that I am able to
> > build and view .info files from it.
> 
> I don't understand -- why can't you just build it as part of the Hurd
> manual?

Initially I thought that keeping the documentation in a separate place
would be better than adding something to the Hurd documentation.
However, now I'm not sure that reasoning of mine was sufficiently
reasonable.

OTOH, there are some other issues.  The location of unionfs
documentation should be reflective of the status of unionfs itself,
IMHO.  However, I remember that there are opinions both for and
against moving unionfs under the main Hurd source tree, which
virtually freezes the status quo.  I must confess that I'm more
inclined to keeping unionfs out of the Hurd source tree, and hence
keeping unionfs documentation stand-alone, but since on a large scale
I'm a newcomer to the Hurd, I'm not the one to decide.

In case everybody agrees to keep unionfs documentation in the Hurd
reference, could somebody tell me how the Hurd documentation is
maintained?  Is there a repository to checkout it from?

> > However, according to http://preview.tinyurl.com/lfy436, I will be
> > able to add this document to the Hurd documentation using one or two
> > @lowersections.
> 
> Considering all the headers that simply do not make sense when included
> within the Hurd manual, I don't think this is really an option.
> 
> Also, even if it worked, it would be ugly IMHO. Just do it as I asked
> you from the beginning: put it in the normal Hurd manual, where the
> "shadowfs" placeholder used to be.

I should paste a patch to the Hurd documentation in this case, right?
 
> > Also note that I'm not sending a git patch, since it makes little
> > sense to me: it would have been the same document but with some patch
> > headers and +'s added to each line...
> 
> This doesn't really matter in this case; but in general, even if it's
> only adding new files, a patch is better: much easier to apply; has all
> the author information etc. right.

Ah, I forgot about these ``minor'' details :-(
 
> > \input texinfo
> > 
> > @c %**start of header
> > @setfilename unionfs.info
> > @settitle GNU/Hurd unionfs Documentation
> > @c %**end of header
> > 
> > @copying
> > Copyright @copyright{} 2009 Free Software Foundation, Inc.
> > 
> > @command{unionfs} is free software; you can redistribute it and/or
> > modify it under the terms of the GNU General Public License as
> > published by the Free Software Foundation, version 2.
> > 
> > @command{unionfs} is distributed in the hope that it will be useful,
> > but WITHOUT ANY WARRANTY; without even the implied warranty of
> > MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> > General Public License for more details.
> 
> I don't think it's right to include the copyright notice of the
> *program* in the manual?...
> 
> Of course, once it is a chapter in the Hurd manual, the question doesn't
> arise anymore :-)
> 
> > You should have received a copy of the GNU General Public License
> > along with GNU make; see the file COPYING.
> 
> "make"?... :-)

Yeah, you are right, keeping everything in one manual would solve this
problem.  I have nothing against keeping everything in one file, since
it will allow me to just remove these lines instead of searching for
existing examples :-)

> > @node Introduction
> > @chapter Introduction
> > @cindex introduction, example
> > 
> > The @command{unionfs} translator is a GNU/Hurd implementation of the
> > union mounting functionality,
> 
> Not sure "union mounting" is a good term for "normal" unionfs?...

I took this expression from the corresponding Wikipedia page:
http://en.wikipedia.org/wiki/UnionFS:

  "UnionFS is a filesystem service for Linux, FreeBSD and NetBSD which
  implements a union mount for other file systems."

It occurred to me that *just* ``union mount'' in the non-Hurd world
refers exactly to what unionfs is doing.  But you are right, we should
change this expression here, because unionmount on the Hurd means
something different.

> > which basically consists in merging the
> > contents of several file systems together and mounting the result on a
> > single node.
> 
> The unioned directories do *not* need to be distinct file systems...

Yeah, sure.  I've corrected this.
 
> > An implementation of this functionality exists on
> > platforms other than GNU/Hurd and is known under the name of
> > @samp{UnionFS}.
> > 
> > One of the best known use cases for union mounting is met in LiveCDs,
> > where it is often necessary to combine a read-only file system,
> > residing on the CD, with a read/write RAM disk file system.
> > 
> > It may happen that the merged file systems will contain files with the
> > same path. @command{unionfs} uses the priority approach to solve such
> > conflicts. @xref{Resolving Conflicts}.
> > 
> > 
> > To join @file{foo/}, @file{bar/} and @file{baz/} under @file{quux/} do
> > the following:
> > @example
> > 
> > @code{settrans -a quux/ unionfs foo/ bar/ baz/}
> > 
> > @end example
> > 
> > If one would also like to include the directory tree under
> > @file{quux/} in the list of merged file systems, use @code{unionfs
> > -u}.
> 
> Is it really usful and/or customary to provide specific examples of
> command lines and options in the introduction?... Seems to me that it
> would be better only to explain the general possibilities here.

Hm, it just occurred to me that I giving an example is a nice
demonstration of general capabilities.  Do you think I should remove
this example altogether or rather create a separate section for
examples?
 
> > File systems can be added at run-time using
> > @command{fsysopts}. @xref{Run-time Options}.
> > 
> > @node Command Line Interface
> > @chapter Command Line Interface
> > @cindex man, manual, command line, interface
> > 
> > @section Synopsis
> > @cindex synopsis, usage
> > 
> > @example
> > @code{unionfs [ options ] --add @var{file systems} --remove @var{file
> > systems}}
> > @end example
> 
> I don't think this is a good synopsis: it looks as if --add and --remove
> were mandatory...
> 
> I think it would be more correct to just use: "unionfs [options]
> directories ..."
> 
> Or, if you want to be more verbose, perhaps something like: "unionfs
> [global options] ( [directory-specific options] directories ) ..."

I think I'll choose the more verbose variant in this case.  Also, I'll
try to pay more attention to terminology (``directory'' vs ``file
system'').
 
> (I'm not sure about the exact syntax used in info manuals.)

Neither am I, since I usually rely on manpages :-( I tried to follow
some rules which appear to me as generally accepted.

> > @noindent
> > Where @samp{options} may be any option (or options) enumerated in the
> > section below. @samp{file systems} is a list of paths to directory
> > trees (file system) to be merged.
> > 
> > @noindent
> > @strong{Note}: Since @command{unionfs} is a translator, the node to
> > mount the merged file systems on should be specified as an argument to
> > @command{settrans}.
> 
> I don't think this is really a useful note -- using translators is
> explained elsewhere in the manual.

Yeah, this ``Note:'' is totally redundant.

> > @section Options
> > @cindex option, start-up, run-time
> > 
> > @subsection Start-up Options
> > 
> > @table @option
> > @item -c @var{size}
> > @itemx --cache-size=@var{size}
> > Specify the maximal number of nodes in the node cache.
> > 
> > @item -u
> > @itemx --underlying
> > Add the underlying file system to the list of union mounted file
> > systems.
> > 
> > @item -w
> > @itemx --writable
> > Specify the following file system as writable. This makes it possible
> > to create new nodes in the specified file system.
> 
> Are these two really only possible on startup? Seems like a major
> limitation, which should be fixed...
> 
> Also, I wonder whether it is really useful to have extra sections for
> startup and runtime options? I tend to think it would be better to just
> flag the startup-only ones in the descriptions... (Except for those
> where it is obvious, like --help, where it's not necessary to mention in
> explicitely at all.)

These options are available at runtime.  I tried to mimic the output
of ``unionfs --help'' by splitting the Options section into two parts.
Note, however, that ``unionfs --help'' does not say that these options
are startup-only, so I made a mistake while mimicking.  I think I'll
just flag the startup-only options, just like you suggest -- this
looks the most reasonable to me.
 
> > @end table
> > 
> > @subsection Start-up and Run-time Options
> > @anchor{Run-time Options}
> > @table @option
> > @item -a
> > @itemx --add
> > Add the following file system. This option is on by default, i.e. all
> > file systems for which neither @option{--add} nor @option{--remove}
> > was specified are added to the list of merged file systems.
> > 
> > @item -r
> > @itemx --remove
> > Remove the following file system from the list of merged file systems.
> 
> Only one? My understanding is that *all* following ones will be
> added/removed...

Only the immediately following filesystem is influenced by this
option.  In unionfs/options.c (argp_parse_common_options) it can be
seen that when a key argument (filesystem or directory) is
encountered, it is processed according to the previously collected
options and then all flag variables having to do with directories are
reset.
 
> > @item -p @var{value}
> > @itemx --priority=@var{value}
> > Set the priority for the following file system to @var{value}.
> > 
> > @item -s @var{stowdir}
> > @itemx --stow=@var{stowdir}
> > Use the given directory as a stow. @xref{Stowing}.
> 
> "stow" is a verb :-)

http://en.wiktionary.org/wiki/stow -- this one says that it can be a
noun, too (though an obsolete noun :-) )

Turning back to seriousness: I'm not sure that the formulation adopted
in the argp part of unionfs is really right.  It says: ``-s,
--stow=STOWDIR stow given directory''.  AIUI, ``stow'' is something
like ``stash'', ``store''.  When we specify a directory as a stowing
directory, we don't actually *stow* this directory; rather, we are
using it as a place where the directories to be merged can be found.
This is why I decided to use a different formulation in this piece of
documentation.

Also, should a potential reader of this documentation feel confused
about the meaning of the word ``stow'', there is a cross reference
immediately coming to solve any possible ambiguity.

Did I manage to convince you or do you still consider that this
formulation should be changed?  If you opt for reformulating, could
please approximately point out how to do that?
 
> > @item -m @var{pattern}
> > @itemx --match @var{pattern}
> > Add only the nodes of the stow directory which match
> > @var{pattern}. The pattern must be a valid shell wildcard pattern
> > (suitable for @samp{<fnmatch.h>} functions).
> 
> Is this really only for --stow? The description from --help doesn't seem
> to imply that...

Yes, it's really only for ``--stow''.

The check is very simple: grep for ``pattern'' under the unionfs
source tree.  Ignoring the lines referring to README and Makefile, one
can see that anything referring to pattern matching is concentrated in
pattern.[ch] and stow.[ch].  Looking into option.c, one can see that
anything concerning patterns is used only in connection with stowing.
 
> Generally, I think some of the --help description are actually
> clearer... I guess you should just copy them, unless you have a
> description that is actually more verbose than the --help one.

I did copy most of the descriptions from ``unionfs --help'',
introducing some changes where I considered it necessary.
 
> (If you think that some of them can just be described better than what
> is presenly in --help, without being more verbose, this should be
> patched already in the --help output; not only in the manual...)

This did occur to me indeed, especially in connection with ``--stow''
and ``--match''
 
> > @example
> > @code{settrans -a foo/ unionfs -m bar --stow=/stow}
> > @end example
> 
> Probably better to use a real-world example, like "-m bin" -- this is
> the kind of stuff stowfs is really meant for...

I changed this to ``-m bin'', but I must confess that I'm not really
up-to-date with respect to what stowfs is really meant for :-( Could
you please dwell on this a bit longer? 

> > Note that this syntax actually means that all file system nodes
> > matching @file{/stow/*/bar} will be included in the merged file
> > system.
> 
> I don't understand that part.

Please, see my explanation in the ``unionfs stowing feature'' thread.

> > @node Basic Internals
> > @chapter Basic Internals
> > @cindex internals, node, light node, conflict
> > 
> > In this chapter a short description of how @command{unionfs} works
> > will be done. This description is intended for people who have at
> > least a vague idea about GNU/Hurd translator programming.
> > 
> > Note that what follows is an overview of the basic functionality.
> 
> I don't think the introduction paragraphs are necessary. And drop the
> "basic".

Corrected.
 
> > @command{unionfs} is a @samp{libnetfs}-based translator. At the base
> > of the file system published by this translator lies the concept of a
> > @samp{libnetfs} node. Like many other @samp{libnetfs}-based
> > translators,
> 
> Probably better to actually list the other examples.

I added three translators in the list (fakeroot, ftpfs, and hostmux).
Do you think I should also provide cross-references?
 
> > Since @samp{libnetfs} does not impose the necessity of keeping the
> > nodes in a truly hierarchical structure and since a @samp{libnetfs}
> > node carries along with it some bits of technical information,
> > @command{unionfs} introduces the concept of a @dfn{light node}. A
> > light node is a custom structure (by no means connected to
> > @samp{libnetfs}) which contains the information about a file system
> > entry and some links to other light nodes. These links are meant to
> > organize the light nodes in a hierarchical structure, corresponding to
> > the virtual file system exposed by @command{unionfs} translator.
> 
> Is this really specific to unionfs? My impression is that this is
> essentially a general description of the libnetfs "node->nn" concept,
> with only a few specific bits mixed in...

I don't think that this concept is really specific to unionfs, but it
is not a netnode (``node->nn'') concept.  A netnode is something
connected to a libnetfs node rather firmly; usually the relation
between a node and a netnode is one-to-one.  In contrast, the relation
between a libnetfs node and a light node is often many-to-one.

Generally speaking, you would need a light node when you need
filesystem mirroring functionality.  For instance, eth-multiplexer
could have been implemented without recurring to light nodes at all:
Zheng's light nodes are mostly device structures.  fakeroot is
implemented without any light nodes, too (but it uses netnodes).

> > When asked to look up a file, @command{unionfs} creates light nodes
> > per each directory in the path to the file. Correspondingly, a normal
> > @samp{libnetfs} node is created for each light node. Each
> > @samp{libnetfs} node is stored in a node cache, which is actually a
> > list of references to nodes. Once a node stored in the node cache is
> > accessed, it is pushed to the @acronym{MRU, Most Recently Used} end of
> > the list. When, after a series of additions of new nodes the node
> > cache becomes full, the references to the nodes at the @acronym{LRU,
> > Least Recently Used} are dropped. When there are no references
> > pointing to a node, it is destroyed. The same thing concerns light
> > nodes: when the reference count stored in a light node gets to zero,
> > the light node is destroyed.
> 
> But the light nodes are not managed in an LRU cache, right? So how can
> references go away?...

The only holders of references to light nodes are libnetfs nodes, so
it's sufficient to modify the reference count in a light node when a
libnetfs node referring to it is created or destroyed.  This is done
in functions node_create and node_destroy (which is called from
netfs_node_norefs, which is invoked when the reference count of a
libnetfs node becomes 0).

> > @section Problems
> > @itemize @bullet
> > @item
> > If there is a name conflict in underlying file systems between
> > directories and files -- say that @file{foo} is a directory in
> > underlying file system @file{a} while is a file in the underlying file
> > system @file{b}" -- then unionfs will be unable to delete this
> > entry. This is a structural bug (there's no clean way to solve it),
> > and should be fixed.
> 
> There is no way to solve it, but it should be fixed?...

I took this text from unionfs/CAVEAT and the only modifications
introduced by me are additions of Texinfo commands.  I guess the idea
of ``solving a structural bug'' is somehow related to the comment in
unionfs/ulfs.c about the whole ULFS management being ugly and wanting
to be rewritten :-)

> > @item
> > If there's a name conflict in underlying file systems between
> > directories (or between files), and the user has no permission to
> > delete all the entries -- e.g. one hidden entry is read-only -- then
> > he will get an @samp{EPERM} even if permissions seems ok. This is a
> > structural bug (there's no clean way to solve it), and should be
> > fixed.
> 
> I think both of these problems are actually a result of handling file
> deletion fundamentally wrong; or perhaps even handling writable entries
> wrong in general. It's probably never right to write to more than one
> directory at a time.

I think the problem is mostly about file deletion only.  After all,
unionfs proxies only directories and directories are usually modified
in a different way than writing to them.

I strongly believe that both these problems can be solved pretty
easily.  ATM, unionfs uses different functions to remove a directory
and a file.  I think it is much better to check what type of entry is
doomed to deletion in the current filesystem and use dir_unlink or
dir_rmdir accordingly.  This solves the first problem.

The second problem might be solved by checking for permissions before
deleting a filesystem entry and just not invoking the corresponding
deletion function in case the permissions are insufficient.

Nevertheless, the fundamental issue here is (as you point out) *how
many* directories should be modified by a deletion request.
 
> This is a non-trivial problem. Other unionfs implementations probably
> spent considerable time figuring out how to do it best. I entreated
> Gianluca to check what over implementations do, instead of trying to
> reinvent the wheel -- but he wouldn't listen :-(

A short search gave me the following:

  "By default, Unionfs attempts to delete all instances of a file (or
  directory) in all branches; this mode is called DELETE_ALL. Aside
  from DELETE_ALL, Unionfs also supports two more deletion modes,
  DELETE_WHITEOUT and DELETE_FIRST. DELETE_WHITEOUT behaves like the
  default mode externally, but instead of removing all files in the
  Union, a whiteout is created. This has the advantage that the
  lower-priority files still are available through the underlying
  filesystem. DELETE_FIRST departs from classical UNIX semantics. It
  only removes the highest-priority entry in the union and, thus,
  allows the lower-priority entries to show through. These modes also
  are used for the RENAME operation, as it is a combination of a
  create followed by a delete."
  (http://www.linuxjournal.com/article/771, Section ``File Deletion
  Semantics'')

I think this approach is pretty reasonable.  What is your opinion?

> In general, there are various places where I believe I could improve the
> wording -- but instead of explaining each seperately, I think it's
> easier if I just post a followup patch, or a revised version of your
> patch, once all issues with the actual content are sorted out...

Great :-) That's a very nice idea!

Regards,
scolobb

---
 unionfs.texinfo |   82 +++++++++++++++++++++++--------------------------------
 1 files changed, 34 insertions(+), 48 deletions(-)

diff --git a/unionfs.texinfo b/unionfs.texinfo
index 6c9138b..c38759a 100644
--- a/unionfs.texinfo
+++ b/unionfs.texinfo
@@ -44,7 +44,7 @@ A short GNU/Hurd unionfs translator documentation.
 * Introduction::
 * Command Line Interface::
 * Stowing::
-* Basic Internals::
+* Internals::
 * Caveat::
 
 * Index::
@@ -55,11 +55,11 @@ A short GNU/Hurd unionfs translator documentation.
 @cindex introduction, example
 
 The @command{unionfs} translator is a GNU/Hurd implementation of the
-union mounting functionality, which basically consists in merging the
-contents of several file systems together and mounting the result on a
-single node. An implementation of this functionality exists on
-platforms other than GNU/Hurd and is known under the name of
-@samp{UnionFS}.
+filesystem merging (``unioning'') functionality, which basically
+consists in merging the contents of several directories together and
+mounting the result on a single node. An implementation of this
+functionality exists on platforms other than GNU/Hurd and is known
+under the name of @samp{UnionFS}.
 
 One of the best known use cases for union mounting is met in LiveCDs,
 where it is often necessary to combine a read-only file system,
@@ -82,8 +82,8 @@ If one would also like to include the directory tree under
 @file{quux/} in the list of merged file systems, use @code{unionfs
 -u}.
 
-File systems can be added at run-time using
-@command{fsysopts}. @xref{Run-time Options}.
+Filesystems can be added at run-time using the corresponding options.
+@xref{Options}.
 
 @node Command Line Interface
 @chapter Command Line Interface
@@ -93,8 +93,8 @@ File systems can be added at run-time using
 @cindex synopsis, usage
 
 @example
-@code{unionfs [ options ] --add @var{file systems} --remove @var{file
-systems}}
+@code{unionfs [ global options ] ( [ directory-specific options ]
+@var{directories} ) ...}
 @end example
 
 @noindent
@@ -102,35 +102,16 @@ Where @samp{options} may be any option (or options) 
enumerated in the
 section below. @samp{file systems} is a list of paths to directory
 trees (file system) to be merged.
 
-@noindent
-@strong{Note}: Since @command{unionfs} is a translator, the node to
-mount the merged file systems on should be specified as an argument to
-@command{settrans}.
-
 @section Options
+@anchor{Options}
 @cindex option, start-up, run-time
 
-@subsection Start-up Options
-
 @table @option
 @item -c @var{size}
 @itemx --cache-size=@var{size}
-Specify the maximal number of nodes in the node cache.
-
-@item -u
-@itemx --underlying
-Add the underlying file system to the list of union mounted file
-systems.
-
-@item -w
-@itemx --writable
-Specify the following file system as writable. This makes it possible
-to create new nodes in the specified file system.
-@end table
+Specify the maximal number of nodes in the node cache (@emph{startup
+only}).
 
-@subsection Start-up and Run-time Options
-@anchor{Run-time Options}
-@table @option
 @item -a
 @itemx --add
 Add the following file system. This option is on by default, i.e. all
@@ -141,10 +122,20 @@ was specified are added to the list of merged file 
systems.
 @itemx --remove
 Remove the following file system from the list of merged file systems.
 
+@item -u
+@itemx --underlying
+Add the underlying file system to the list of union mounted file
+systems.
+
 @item -p @var{value}
 @itemx --priority=@var{value}
 Set the priority for the following file system to @var{value}.
 
+@item -w
+@itemx --writable
+Specify the following file system as writable. This makes it possible
+to create new nodes in the specified file system.
+
 @item -s @var{stowdir}
 @itemx --stow=@var{stowdir}
 Use the given directory as a stow. @xref{Stowing}.
@@ -156,7 +147,6 @@ Add only the nodes of the stow directory which match
 (suitable for @samp{<fnmatch.h>} functions).
 @end table
 
-@subsection Self-documenting Options
 @table @option
 @item -?
 @itemx --help
@@ -196,12 +186,13 @@ wildcard. The following is a typical example of using a
 pattern-controlled stow:
 
 @example
-@code{settrans -a foo/ unionfs -m bar --stow=/stow}
+@code{settrans -a foo/ unionfs -m bin --stow=/stow}
 @end example
 
-Note that this syntax actually means that all file system nodes
-matching @file{/stow/*/bar} will be included in the merged file
-system.
+This tells unionfs to scan all second-level subdirectories of
+@file{/stow/} (all directories from @file{/stow/*/} and to merge only
+the subdirectories matching the mask (subdirectories called ``bar'' in
+this case).
 
 One can specify multiple patterns, which will be combined by logical
 @samp{OR} operation. In other words, the following two commands are
@@ -209,24 +200,19 @@ equivalent:
 
 @example
 @code{settrans -a foo/ unionfs -m bin -m sbin --stow=/stow}
-@code{settrans -a foo/ unionfs -m [s]bin --stow=/stow}
+@code{settrans -a foo/ unionfs -m "[s]bin" --stow=/stow}
 @end example
 
-@node Basic Internals
-@chapter Basic Internals
+@node Internals
+@chapter Internals
 @cindex internals, node, light node, conflict
 
-In this chapter a short description of how @command{unionfs} works
-will be done. This description is intended for people who have at
-least a vague idea about GNU/Hurd translator programming.
-
-Note that what follows is an overview of the basic functionality.
-
 @command{unionfs} is a @samp{libnetfs}-based translator. At the base
 of the file system published by this translator lies the concept of a
 @samp{libnetfs} node. Like many other @samp{libnetfs}-based
-translators, @command{unionfs} does not maintain a node for every file
-that appears in the merged file system it publishes, because it would
+translators (@command{fakeroot}, @command{ftpfs}, @command{hostmux},
+etc.), @command{unionfs} does not maintain a node for every file that
+appears in the merged file system it publishes, because it would
 present the disadvantage of consuming more space and (what is more
 important) will put @command{unionfs} in the middle of every I/O
 request coming from the client to the underlying file system. Instead,
-- 
1.6.3.3