Re: extended streams

[Bruno Haible]

Hi Eric,

> For platforms without stdio hooking

These include AIX, HP-UX, IRIX, OSF/1, Solaris, mingw!

> the simplest thing we could think  
> of is to create a temporary file under the hood, then provide gnulib
> overrides of fflush and fclose (the only two points at which POSIX
> requires that the original arguments to open_memstream are in sync
> with what has happened to the stream).

Such an implementation would be gross. open_memstream can be used for
light-weight operations like an extended sprintf with bells and whistles,
or for building a HTTP response in memory. If you provide an implementation
that does file system accesses and is therefore orders of magnitude slower
than expected, you could as well leave open_memstream undefined in gnulib.

> Other alternatives we considered were using mmap of a temporary file

Don't do that: it causes disk I/O that other solutions don't. mmap of
anonymous memory is always preferable to mmap of a temporary file.

> We also considered hooking every single function that takes a FILE* to
> understand string streams, but that was ruled out as too invasive to
> be practical.

But here you are on a better track for writing portable, powerful, and
efficient programs.

Really, you have to accept that in AIX, HP-UX, IRIX, OSF/1, Solaris, mingw
a 'FILE' contains only some data fields, no function pointers. All the logic
of fwrite, fflush, fclose is hardwired and not extensible.

The string streams are the first case - and a particular case - of extensible
streams. In object oriented libraries like Java's class libraries, you can
define input streams and output streams that do arbitrary things.
Input streams, for example:
  - can log their input to a file (like 'tee'),
  - can prepend some fixed content to an input stream,
  - can decrypt encoded input,
  - etc.
Output streams, for example:
  - can encrypt output on the fly,
  - can convert to precomposed Unicode on the fly,
  - can convert to HTML by escaping non-ASCII characters,
  - can add some time stamp or marker to each line before it gets output to a
    log file,
  - etc.

Will string streams really be the only kind of non-file stream that you need
in libvirt? Are you sure you will never need to do additional processing on
the strings? Are you sure the amount of data is small enough that it is OK
to store it in memory?

I would suggest to define an abstraction of "input stream" or "output stream"
or "stream", depending on the needs of libvirt, and use that.

Yes, it is invasive to use a different type than 'FILE *' in your APIs. But
  1) it allows you to do more fancy operations,
  2) it allows you to do the string streams without portability problems
     and without inefficient file accesses.

An object oriented approach is possible in C. See, for example, the
output stream class and its subclasses in GNU gettext [1]: there are
modules
  ostream                Abstract output stream data type.
  fd-ostream             Output stream referring to a file descriptor.
  file-ostream           Output stream referring to an stdio FILE.
  html-ostream           Output stream that produces HTML output.
  iconv-ostream          Output stream that converts the output to another 
encoding.
  memory-ostream         Output stream that accumulates the output in memory.
  term-ostream           Output stream for attributed text, producing ANSI 
escape sequences.
  styled-ostream         Abstract output stream for CSS styled text.
  html-styled-ostream    Output stream for CSS styled text, producing HTML 
output.
  term-styled-ostream    Output stream for CSS styled text, producing ANSI 
escape sequences.

I'm not suggesting to include these modules in gnulib. (They are written in
a somewhat idiosyncratic way.) Only to mention that an object-oriented approach
is possible in C and brings you much farther than only open_memstream.

Bruno

[1] 
http://git.savannah.gnu.org/gitweb/?p=gettext.git;a=tree;f=gnulib-local/modules;hb=HEAD