Re: burned by Makefile.in.in's $(DOMAIN).pot-update rule

[Bruno Haible]

Hi Jim,

> I ran a "make dist" for coreutils in the background
> and went back to work in another window.
> A few minutes later, the system becomes 99% unresponsive
> (memory abuse) in spite of 8GB of RAM.  Keyboard and mouse
> respond only after 10-20-second delay
...
> I happened to have a sparse file with tiny actual size,
> but an apparent size of a terabyte or two

Ouch!

> This rule is the culprit:
> 
> $(DOMAIN).pot-update: $(POTFILES) $(srcdir)/POTFILES.in remove-potcdate.sed
>       if LC_ALL=C grep 'GNU @PACKAGE@' $(top_srcdir)/* 2>/dev/null | grep -v 
> 'libtool:' >/dev/null; then \
>         package_gnu='GNU '; \
>       else \
>         package_gnu=''; \
>       fi; \

These commands inspect only the files in the package to which that Makefile
belongs. I think it is reasonable to assume that a user doesn't have weird
files (terabyte-sized files, sockets, named pipes, etc.) in such a place.
Otherwise he can not even reasonably backup this directory.

You (or some auxiliary script) might have issued similar 'grep' commands as
well.

> I happened to have a sparse file with tiny actual size,
> but an apparent size of a terabyte or two, so in a sense
> it's my fault -- or maybe grep's for allocating so much
> memory while looking for a newline.

I think it is reasonable for 'grep' to allocate as much memory as the
longest line in the file has. (There is some disagreement as to whether
the time to process a sparse file should only take time proportional to
the number of physical disk blocks, not to the file length. But given
that there are no POSIX APIs to test for the extent of "holes" in files,
only low-level ioctl calls, you can't expect all the utilities to use
non-POSIX APIs.)

I think 'core' files of a.out format had holes under Linux, but this was fixed
with the adoption of ELF, in 1995: AFAICS, ELF core files are not sparse.

> One solution would be to limit the searched files
> to those under version control, but that does not seem feasible.

This rule that determines whether the package has the honorific "GNU" is
known to be a heuristic. If you know a better way to do it, please say so?
Probing www.gnu.org is probably not a good idea either, since that would
not work without an internet connection.

> Another is to allow people like me to replace the grep command
> with something else.

I think this would be too much tailored to the particular failure that you
experienced. It is not general enough.

> Finally, if you want to continue to use that libtool exclusion,
> I suggest you anchor its regexp:
> 
> -     if LC_ALL=C grep 'GNU @PACKAGE@' $(top_srcdir)/* 2>/dev/null | grep -v 
> 'libtool:' >/dev/null; then \
> +     if LC_ALL=C grep 'GNU @PACKAGE@' $(top_srcdir)/* 2>/dev/null | grep -v 
> '^libtool:' >/dev/null; then \

This second grep command (which is not the one which required a lot of memory)
only filters out some false positives due to libtool (versions 1.5.x I guess).
There is no need to change it.

Bruno