Re: RFC: fixing the 32-bit size and time limits in gzip file format

[Greg Roelofs]

[Cc'ing bug-gzip, as requested.]

Mark Adler wrote:

> On Aug 15, 2010, at 5:25 PM, Paul Eggert wrote:
> > * If the file size is 2**32 or larger, gzip should emit an extra field
> >  that records the size divided by 2**32 (discarding fractions).  gzip
> >  -l should read this field when reporting the size.

> The problem is that that cannot be supported in piped mode.  The extra
> field is at the beginning but the size won't be known until the end.

That's true, but the whole point of extra fields is that they're optional
(...obviously, since no previous version of gzip writes such a field, and
they're not all going away anytime soon).  This would support a huge class
of cases, though, so it seems like a no-brainer to add it.

It might be worthwhile to add a special placeholder subcase that means "I
don't yet know the full size and won't be able to write it to the extra
field, but I'll report it at the end in case you have an after-the-fact
utility that can update it in place."  For example, I've archived disk
images via "dd if=/dev/foo | gzip | check > outfile.gz", and it would be
trivial in such cases to update the EF if gzip reported the intended value.
(Of course, check reports something similar, but not many of us use it. ;-) )

Greg