Re: [Mingw-cross-env-list] New features: automatic "du", automatic log compression

[Volker Grabsch]

Tony Theodore schrieb:
> On 4 September 2011 08:04, Volker Grabsch <address@hidden> wrote:
> > On my system (Debian/Testing on amd64), this produces:
> >
> >    log/wxwidgets:du: 1737716 KiB
> >    log/gcc:du: 1551720 KiB
> 
> Curiously, the size varies significantly:
> 
> DFly- log/gcc:du: 1288086 KiB
> FBSD- log/gcc:du: 1354568 KiB
> OS X- log/gcc:du: 1001064 KiB

Note that gcc is compiled natively, so differences are to be
expected.

Maybe the sizes of *.o files vary a lot, probably due to
differences in the generated assembly code, or different
names (and thus name lengths) of the debug symbols.

It would be more interesting to check for differences in
the cross-compiled packages, because those are compiled
by the same compiler on all systems.

> I wonder if this it caused by the difference in block size between 
> filesystems?

That is also possible, but would be more visible on purely
cross-compiled packages rather than natively compiled packages.

> > So I'm thinking about compressing the log files on the fly.
> > [...]
> >    zgrep -H ^du: log/*.gz | sort -rnk2
> >    bzgrep -H ^du: log/*.bz2 | sort -rnk2
> >
> > So what do you think about this?
> 
> I imagine compressing the logs on the fly may become inconvenient when
> debugging (depending on your text editor of choice).

Well, I'm using less (or zless) for that purpose.

But I find this interesting. Do you open the logs in a
text editor? I've never thought about that.

What exactly are you doing, and in how far does this
conflict with on-the-fly compression?

> If we only
> compressed after a successful build, it would be fine, but the use
> case of sending uncompressed logs to the list wouldn't be served.

We could also compress them after an unsuccessful build,
because the build itself runs in a different Make process
than the surrounding build which catched stdout/stderr,
writes the log, and show the last few lines in case of
a failure.

So what's the important point here for you?

a) that you don't compress on-the-fly during the build.
b) that you only compress after a successful package build.

Those are two different things. (although b) implies a))

> Maybe we could have the compression switched on in the stable branch
> only, with the default branch only compressing on success (or not at
> all)?

Well, the compressed logs are especially useful for
development, I think, because there we'll have lots of
builds and partly builds, where the log files quickly
sum up to 100s of MiB or more.

> > If we do that, should we use gzip or bzip2? gzip decompresses
> > a lot faster, but bzip2 will make the logs grow about 22%
> > slower. So I guess this is about archiving vs. analyzing.
> >
> > I'd love to hear some more opinions / ideas about this.
> 
> Space isn't that much of a concern for me, so I'd opt for gzip (or
> xz). bzip2 makes sense when sending files though, particularly if
> there's a limit.

I just checked that this makes a difference for only about 3
packages. All other logs are either already small enough with
gzip, or are still too big for bzip2, assuming a 100 KiB limit.

> Anther option to reduce log size is to switch off the verbose rm (the
> -v option is unrecognised on OpenBSD, from memory), though this has
> been rejected before.

Do you have any data about how big the ratio between the "rm -v"
output and the rest of the log files is? I believe we could shorten
it by about 50% that way, while by compression we'll get down to 5%.

But I didn't measure those 50%. Do you have any data on that?

Also, maybe it makes sense to combine this with compression, but
I believe that the "rm -v" part is especially well compressible
anyway, exactly because of its redundancy.

> While we're talking about logs, one thing that I'd like to have
> automated would be replacing the value of $(TOP_DIR) with a short
> ~/mce (or similar) in all log files. It makes it much easier to
> compare logs from different systems, and would probably reduce the
> file size also.

I like this idea, although I need to clarify two things:

1) "~/mce" is misleading, as the build isn't necessarily
   anywhere in any home directory.

2) $(TOP_DIR) is only the place of the Makefile. I think you meant
   $(PWD), which is normally the same, but will do the right thing
   in case "make -f .../Makefile" is used. Yes, it is quite uncommon
   to build outside the source tree, but in theory we do support
   it, and I don't want to throw that feature away unless we get
   a real benefit (i.e. simplification) by doing so.

Regarding 1), we could simply use the exact string "$(PWD)" as
placeholder. :-)

Another possibility would be replacing $(PREFIX) and $(TMP_DIR)
instead of $(PWD), because theoretically, those could be set to
different values by the user.

On the other hand, $(PREFIX) and $(TMP_DIR) might be less
intuitively to grasp than $(PWD)/usr and $(PWD)/tmp-PKG.

What do you think?
   

Greets,
Volker

-- 
Volker Grabsch
---<<(())>>---