Re: [Mingw-cross-env-list] New features: automatic "du", automatic log compression

[Tony Theodore]

On 4 September 2011 08:04, Volker Grabsch <address@hidden> wrote:
> Dear mingw-cross-env people,
>
> I just added a new feature to mingw-cross-env so that "du"
> is called immediately after each successful build of a
> package. It should also work on BSD systems.

Works on DragonFly, FreeBSD, and OSX; though none of these recognise
the --max-depth option. Should the order of statements be reversed?

[...]

> On my system (Debian/Testing on amd64), this produces:
>
>    log/wxwidgets:du: 1737716 KiB
>    log/gcc:du: 1551720 KiB

Curiously, the size varies significantly:

DFly- log/gcc:du: 1288086 KiB
FBSD- log/gcc:du: 1354568 KiB
OS X- log/gcc:du: 1001064 KiB

I wonder if this it caused by the difference in block size between filesystems?

[...]

> So I'm thinking about compressing the log files on the fly.
>
> This would keep the logs small. It would also solve the
> problem that people accidently post uncompressed logs to this
> list, triggering the maximum attachment size. Statistical
> analysis would still be possible using zgrep or bzgrep, e.g.
>
>    zgrep -H ^du: log/*.gz | sort -rnk2
>    bzgrep -H ^du: log/*.bz2 | sort -rnk2
>
> So what do you think about this?

I imagine compressing the logs on the fly may become inconvenient when
debugging (depending on your text editor of choice). If we only
compressed after a successful build, it would be fine, but the use
case of sending uncompressed logs to the list wouldn't be served.
Maybe we could have the compression switched on in the stable branch
only, with the default branch only compressing on success (or not at
all)?

> If we do that, should we use gzip or bzip2? gzip decompresses
> a lot faster, but bzip2 will make the logs grow about 22%
> slower. So I guess this is about archiving vs. analyzing.
>
> I'd love to hear some more opinions / ideas about this.

Space isn't that much of a concern for me, so I'd opt for gzip (or
xz). bzip2 makes sense when sending files though, particularly if
there's a limit.

Anther option to reduce log size is to switch off the verbose rm (the
-v option is unrecognised on OpenBSD, from memory), though this has
been rejected before.

While we're talking about logs, one thing that I'd like to have
automated would be replacing the value of $(TOP_DIR) with a short
~/mce (or similar) in all log files. It makes it much easier to
compare logs from different systems, and would probably reduce the
file size also.

Cheers,

Tony