Re: [Mingw-cross-env-list] New features: automatic "du", automatic log compression

[Tony Theodore]

On 4 September 2011 21:39, Volker Grabsch <address@hidden> wrote:

[snip...]

>> > So I'm thinking about compressing the log files on the fly.
>> > [...]
>> >    zgrep -H ^du: log/*.gz | sort -rnk2
>> >    bzgrep -H ^du: log/*.bz2 | sort -rnk2
>> >
>> > So what do you think about this?
>>
>> I imagine compressing the logs on the fly may become inconvenient when
>> debugging (depending on your text editor of choice).
>
> Well, I'm using less (or zless) for that purpose.
>
> But I find this interesting. Do you open the logs in a
> text editor? I've never thought about that.

All the time... I've never really thought of another way. I normally
skim through with vi, then if I can't find anything obvious, open with
a gui editor. Most are frustratingly slow, but gvim (and relatives)
and bbedit can open and navigate large files easily.

> What exactly are you doing, and in how far does this
> conflict with on-the-fly compression?

I recently discovered that vi opens compressed files without any fuss,
so there's no conflict with on-the-fly compression (unless you want to
watch them with tail).

[...]

>> > If we do that, should we use gzip or bzip2? gzip decompresses
>> > a lot faster, but bzip2 will make the logs grow about 22%
>> > slower. So I guess this is about archiving vs. analyzing.
>> >
>> > I'd love to hear some more opinions / ideas about this.
>>
>> Space isn't that much of a concern for me, so I'd opt for gzip (or
>> xz). bzip2 makes sense when sending files though, particularly if
>> there's a limit.
>
> I just checked that this makes a difference for only about 3
> packages. All other logs are either already small enough with
> gzip, or are still too big for bzip2, assuming a 100 KiB limit.

It's probably safe to assume that failed logs will always be smaller
than successful ones, so gzip should be fine.

>> Anther option to reduce log size is to switch off the verbose rm (the
>> -v option is unrecognised on OpenBSD, from memory), though this has
>> been rejected before.
>
> Do you have any data about how big the ratio between the "rm -v"
> output and the rest of the log files is? I believe we could shorten
> it by about 50% that way, while by compression we'll get down to 5%.
>
> But I didn't measure those 50%. Do you have any data on that?
>
> Also, maybe it makes sense to combine this with compression, but
> I believe that the "rm -v" part is especially well compressible
> anyway, exactly because of its redundancy.

I didn't check the size difference, but thinking some more, the "rm
-v" stage will likely never occur in a failed build, so this won't
affect the file size where it matters the most (sending to the list).

There is a rare case you can trigger on OSX by opening the tmp-*
directory in Finder which will create a .DS_Store file and cause rm to
fail with a "directory not empty" error, but this isn't worth
considering.

>> While we're talking about logs, one thing that I'd like to have
>> automated would be replacing the value of $(TOP_DIR) with a short
>> ~/mce (or similar) in all log files. It makes it much easier to
>> compare logs from different systems, and would probably reduce the
>> file size also.
>
> I like this idea, although I need to clarify two things:
>
> 1) "~/mce" is misleading, as the build isn't necessarily
>   anywhere in any home directory.
>
> 2) $(TOP_DIR) is only the place of the Makefile. I think you meant
>   $(PWD), which is normally the same, but will do the right thing
>   in case "make -f .../Makefile" is used. Yes, it is quite uncommon
>   to build outside the source tree, but in theory we do support
>   it, and I don't want to throw that feature away unless we get
>   a real benefit (i.e. simplification) by doing so.
>
> Regarding 1), we could simply use the exact string "$(PWD)" as
> placeholder. :-)
>
> Another possibility would be replacing $(PREFIX) and $(TMP_DIR)
> instead of $(PWD), because theoretically, those could be set to
> different values by the user.
>
> On the other hand, $(PREFIX) and $(TMP_DIR) might be less
> intuitively to grasp than $(PWD)/usr and $(PWD)/tmp-PKG.
>
> What do you think?

I like the idea to use a literal "$(...)" placeholder - it's very
clear. I'm ambivalent about the difference between $(PREFIX) and
$(PWD)/usr, other than with $(PREFIX) it's obvious where the
placeholder ends.

My main use case for this is doing diffs, so I'd be pleased with
anything consistent. If I had a preference, it would be lowercase
$(prefix) and $(tmp_dir) to be less distracting :)

Thanks,

Tony