bug-make
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [bug #40639] GNU Make with profiling information


From: Tim Murphy
Subject: Re: [bug #40639] GNU Make with profiling information
Date: Sun, 15 Dec 2013 13:38:56 +0000

I suppose I'm skirting around saying that I think gnu make needs an
output format in the same way that valgrind has "--xml=yes".  I'm not
an XML fan really - JSON might be an alternative.

It isn't your problem to provide such a mechanism and I realise it's
unfair of me to give you any sort of hard time about it. This feature
is just a small example of how gnu make will evolve an irregular
output format that's not easy to change once its "finalised" because
it's not designed to be extendable.

Extra information like timing could be a new tag and any parser that
didn't understand the tag could just ignore it and would be able to do
so without ever making a mistake.   Also the tag would be within the
tags of the job it referred to so it would be easy to see which target
it related to and any other statistical information that one wanted to
record about a job would also be there so you would have a guaranteed
way to tie information together.  e.g. one might want to record how
much IO some job did or it might be important for the recipe itself to
be able to report metadata about the file like that it's meant to be
placed in the output ROFS image in a certain directory.

If you're timing things then you might think about timing out (I have
needed that) and also about retries and this might lead you to add new
output fields e.g. "num_tries=2".  With JSON or XML new attributes can
be easily and reliably ignored by a parser and what's more the
nastiest parts of the parse job can be done by any number of libraries
out there.

I like the idea of external files because it means that log output is
unaffected so nobody's parsing will be upset and thus we are not
committed in the future to maintaining (or maintaining parser for)
something that isn't really the final answer to make output.

I also think that floating point output is great - much easier to
parse.  Even if your time storage mechanism can't be a double on some
platform, it would be great if the output could be 1.xyxyyx seconds
where the precision might vary depending on the architecture and OS.

Anyhow it's just my 2p.

Cheers and merry christmas,

Tim

On 15 December 2013 09:00, Eddy Petrișor <address@hidden> wrote:
>
> Pe 29.11.2013 12:30, "Tim Murphy" <address@hidden> a scris:
>
>
>>
>> When I did something similar (which I am not allowed to post) I made a
>> new file for each submake and the output filename "base" was in an
>> environment variable.  I realise that nobody ever wants to change the
>
> Does it make sense to separate the profiling info into one file per process
> and then try to merge the information back?
>
> The second proposal I sent prints the pid for the 'short' and 'long'
> formats, so if one wants to focus on a specific submake, one can grep for
> the pid in the whole profile listing. The first proposal had a flexible
> output format, so one could choose to print the pid info then filter that.
>
>
>
>> way they implemented something but I thought I'd mention it.
>>
>> Also, having written a few external tools, I always appreciate it when
>> the creator of some information source makes it easy for me to recover
>> data and the relationships between each item.  I also like it when
>
> Not sure what you are suggesting here. What relation would you want and
> think is not available?
>
>> there's not really any way for my parser to potentially make some
>> mistake just because a tool prints out something similar to what I'm
>
> The first version allowed a custom format which allowed any literal string
> to be printed anywhere, alongside with the target name, start and end
> timestamps, pid duration and recursivity level.
>
> The second proposal allows a custom prefix before the profiling info.
>
> So in both cases a simple grep for a specific prefix which can be chosen by
> the user is possible. Do you think that would be enough for your
> hypothetical parser?
>
>> looking for.  I also like it when the format is such that it can
>> change a bit without necessarily forcing me to rewrite my parser.
>
> Not sure what would you think could change in the output without informing
> the parser at all.
>
> Currently the prefix and the field separators are customisable.
>
> Maybe an example could help.
>
>>
>> Again, this is only an opinion - it carries no weight.
>>
>> Regards,
>>
>> Tim
>>
>> On 29 November 2013 02:43, Eddy Petrișor <address@hidden> wrote:
>> >
>> > Pe 27.11.2013 13:12, "Tim Murphy" <address@hidden> a scris:
>> >
>> >
>> >>
>> >> FWIW
>> >>
>> >> As for profiling output, this should probably go to a file (possibly
>> >> with a .PID on the end) , not stdout .....unless..... you start to
>> >> embrace the idea of structured output for everything that make
>> >> produces.
>> >
>> > The profiling info is printed on stderr, not on stdout.
>> > I also initially thought I should isolate the information in a different
>> > file, but the problem is that for recursive make that file descriptor
>> > must
>> > be passed on every fork, which is a very intrusive change to make and
>> > very
>> > wasteful (passing an extra file descriptor, even when unnecessary).
>> >
>> > So I opted for the next best thing, prefixing the output with something
>> > a
>> > grep could anchor to.
>> >
>> >>
>> >> I have used XML before and it has advantages, not the least of which
>> >> is that it is somewhat robust in the face of corruption and that does
>> >> happen. I prefer JSON though because it's more readable and it's very
>> >> easy to parse.
>> >
>> > As I said before, I think that's the job of an external tool.
>> >
>> >>
>> >> On 27 November 2013 07:56, Eddy Petrișor <address@hidden>
>> >> wrote:
>> >> >
>> >> > Pe 25.11.2013 11:09, "Reinier Post" <address@hidden> a scris:
>> >> >
>> >> >
>> >> >> > > Can't this functionality be provided by a wrapper $SHELL?
>> >> >> > > Sure, it's an extra exec(),
>> >> >> > > but it will keep the make code base simpler.
>> >> >> >
>> >> >> > I'm not sure if I understand what you mean. Do you mean wrapping
>> >> >> > all
>> >> >> > target invokes in a $SHELL?
>> >> >>
>> >> >> Yes; something line /usr/bin/time.
>> >> >
>> >> > Not on Windows.
>> >> >
>> >> >> > If that's the case, you don't get the actual/real-time
>> >> >> > measurement,
>> >> >> > and
>> >> >> > that might skew your profiling information.
>> >> >>
>> >> >> True.  Hnow how much of a difference does it make?
>> >> >
>> >> > Depends on how badly the build system is written. And that's what the
>> >> > target
>> >> > of the profiling is, finding out how bad is it. I feel that
>> >> > penalising
>> >> > the
>> >> > most exactly the build systems that need the least the slow down it's
>> >> > bad
>> >> > design.
>> >> >
>> >> >> > I have some build systems that can take easily over 12 hours to do
>> >> >> > a
>> >> >> > full build (several thousands of elfs in recursive
>> >> >> > make calls - I know that's bad, I am working on it), so any extra
>> >> >> > time
>> >> >> > waste just for the measurement can mean
>> >> >> > significant delays in getting the results.
>> >> >>
>> >> >> If it's 15 minutes, they are not the thing you want to optimize
>> >> >> first.
>> >> >> The build is probably running overnight anyway.
>> >> >
>> >> > If it's a live/production system, then any unknown slow down time
>> >> > might
>> >> > not
>> >> > be acceptable.
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > Bug-make mailing list
>> >> > address@hidden
>> >> > https://lists.gnu.org/mailman/listinfo/bug-make
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> You could help some brave and decent people to have access to
>> >> uncensored news by making a donation at:
>> >>
>> >> http://www.thezimbabwean.co.uk/friends/
>>
>>
>>
>> --
>> You could help some brave and decent people to have access to
>> uncensored news by making a donation at:
>>
>> http://www.thezimbabwean.co.uk/friends/



-- 
You could help some brave and decent people to have access to
uncensored news by making a donation at:

http://www.thezimbabwean.co.uk/friends/



reply via email to

[Prev in Thread] Current Thread [Next in Thread]