Re: [Qemu-devel] [PATCH] makefile: detect corrupted elf files

[Blue Swirl]

On Sun, May 26, 2013 at 8:15 PM, Michael S. Tsirkin <address@hidden> wrote:
> On Sun, May 26, 2013 at 07:28:40PM +0000, Blue Swirl wrote:
>> On Sun, May 26, 2013 at 6:24 PM, Michael S. Tsirkin <address@hidden> wrote:
>> > On Sun, May 26, 2013 at 06:20:17PM +0000, Blue Swirl wrote:
>> >> On Sun, May 26, 2013 at 1:40 PM, Michael S. Tsirkin <address@hidden> 
>> >> wrote:
>> >> > On Sun, May 26, 2013 at 02:36:28PM +0100, Peter Maydell wrote:
>> >> >> On 26 May 2013 13:31, Michael S. Tsirkin <address@hidden> wrote:
>> >> >> > On Sun, May 26, 2013 at 10:12:21AM +0100, Peter Maydell wrote:
>> >> >> >> I definitely think individual project makefiles are the wrong place
>> >> >> >> to fix this. If create-as-temp-and-rename is useful functionality
>> >> >> >> it needs to go in the compiler so that everybody benefits.
>> >> >> >
>> >> >> > This will not help users on existing systems.
>> >> >> > Also it's not just compiler. We'd have to do it in linker,
>> >> >> > asm, ... lots of work.
>> >> >>
>> >> >> This is clearly less work than implementing it in the makefile
>> >> >> of every single open source project in the world (or even every
>> >> >> single open source project in Debian).
>> >> >
>> >> > You seem to have removed the part that explained that
>> >> > 1. we run scripts in our makefiles so need to handle that anyway
>> >> > 2. we care about users on existing systems
>> >>
>> >> A generic hook (default none, or maybe "test -s") after object
>> >> production and before linkage should be enough but would scale to SHA
>> >> producing/verifying tools.
>> >>
>> >> >
>> >> > This means that we would need the fix in our makefiles even
>> >> > if compiler and linker gain this feature.
>> >>
>> >> Depends on the feature. If the object files have robust checksums
>> >> which are checked after output and before input, this should be
>> >> transparent to the build system.
>> >>
>> >> >
>> >> >> > You are wellcome to implement this in compiler/linker/etc if you like
>> >> >> > but we will still want to handle it in our makefile as well.
>> >> >>
>> >> >> I specifically don't want it handled in our makefiles because
>> >> >> it's the wrong place to fix the problem and it will make
>> >> >> our build system more complicated.
>> >>
>> >> +1
>> >>
>> >> Also, what is the worst case scenario? The link fails and you have to
>> >> clean up and rebuild? An automated build system can't produce the
>> >> expected output if the build machine is unreliable?
>> >
>> > It's a simple issue.
>> > Each time I reboot during build, I have to make clean and rebuild.
>> > This wastes my time so I looked for ways to save the time.
>>
>> Compile under a stable kernel and test the bleeding edge kernel only
>> as KVM guest? Get a different box for testing or compiling? Run 'sync'
>> every time gcc finishes?
>
> What's the question here?

The question is if any of the suggestions solves the problem?

Also how about something this: post boot, find -name '*.o' | xargs -iF
sh -c 'if test ! -s F; then rm F;fi'

>
>> Don't you have bigger problems with file systems due to the crashes?
>
> As it happens, no. Maybe because I'm using ext4.
> Maybe I'm lucky.
>
>> > On my system at least, it has no measureable cost,
>> > likely also because size only looks at headers and metadata.
>>
>> For example on OpenBSD, 'size' does not seem to come from binutils, so
>> there could be portability issues.
>
> True, I'm not saying it's perfect.
>
>> >
>> > If others are not interested, I can keep it out of tree.
>>
>> I've had problems with disk close to full, so I'm semi-interested if
>> the solution does not slow down others and it's not too ugly.
>
> I think the simplest way to do it is to change makefile to unlike, create
> then rename. Then you are safe against abrupt killing or crashing make.
> And with a journaled fs, if you also have e.g. linux ext4 and mount with
> data=ordered, you are safe against power failures.
>
> It shouldn't be hard to do and I don't expect this to have any
> measureable speed impact.  What do you think?

I'd prefer a more generic solution, like the hook. What you propose
wouldn't protect from the disk full scenario.

>
>> >
>> >> >>
>> >> >> -- PMM
>> >> >
>> >> >