Re: [Qemu-devel] uniquely identifying KDUMP files that originate from QEMU

[Laszlo Ersek]

On 11/11/14 18:27, Christopher Covington wrote:
> On 11/11/2014 06:22 AM, Laszlo Ersek wrote:
>> (Note: I'm not subscribed to either qemu-devel or the kexec list; please
>> keep me CC'd.)
>>
>> QEMU is able to dump the guest's memory in KDUMP format (kdump-zlib,
>> kdump-lzo, kdump-snappy) with the "dump-guest-memory" QMP command.
>>
>> The resultant vmcore is usually analyzed with the "crash" utility.
>>
>> The original tool producing such files is kdump. Unlike the procedure
>> performed by QEMU, kdump runs from *within* the guest (under a kexec'd
>> kdump kernel), and has more information about the original guest kernel
>> state (which is being dumped) than QEMU. To QEMU, the guest kernel state
>> is opaque.
>>
>> For this reason, the kdump preparation logic in QEMU hardcodes a number
>> of fields in the kdump header. The direct issue is the "phys_base"
>> field. Refer to dump.c, functions create_header32(), create_header64(),
>> and "include/sysemu/dump.h", macro PHYS_BASE (with the replacement text
>> "0").
>>
>> http://git.qemu.org/?p=qemu.git;a=blob;f=dump.c;h=9c7dad8f865af3b778589dd0847e450ba9a75b9d;hb=HEAD
>>
>> http://git.qemu.org/?p=qemu.git;a=blob;f=include/sysemu/dump.h;h=7e4ec5c7d96fb39c943d970d1683aa2dc171c933;hb=HEAD
>>
>> This works in most cases, because the guest Linux kernel indeed tends to
>> be loaded at guest-phys address 0. However, when the guest Linux kernel
>> is booted on top of OVMF (which has a somewhat unusual UEFI memory map),
>> then the guest Linux kernel is loaded at 16MB, thereby getting out of
>> sync with the phys_base=0 setting visible in the KDUMP header.
>>
>> This trips up the "crash" utility.
>>
>> Dave worked around the issue in "crash" for ELF format dumps -- "crash"
>> can identify QEMU as the originator of the vmcore by finding the QEMU
>> notes in the ELF vmcore. If those are present, then "crash" employs a
>> heuristic, probing for a phys_base up to 32MB, in 1MB steps.
> 
> What advantages does KDUMP have over ELF?

This has been discussed, but I'd like to give a short perspective from
personal experience.

The more obvious advantage is the smaller size, due to (a) per-page
compression (which preserves random-access for "crash"), and (b) zero
page sharing. A smaller dump file is easier to store, and easier to
upload if you're requesting assitance with debugging.

The perhaps less obvious advantage is the speed at which qemu writes the
dump. We're talking orders of magnitude, especially on rotational media.
This is because lzo and snappy are *incredibly* fast (put differently:
they incur very little CPU penalty for the same guest RAM size). The CPU
penalty is actually so small that in almost all cases the dumping
procedure stays IO-bound (in my experience: even on an SSD!). Now
combine that with a potential reduction of 4GB -> 256MB in size: that's
a sixteen-fold speedup.

(I'm allowed to praise this qemu feature, I didn't write it. :))

Thanks
Laszlo