Re: [Qemu-devel] [PATCH v4 0/9] Make 'dump-guest-memory' dump in kdump-compressed format

[Stefan Hajnoczi]

On Thu, Jun 27, 2013 at 03:11:09PM +0800, Qiao Nuohan wrote:
> You asked about using ELF more efficiently. For implementing *excluding zero*
> pages, *PT_LOAD* can be made use of. p_memsz and p_filesz fields of PT_LOAD
> entry are used to describe memory size and the size of corresponding data in
> dump file respectively. Blocks only get zero pages in it will have *p_filesz*
> set to 0.
> 
> However, such implementation faces a problem: the number of PT_LOAD may
> *exceed* the range. As zero pages occur *arbitrarily*, the number of PT_LOAD,
> at the worst case, may reach the number of the total physical page frames. For
> example, 256MB have 2^16 page frames may exceed the range of e_phnum. Although
> sh_info is extended to store PT_LOAD when e_phnum is not enough, sh_info that
> has 2^32 range may also exceed if the guest machine has more-than-16TB 
> physical
> memory (it won't occur soon, but it will happen one day).

A simple patch to do this would scan memory for zero regions
write_elf_loads().  It could avoid a huge number of PT_LOAD entries by
setting a minimum threshold like 64 KB or 128 KB.  This would eliminate
the big zero regions from the ELF file.

Of course it doesn't compress the memory contents so it's still going to
be bigger than a compressed kdump FILEDUMP.

The interesting question is how effective this approach is.  If it's
good enough then it would be a fairly simple modification to dump.c.

> OTOH, the reason why I chose kdump-compressed format is ELF doesn't support
> compression and filtering yet. To implement compression and filtering on ELF,
> we need to consider specific ABI among qemu, crash utility and makedumpfile.
> After that, another work needs to be done to port them in specific
> distributions.
> 
> Compared kdump-compressed format with ELF, compression and filtering is
> supported and we don't need to modify tools like crash and makedumpfile.
> Considering these reasons, kdump-compressed format is better than ELF. It get
> more merit.

I meant simply compressing the ELF output during creation.  But I guess
the tools need random access to the ELF file, which is inefficient with
zlib and friends.

> According to your comments, I think your objection first comes from the
> *temporary files*.

I'm not happy with duplicating the kdump FILEDUMP format code into QEMU
because it is relatively big and ugly (temporary files make up a large
part of that).

That said, compressed kdump FILEDUMP might really be the only option
that meets your needs.  I'm just surprised at all the extra work
necessary to make the dump compact.

> What if temporary files won't be used? Flatten kdump-
> compressed format is supported by crash and makedumpfile which offers a
> mechanism to avoid seek in the case of sending data through *pipe*, then I
> don't need to cache pages' data in temporary files.

If it makes the code simpler and smaller it would be nice.

Stefan