Re: [Qemu-devel] QEMU patch to allow VM introspection via libvmi

[Valerio Aimale]

On 10/27/15 9:00 AM, Markus Armbruster wrote:
> Valerio Aimale <address@hidden> writes:
>
> > On 10/26/15 11:52 AM, Eduardo Habkost wrote:
> > > I was trying to advocate the use of a shared mmap'ed region. The sharing
> > > would be two-ways (RW for both) between the QEMU virtualizer and the libvmi
> > > process. I envision that there could be a QEMU command line argument, such
> > > as "--mmap-guest-memory <filename>" Understand that Eric feels strongly the
> > > libvmi client should own the file name - I have not forgotten that. When
> > > that command line argument is given, as part of the guest initialization,
> > > QEMU creates a file of size equal to the size of the guest memory containing
> > > all zeros, mmaps that file to the guest memory with  PROT_READ|PROT_WRITE
> > > and MAP_FILE|MAP_SHARED, then starts the guest.
> > > This is basically what memory-backend-file (and the legacy -mem-path
> > > option) already does today, but it unlinks the file just after opening
> > > it. We can change it to accept a full filename and/or an option to make
> > > it not unlink the file after opening it.
> > >
> > > I don't remember if memory-backend-file is usable without -numa, but we
> > > could make it possible somehow.
> > Eduardo, I did try this approach. It takes 2 line changes in exec.c:
> > comment the unlink out, and making sure MAP_SHARED is used when
> > -mem-path and -mem-prealloc are given. It works beautifully, and
> > libvmi accesses are fast. However, the VM is slowed down to a crawl,
> > obviously, because each RAM access by the VM triggers a page fault on
> > the mmapped file. I don't think having a crawling VM is desirable, so
> > this approach goes out the door.
> Uh, I don't understand why "each RAM access by the VM triggers a page
> fault".  Can you show us the patch you used?
Sorry, too brief of an explanation. Every time the guest flips a byte in 
physical RAM, I think that triggers a page write to the mmaped file. My 
understanding is that, with MAP_SHARED, each write to RAM triggers a 
file write, hence the slowness. These are the simple changes I made, to 
test it - as a proof of concept.

in exec.c of the qemu-2.4.0.1 change

---
    fd = mkstemp(filename);
    if (fd < 0) {
        error_setg_errno(errp, errno,
                         "unable to create backing store for hugepages");
        g_free(filename);
        goto error;
    }
    unlink(filename);
    g_free(filename);

    memory = (memory+hpagesize-1) & ~(hpagesize-1);

    /*
     * ftruncate is not supported by hugetlbfs in older
     * hosts, so don't bother bailing out on errors.
     * If anything goes wrong with it under other filesystems,
     * mmap will fail.
     */
    if (ftruncate(fd, memory)) {
        perror("ftruncate");
    }

    area = mmap(0, memory, PROT_READ | PROT_WRITE,
                (block->flags & RAM_SHARED ? MAP_SHARED : MAP_PRIVATE),
                fd, 0);
---

to

---
    fd = mkstemp(filename);
    if (fd < 0) {
        error_setg_errno(errp, errno,
                         "unable to create backing store for hugepages");
        g_free(filename);
        goto error;
    }
    /* unlink(filename); */ /* Valerio's change to persist guest RAM 
mmaped file */
    g_free(filename);

    memory = (memory+hpagesize-1) & ~(hpagesize-1);

    /*
     * ftruncate is not supported by hugetlbfs in older
     * hosts, so don't bother bailing out on errors.
     * If anything goes wrong with it under other filesystems,
     * mmap will fail.
     */
    if (ftruncate(fd, memory)) {
        perror("ftruncate");
    }

    area = mmap(0, memory, PROT_READ | PROT_WRITE,
                MAP_FILE | MAP_SHARED, /* Valerio's change to persist 
guest RAM mmaped file */
                fd, 0);
---

then, recompile qemu.

Launch a VM as

/usr/local/bin/qemu-system-x86_64 -name Windows10 -S -machine 
pc-i440fx-2.4,accel=kvm,usb=off [...] -mem-prealloc -mem-path /tmp/maps

# I know -mem-path is deprecated, but I used for speeding up the proof 
of concept.

With the above command, I have a the following file

$ ls -l /tmp/maps/
-rw------- 1 libvirt-qemu kvm 2147483648 Oct 27 08:31 
qemu_back_mem.pc.ram.fP4sKH

which is a mmap of the Win VM physical RAM

$ hexdump -C /tmp/maps/qemu_back_mem.val.pc.ram.fP4sKH

00000000  53 ff 00 f0 53 ff 00 f0  c3 e2 00 f0 53 ff 00 f0 
|S...S.......S...|
[...]
00000760  24 02 c3 49 6e 76 61 6c  69 64 20 70 61 72 74 69 |$..Invalid 
parti|
00000770  74 69 6f 6e 20 74 61 62  6c 65 00 45 72 72 6f 72  |tion 
table.Error|
00000780  20 6c 6f 61 64 69 6e 67  20 6f 70 65 72 61 74 69  | loading 
operati|
00000790  6e 67 20 73 79 73 74 65  6d 00 4d 69 73 73 69 6e  |ng 
system.Missin|
000007a0  67 20 6f 70 65 72 61 74  69 6e 67 20 73 79 73 74  |g operating 
syst|
000007b0  65 6d 00 00 00 63 7b 9a  73 d8 99 ce 00 00 80 20 
|em...c{.s...... |
[...]

I did not try to mmap'ing to a file on a RAMdisk. Without physical disk 
I/O, the VM might run faster.