Re: [Qemu-devel] [PATCH v8 4/8] ACPI: Add Virtual Machine Generation ID support

[Ben Warren]

> On Feb 17, 2017, at 12:07 PM, Laszlo Ersek <address@hidden> wrote:
> 
> On 02/17/17 17:03, Laszlo Ersek wrote:
>> On 02/17/17 16:33, Ben Warren wrote:
>>> 
>>>> On Feb 17, 2017, at 2:43 AM, Igor Mammedov <address@hidden
>>>> <mailto:address@hidden>> wrote:
>>>> 
>>>> On Thu, 16 Feb 2017 15:15:36 -0800
>>>> address@hidden <mailto:address@hidden> wrote:
>>>> 
>>>>> From: Ben Warren <address@hidden <mailto:address@hidden>>
>>>>> 
>>>>> This implements the VM Generation ID feature by passing a 128-bit
>>>>> GUID to the guest via a fw_cfg blob.
>>>>> Any time the GUID changes, an ACPI notify event is sent to the guest
>>>>> 
>>>>> The user interface is a simple device with one parameter:
>>>>> - guid (string, must be "auto" or in UUID format
>>>>>  xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
>>>> I've given it some testing with WS2012R2 and v4 patches for Seabios,
>>>> 
>>>> Windows is able to read initial GUID allocation and writeback
>>>> seems to work somehow:
>>>> 
>>>> (qemu) info vm-generation-id
>>>> c109c09b-0e8b-42d5-9b33-8409c9dcd16c
>>>> 
>>>> vmgenid client in Windows reads it as 2 following 64bit integers:
>>>> 42d50e8bc109c09b:6cd1dcc90984339b
>>>> 
>>>> However update path/restore from snapshot doesn't
>>>> here is as I've tested it:
>>>> 
>>>> qemu-system-x86_64 -device vmgenid,id=testvgid,guid=auto -monitor stdio
>>>> (qemu) info vm-generation-id
>>>> c109c09b-0e8b-42d5-9b33-8409c9dcd16c
>>>> (qemu) stop
>>>> (qemu) migrate "exec:gzip -c > STATEFILE.gz"
>>>> (qemu) quit
>>>> 
>>>> qemu-system-x86_64 -device vmgenid,id=testvgid,guid=auto -monitor stdio
>>>> -incoming "exec: gzip -c -d STATEFILE.gz"
>>>> (qemu) info vm-generation-id
>>>> 28b587fa-991b-4267-80d7-9cf28b746fe9
>>>> 
>>>> guest
>>>> 1. doesn't get GPE notification that it must receive
>>>> 2. vmgenid client in Windows reads the same value
>>>>     42d50e8bc109c09b:6cd1dcc90984339b
>>>> 
>>> Strange, this was working for me, but with a slightly different test method:
>>> 
>>>  * I use virsh save/restore
>> 
>> Awesome, this actually what I should try. All my guests are managed by
>> libvirt (with the occasional <qemu:arg>, for development), and direct
>> QEMU monitor commands such as
>> 
>>  virsh qemu-monitor-command ovmf.rhel7 --hmp 'info vm-generation-id'
>> 
>> only work for me if they are reasonably non-intrusive.
>> 
>>>  * While I do later testing with Windows, during development I use a
>>>    Linux kernel module I wrote that keeps track of GUID and
>>>    notifications.  I’m happy to share this with you if interested.
>> 
>> Please do. If you have a public git repo somewhere, that would be
>> awesome. (Bonus points if the module builds out-of-tree, if the
>> kernel-devel package is installed.)
>> 
>> NB: while the set-id monitor command was part of the series, I did test
>> it to the extent that I checked the SCI ("ACPI interrupt") count in the
>> guest, in /proc/interrupts. I did see it increase, so minimally the SCI
>> injection was fine.
> 
> So, I did some testing with a RHEL-7 guest. I passed '-device
> vmgenid=auto' to QEMU using the <qemu:arg> element in the domain XML.
> 
> (1) I started the guest normally, and grepped /proc/interrupts for
> "acpi". Zero interrupts on either VCPU.
> 
> (2) Dumped the guest RAM to a file with "virsh dump ... --memory-only",
> opened it with crash, and listed the 16 GUID bytes at the offset that
> the firmware (OVMF) reported at startup.
> 
> (3) cycled through "virsh managedsave" and "virsh start"
> 
> (4) grepped /proc/interrupts again for "acpi". One interrupt had been
> delivered to one of the VCPUs, all others were zero.
> 
> (5) Repeated step (2). The bytes listed this time were different.
> 
> (6) Issued "virsh qemu-monitor-command ovmf.rhel7 --hmp 'info
> vm-generation-id", and compared the output against the bytes dumped
> (with crash) from guest memory, in step 5. They were a match.
> 
> So, to me it seems like the SCI is injected, and the memory contents are
> changed.
> 
> ---*---
> 
> Windows Server 2012 R2 test:
> 
> (7) booted the guest similarly with '-device vmgenid=auto' via
> <qemu:arg> in the domain XML.
> 
> (8) Initial check from the host side:
> 
> $ virsh qemu-monitor-command ovmf.win2012r2.q35 \
>    --hmp 'info vm-generation-id'
> a3f7c334-7dc4-4694-8b8f-abf52abb072f
> 
> (9) Verifying the same from within, using Vadim's program (note: I
> logged into the VM with ssh, using Cygwin's SSHD in the guest):
> 
> $ ./vmgenid.exe
> VmCounterValue: 46947dc4a3f7c334:2f07bb2af5ab8f8b
> 0x34 0xc3 0xf7 0xa3 0xc4 0x7d 0x94 0x46 0x8b 0x8f 0xab 0xf5 0x2a 0xbb
> 0x07 0x2f
> 
> This is a match, so the initial setup works. (Look only at the raw byte
> dump in the second line -- it matches the Little Endian UUID
> representation as specified in the SMBIOS spec!)
> 
> (10) Logged out of the guest (with ssh), cycled through "virsh
> managedsave" and "virsh start" for the domain, logged back in.
> 
> (11) in the guest:
> 
> $ ./vmgenid.exe
> VmCounterValue: 4a12296b382162da:6c00d1a52699b7bd
> 0xda 0x62 0x21 0x38 0x6b 0x29 0x12 0x4a 0xbd 0xb7 0x99 0x26 0xa5 0xd1
> 0x00 0x6c
> 
> (12) on the host:
> 
> $ virsh qemu-monitor-command ovmf.win2012r2.q35 \
>      --hmp 'info vm-generation-id'
> 382162da-296b-4a12-bdb7-9926a5d1006c
> 
> This is again a match. (Again, look only at the raw byte dump from
> vmgenid.exe under (11), and consider the BE/LE conversion for the first
> three segments!)
> 
> (13) Logged out of the guest with ssh, and started Vadim's other program
> (vmgenid_wait.exe), this time from a normal CMD window on the GUI. The
> program started, reproduced the above output (seen under (11)), and then
> went to sleep (waiting).
> 
> (14) cycled through "virsh managedsave" and "virsh start" for the domain.
> 
> (15) The domain resumed, and Vadim's vmgenid_wait.exe woke up, printing
> (manual transcript):
> 
> VmCounterValue changed to: 495ba7807ed37772:195d0cff681f7a7
> 
> Please refer to the following screenshot:
> 
> http://people.redhat.com/~lersek/vmgenid-dd1f68c5-89b0-4458-84fa-de9e3d23f4cb/Screenshot_ovmf.win2012r2.q35_2017-02-17_20:34:41.png
> 
> (16) on the host:
> 
> $ virsh qemu-monitor-command ovmf.win2012r2.q35 \
>      --hmp 'info vm-generation-id'
> 
> 7ed37772-a780-495b-a7f7-81f6cfd09501
> 
> This is again a match. It is not easy to see, because Vadim's
> "vmgenid_wait.exe" does not print the raw byte dump after it wakes up;
> the raw byte dump is only printed before it goes to sleep. After wakeup,
> it only dumps the composed values. Somewhat more confusingly, the 64-bit
> hex integers are not zero padded, we'll have to make up for that manually.
> 
> So here goes:
> 
>   [A]      [B]  [C]  [D]  [E]
>   7ed37772-a780-495b-a7f7-81f6cfd09501 (from the host)
> 
>   [C]  [B]  [A]        [E']         [D']
>   495b a780 7ed37772 : 0195d0cff681 f7a7 (from vmgenid_wait.exe)
>                        ^
>                        zero padding added manually
> 
> The parts marked with an apostrophe (') are reversed, byte-wise.
> 
> So, I'm going to have to declare this "working by design".
> 
> Confirming my earlier Tested-by (same patches as before):
> 
> Tested-by: Laszlo Ersek <address@hidden>
> 
> What could be the difference between Igor's setup and mine? Perhaps the
> BIOS? (Again, I used OVMF.)
> 
> The "managedsave" command of virsh boils down to (see
> "src/qemu/qemu_driver.c" in the libvirt source):
> 
> qemuDomainManagedSave()
>  qemuDomainSaveInternal()
>    qemuProcessStopCPUs()
>    qemuDomainSaveMemory()
>      qemuDomainSaveHeader()
>      qemuMigrationToFile()
>        qemuMonitorMigrateToFd()
>          ...
>    qemuProcessStop()
> 
> I capture all traffic between libvirt and the QEMU monitor, and between
> libvirt and the QEMU guest agent, in the libvirt log file, as a rule, so
> I can paste the relevant lines:
> 
> Libvirt sending the file descriptor to QEMU:
> 
> 2017-02-17 19:31:54.305+0000: 16586: debug :
> qemuMonitorJSONCommandWithFd:296 : Send command
> '{"execute":"getfd","arguments":{"fdname":"migrate"},"id":"libvirt-30"}'
> for write with FD 26
> 
> Libvirt starting the migration:
> 
> 2017-02-17 19:31:54.306+0000: 16586: debug :
> qemuMonitorJSONCommandWithFd:296 : Send command
> '{"execute":"migrate","arguments":{"detach":true,"blk":false,"inc":false,"uri":"fd:migrate"},"id":"libvirt-31"}'
> for write with FD -1
> 
> Then loading it:
> 
> 2017-02-17 19:32:02.083+0000: 16585: debug :
> qemuMonitorJSONCommandWithFd:296 : Send command
> '{"execute":"migrate-incoming","arguments":{"uri":"fd:25"},"id":"libvirt-17"}'
> for write with FD -1
> 
> I don't have the slightest idea why it failed for Igor -- I can only
> suspect the SeaBIOS patches.
> 
> Note that in the SeaBIOS discussion, Ben mentioned that he wasn't
> actually seeing the fw_cfg writes in QEMU on the S3 resume path, despite
> SeaBIOS attempting them. So, perhaps, is there a bug in the latest
> SeaBIOS patches that prevent fw_cfg writes completely, even on the
> normal boot path? That would be consistent with Igor's results: the
> initial download works (hence the first GUID can be seen), but then the
> update does not work (because QEMU has not received the address).
> 
> Thanks
> Laszlo

My functional testing was also successful.  I did make the one SeaBIOS change 
suggested by Laszlo, which I’ll post in a minute as v5.

In my case, I used a Windows 2012R2 VM that is a fully-joined Active Directory 
Domain Controller, so was able to query some VM Generation ID-derived data via 
power shell.  This use case is the reason why VM Generation ID exists in the 
first place.
We manage the VM using libvirt with this extra snippet in the domain XML:

  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vmgenid,guid=auto'/>
  </qemu:commandline>

The commands I used within Windows are:

1.  repadmin /showrepl
        This shows the replication state of the Domain Controller, most 
importantly printing the “DSA invocation ID”.  This number is derived from VM 
Generation ID, and changes whenever Active Directory considers its local data 
dirty.  It’s an indicator the Windows detected a change in VM Generation ID.

2.  (Get-ADObject <x500 stuff> -server <domain controller> -property 
meds-generationid).’msds-generationid’
        This gets the least-significant 64 bits of the VM Generation ID GUID
        
        http://blogs.metcorpconsulting.com/tech/?p=1594


Initial State:
=========

virsh # qemu-monitor-command --hmp 20 info vm-generation-id
d43edc6c-9686-4bf3-8621-2a4be879eb22

PS C:\Users\bran> (Get-ADObject "cn=whiteharbor,ou=Domain 
Controllers,DC=thenorth,dc=ws" -server whiteharbor.thenorth.ws
 -property msds-generationid).'msds-generationid'
108
220
62
212
134
150
243
75

***
NOTE:  108,220,62,212,212,134,150,243,75 == 6c,dc,3e,d4,86,96,f3,4b  ==> The 
GUID shown in HMP in Little-endian format!
***

PS C:\Users\bran> repadmin /showrepl

Repadmin: running command /showrepl against full DC localhost
Default-First-Site-Name\WHITEHARBOR
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: 3dcfed54-28d4-448e-9087-140dd251e1b4
DSA invocationID: 56ca3644-8bc3-4dd8-9860-df0b3f5ce06f

Reboot within Windows:
==================

Nothing changed


After virsh save, virsh restore:
=======================

virsh # qemu-monitor-command --hmp 22 info vm-generation-id
573a0dbc-6eca-49fd-a632-f4f1e016502a

PS C:\Users\bran> (Get-ADObject "cn=whiteharbor,ou=Domain 
Controllers,DC=thenorth,dc=ws" -server whiteharbor.thenorth.ws
 -property msds-generationid).'msds-generationid'
188
13
58
87
202
110
253
73
PS C:\Users\bran> repadmin /showrepl

Repadmin: running command /showrepl against full DC localhost
Default-First-Site-Name\WHITEHARBOR
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: 3dcfed54-28d4-448e-9087-140dd251e1b4
DSA invocationID: 7b6dda08-d23b-4ca2-982d-774f9c214442


I’m satisfied that the current patches work and don’t really understand Igor’s 
failure, other than maybe the SeaBIOS difference.

regards,
Ben

smime.p7s
Description: S/MIME cryptographic signature