Re: [Qemu-devel] WIP: Migration format: ASN.1/BER schema

[Michael S. Tsirkin]

On Fri, Mar 07, 2014 at 04:55:03PM +0000, Dr. David Alan Gilbert wrote:
> Hi,
>   I've been looking at reviving the migration-as-ber work that Michael and
> Stefan looked at a while ago, and have stuff starting to work, but not
> ready yet, so I thought I'd start by posting my current view
> of an ASN.1 schema.
> 
> Note this is my 1st attempt at ASN.1 so feel free to pick holes!
> 
> This schema successfully passes asn1c and libtasn1's asn1Parser.
> 
> I've currently got a visitor modified Qemu that generated apparently
> valid BER output (and compatible old-format) - unber and openssl asn1parse
> dump it apparently happily, but I've not been able to find a tool to verify
> it against the schema - suggestions welcome.
> (* libtasn1's asn1Decoder OOMs my laptop/and or spits out a DER error, not
> that I'm trying to encode as DER)
> 
> Some notes:
>   * I'm using the universal type codes for most things except the larger
>     scale structures/sequences where I've given them our own types.
>   * ASN type codes are disgustingly decimal, and even more disgustingly 
> encoded
>     in BER, so having things readable in a hex dump is difficult; although the
>     values I've used end up with the last byte as an appropriate ASCII letter
>     and the other bytes would be if the top bit wasn't set.
>   * The 'shape' of it corresponds pretty closely to the shape of the existing
>     migration format; although I intend to clean it up a bit 
>     - RAMSecEntry has the same flags that currently go into our binary format
>       but those can become more of an enum, and sima).
>     - I think SecFull/SecMin can probably merge together.
>   * Most things are just sequences, including most of VMState; I don't really
>     want to go around every VMState structure in the whole of QEMU allocating
>     a type ID, however I'm thinking about making it an optional VMState field
>     which I'd use if it was set.
>   * The only thing that worries me in terms of efficiency so far is the
>     'page is all 0' compression encoding which is rather verbose, for 
> everything
>     else it seems that although it's chatty it's not that much of an overhead.
>   * I've not looked at block migration or spapr yet, (the only other iterative
>     migrate users)
>   * anything with a .get/.put or using the old style migration registration
>     gets bundled up into a blob as an octet-string - this means I don't have
>     to fix everything before it can start working.
> 
> As soon as I get the corresponding BER input visitor working and sweep the
> sawdust out of my code I'll post it; probably a week or two.
> 
> Thoughts welcome,
> 
> Dave


I think this is already useful.
For example this will make input parser
more robust against unexpected input
as octet string blobs are bounded, so if we fail
to parse one we at least will not
interpret it as beginning of the next one.


I think it's a good idea to merge this gradually,
even if this means changing format several times:
as long as we can stay compatible with old machine
types.

> 
> --------------------------------------------------------------------------------
> 
> Qemu {}
> DEFINITIONS IMPLICIT TAGS ::=
> BEGIN
> 
> -- Some basic types used in multiple places --
> QemuString ::= UTF8String (SIZE (1..255))
> 
> -- TODO: 4096 is actually page size whatever that is
> FullPage ::= OCTET STRING (SIZE (4096))
> 
> RAMBlockID ::= SEQUENCE {
>   name   QemuString,
>   len    INTEGER

is len needed here?

> }
> 
> RAMSecEntry ::= [ APPLICATION 8914 ] SEQUENCE {
>   addr         INTEGER,   -- Address or offset or size

hmm so which one?

>   flags        INTEGER,   -- maybe more explicit type?
>   name         QemuString OPTIONAL,
> 
>   body         CHOICE {
>     bl         SEQUENCE OF RAMBlockID,
>     compr      INTEGER (0..255), -- Page filled with this value
>     page       FullPage
>     -- TODO xbzrle --

this one confused me.
I expected RAMBlockID followed by a sequence of pages

>   }
> }
> 
> RAMSecList ::= [ APPLICATION 9810 ] SEQUENCE OF RAMSecEntry
> 
> SubSection ::= [ APPLICATION 10707 ] SEQUENCE {
>   name         QemuString,
>   versionid    INTEGER,
> 
>   contents     SEQUENCE OF VMStateEntries
> }
> 
> SubSecList ::= [ APPLICATION 10700 ] SEQUENCE OF SubSection
> 
> VMStateEntries ::= CHOICE {
>   -- Hmm need to think more --

I'm guessing individual devices will have
tagged entries here?

>   dummy    INTEGER,

why is this here?

>   subsecl  SubSecList,
>   oldblob  OCTET STRING
> }
> 
> VMState ::= SEQUENCE OF VMStateEntries
> 
> -- Restrict to unsigned?
> SectionID ::= INTEGER
> 
> SecFull ::= [ APPLICATION 2003 ] SEQUENCE {
>   name         QemuString,
>   sectionid    SectionID,

what is this id exactly?

>   instanceid   INTEGER,
>   versionid    INTEGER,
> 
>   contents     CHOICE {
>     ramsec     RAMSecList,
>     -- TODO other iterator initial stuff --
>     vmstate    VMState
>   }
> }
> 
> SecMin ::= [ APPLICATION 211 ] SEQUENCE {
>   sectionid   SectionID,
> 
>   contents     CHOICE {
>     ramsec     SEQUENCE OF RAMSecEntry
>     -- TODO other iterator general/end stuff --
>   }
> }
> 
> Sections ::= CHOICE {
>   full    SecFull,
>   min     SecMin
> }
> 
> -- The whole file --
> -- Application tag used to get first 32bits of file
> -- to come out as 7f cd c5 51  - the 51 is Q
> -- the c5 and cd being E,M but with the top bit set
> -- which BER requires
> QemuFile ::= [ APPLICATION 1270481 ] SEQUENCE {
>   version INTEGER (3),
>   top     SEQUENCE OF Sections
> }
> 
> END
> 
> --------------------------------------------------------------------------------
> --
> Dr. David Alan Gilbert / address@hidden / Manchester, UK