Re: [Qemu-devel] WIP: Migration format: ASN.1/BER schema

[Michael S. Tsirkin]

On Wed, Mar 12, 2014 at 01:17:15PM +0000, Dr. David Alan Gilbert wrote:
> * Michael S. Tsirkin (address@hidden) wrote:
> > On Fri, Mar 07, 2014 at 04:55:03PM +0000, Dr. David Alan Gilbert wrote:
> 
> <snip>
> 
> > I think this is already useful.
> > For example this will make input parser
> > more robust against unexpected input
> > as octet string blobs are bounded, so if we fail
> > to parse one we at least will not
> > interpret it as beginning of the next one.
> 
> I've also managed (with a slightly improved schema
> at the end) to validate my output sequence against the schema
> using asn1c (once I'd persuaded it to build stuff and turn
> it's debug on).
> 
> > I think it's a good idea to merge this gradually,
> > even if this means changing format several times:
> > as long as we can stay compatible with old machine
> > types.
> 
> Yes, that would make the merge easier.
> 
> > > --------------------------------------------------------------------------------
> > > 
> > > Qemu {}
> > > DEFINITIONS IMPLICIT TAGS ::=
> > > BEGIN
> > > 
> > > -- Some basic types used in multiple places --
> > > QemuString ::= UTF8String (SIZE (1..255))
> > > 
> > > -- TODO: 4096 is actually page size whatever that is
> > > FullPage ::= OCTET STRING (SIZE (4096))
> > > 
> > > RAMBlockID ::= SEQUENCE {
> > >   name   QemuString,
> > >   len    INTEGER
> > 
> > is len needed here?
> > 
> > > }
> > > 
> > > RAMSecEntry ::= [ APPLICATION 8914 ] SEQUENCE {
> > >   addr         INTEGER,   -- Address or offset or size
> > 
> > hmm so which one?
> > 
> > >   flags        INTEGER,   -- maybe more explicit type?
> > >   name         QemuString OPTIONAL,
> > > 
> > >   body         CHOICE {
> > >     bl         SEQUENCE OF RAMBlockID,
> > >     compr      INTEGER (0..255), -- Page filled with this value
> > >     page       FullPage
> > >     -- TODO xbzrle --
> > 
> > this one confused me.
> > I expected RAMBlockID followed by a sequence of pages
> 
> I've kept with the structure of the way the migrate works at the moment;
> it transmits the 'shape' of RAM as a list of name/length pairs, and only
> once it's validated that it has all the RAM blocks and their lengths match
> does it start shifting pages across.
> This a RAM section is either the initialisation data (the sequence of 
> RAMBlockID's)
> or a stream of pages.
> I don't think it's a bad idea to have that RAM list at the beginning,

okay so why isn't the schema saying this?

RAMSecEntry ::= [ APPLICATION 8914 ] SEQUENCE {
  addr         INTEGER,   -- Address or offset or size
  flags        INTEGER,   -- maybe more explicit type?
  name         QemuString OPTIONAL,

  body         CHOICE {
    bl         SEQUENCE OF RAMBlockID,
    compr      INTEGER (0..255), -- Page filled with this value
    page       FullPage
    -- TODO xbzrle --
  }
}


What you say seems to match what I expected:

    SEQUENCE OF RAMBlockID
  body         CHOICE {
    compr      INTEGER (0..255), -- Page filled with this value
    page       FullPage
    -- TODO xbzrle --
  }


> it catches a lot of silly mismatches - if you've ever hit a complaint
> about a mismatched BIOS length or a missing ROM, it's probably come
> from the check of this incoming list.

well it might become a problem with memory hotplug but it's a
separate question.


> 
> The existing binary format is a stream of things separated by a
> 64bit int that is an address/offset/or size depending on some bits OR'd
> into the bottom of the address; I'd currently split the flags out
> of that 64bit int, but hadn't split that further yet.
> I'm thinking of splitting this down further, especially so I can
> reduce the size of 'all 0' pages.

if flag just tell us the type then just make it a tag.

> > > 
> > > RAMSecList ::= [ APPLICATION 9810 ] SEQUENCE OF RAMSecEntry
> > > 
> > > SubSection ::= [ APPLICATION 10707 ] SEQUENCE {
> > >   name         QemuString,
> > >   versionid    INTEGER,
> > > 
> > >   contents     SEQUENCE OF VMStateEntries
> > > }
> > > 
> > > SubSecList ::= [ APPLICATION 10700 ] SEQUENCE OF SubSection
> > > 
> > > VMStateEntries ::= CHOICE {
> > >   -- Hmm need to think more --
> > 
> > I'm guessing individual devices will have
> > tagged entries here?
> 
> I'm thinking of *allowing* individual devices to have
> tagged entries here,

e.g. context-specific ones? might be good for
sub-sections.

> but I don't think I'd require it  - since
> to require it would mean I'd have to change every existing
> VMState to give it a tag.
> So I was going to allow VMStateEntries to be a generic set
> using universal tags, which we could add specific entries
> where we'd given common devices full tags.
> > >   dummy    INTEGER,
> > 
> > why is this here?
> 
> It's one of the universal entries; my current full version (that
> verifies) is:
> 
> VMStateEntries ::= CHOICE {
>   -- Hmm need to think more --
>   array       SEQUENCE OF VMStateEntries,
>   bool        BOOLEAN,
>   int         INTEGER,
>   oldblob     OCTET STRING,
>   subsecl     SubSecList
> }

Ah I see. It does not do much though.
Looks like you really can put ANY here just as well.

If we want to make it possible to specify devices
fully in the schema, we'll need a per-device tag.


> > 
> > >   subsecl  SubSecList,
> > >   oldblob  OCTET STRING
> > > }
> > > 
> > > VMState ::= SEQUENCE OF VMStateEntries
> > > 
> > > -- Restrict to unsigned?
> > > SectionID ::= INTEGER
> > > 
> > > SecFull ::= [ APPLICATION 2003 ] SEQUENCE {
> > >   name         QemuString,
> > >   sectionid    SectionID,
> > 
> > what is this id exactly?
> 
> It's again a hold over from the old format; Sections
> come in two varieties 'Full' and 'Min'; 'Full'
> sections are typically used for devices that just get
> sent once, and thus have a name - the ID doesn't
> really do much for them; however devices that send
> data iteratively send a Full followed by a series of
> Min's - each Min just has the SectionID matching the Full,
> so it saves it having to send the name each time,
> and it also means that it just has an index rather
> than having to string compare the name for every
> section.
> 
> Note that iterative devices can be interleaved so
> you might have some RAM, followed by some disk blocks
> and then some more RAM etc, which is why you can't
> just have a sequence of pages.

Hmm looks like a premature optimization?
Also, shouldn't name and ID be optional then?

> Dave
> P.S. Here's the current verison of my schema which
> validates against my ber data.
> 
> Qemu {}
> DEFINITIONS IMPLICIT TAGS ::=
> BEGIN
> 
> -- Some basic types used in multiple places --
> QemuString ::= UTF8String (SIZE (1..255))
> 
> -- TODO: 4096 is actually page size whatever that is
> FullPage ::= OCTET STRING (SIZE (4096))
> 
> RAMBlockID ::= SEQUENCE {
>   name   QemuString,
>   len    INTEGER
> }
> 
> RAMSecEntry ::= [ APPLICATION 8914 ] SEQUENCE {
>   addr         INTEGER,   -- Address or offset or size
>   flags        INTEGER,   -- maybe more explicit type?
>   name         QemuString OPTIONAL,
> 
>   body         CHOICE {
>     bl         SEQUENCE OF RAMBlockID,
>     compr      INTEGER (0..255), -- Page filled with this value
>     page       FullPage
>     -- TODO xbzrle --
>   }
> }
> 
> RAMSecList ::= [ APPLICATION 9810 ] SEQUENCE OF RAMSecEntry
> 
> SubSection ::= [ APPLICATION 10707 ] SEQUENCE {
>   name         QemuString,
>   versionid    INTEGER,
> 
>   contents     SEQUENCE OF VMStateEntries
> }
> 
> SubSecList ::= [ APPLICATION 10700 ] SEQUENCE OF SubSection
> 
> VMStateEntries ::= CHOICE {
>   -- Hmm need to think more --
>   array       SEQUENCE OF VMStateEntries,
>   bool        BOOLEAN,
>   int         INTEGER,
>   oldblob     OCTET STRING,
>   subsecl     SubSecList
> }
> 
> VMState ::= SEQUENCE OF VMStateEntries
> 
> -- Restrict to unsigned?
> SectionID ::= INTEGER
> 
> SecFull ::= [ APPLICATION 2003 ] SEQUENCE {
>   name         QemuString,
>   sectionid    SectionID,
>   instanceid   INTEGER,
>   versionid    INTEGER,
> 
>   contents     CHOICE {
>     ramsec     RAMSecList,
>     -- TODO other iterator initial stuff --
>     vmstate    VMState,
>     oldblob  OCTET STRING
>   }
> }
> 
> SecMin ::= [ APPLICATION 211 ] SEQUENCE {
>   sectionid   SectionID,
> 
>   contents     CHOICE {
>     ramsec     RAMSecList
>     -- TODO other iterator general/end stuff --
>   }
> }
> 
> Sections ::= CHOICE {
>   full    SecFull,
>   min     SecMin
> }
> 
> -- The whole file --
> -- Application tag used to get first 32bits of file
> -- to come out as 7f cd c5 51  - the 51 is Q
> -- the c5 and cd being E,M but with the top bit set
> -- which BER requires
> QemuFile ::= [ APPLICATION 1270481 ] SEQUENCE {
>   version INTEGER (3),
>   top     SEQUENCE OF Sections
> }
> 
> END
> --
> Dr. David Alan Gilbert / address@hidden / Manchester, UK