Re: [Qemu-devel] How do we do user input bitmap properties?

[Dr. David Alan Gilbert]

* Andrew Jones (address@hidden) wrote:
> On Thu, Apr 18, 2019 at 03:03:02PM +0100, Dave Martin wrote:
> > On Thu, Apr 18, 2019 at 12:28:47PM +0100, Andrew Jones wrote:
> > > On Thu, Apr 18, 2019 at 11:52:04AM +0100, Dave Martin wrote:
> > > > On Thu, Apr 18, 2019 at 10:46:34AM +0100, Andrew Jones wrote:
> > > > > On Thu, Apr 18, 2019 at 11:28:41AM +0200, Andrew Jones wrote:
> > > > > > Hi all,
> > > > > > 
> > > > > > First some background:
> > > > > > 
> > > > > > For the userspace side of AArch64 guest SVE support we need to
> > > > > > expose KVM's allowed vector lengths bitmap to the user and allow
> > > > > > the user to choose a subset of that bitmap. Since bitmaps are a
> > > > > > bit awkward to work with then we'll likely want to expose it as
> > > > > > an array of vector lengths instead. Also, assuming we want to
> > > > > > expose the lengths as number-of-quadwords (quadword == 128 bits
> > > > > > for AArch64 and vector lengths must be multiples of quadwords)
> > > > > > rather than number-of-bits, then an example array (which will
> > > > > > always be a sequence) might be
> > > > > > 
> > > > > >  [ 8, 16, 32 ]
> > > > > > 
> > > > > > The user may choose a subsequence, but only through truncation,
> > > > > > i.e. [ 8, 32 ] is not valid, but [ 8, 16 ] is.
> > > > > > 
> > > > > > Furthermore, different hosts may support different sequences
> > > > > > which have the same maximum. For example, if the above sequence
> > > > > > is for Host_A, then Host_B could be
> > > > > > 
> > > > > >  [ 8, 16, 24, 32 ]
> > > > > 
> > > > > It doesn't really matter for this discussion, but I just realized
> > > > > that I picked bogus numbers for these examples. 32 would be too
> > > > > big. The largest supported is 16. I probably should have just used
> > > > > the simple [ 1, 2, 4 ] and [ 1, 2, 3, 4 ] arrays, corresponding to
> > > > > vector lengths (in bits) 128, 256, 512 and 128, 256, 384, 512.
> > > > 
> > > > I'd argue differently from the ABI perspective.
> > > > 
> > > > If the host supports vq = [ 1,2,3,4 ], then it is entirely valid to ask
> > > > for [ 1,2,4 ].  KVM will fail the KVM_REG_ARM64_SVE_VLS write with
> > > > EINVAL, but I don't distinguish between a set that is not satisfiable on
> > > > this hardware and a set that is not valid at all.  The kernel basically
> > > > doesn't check for the latter: an architecturally invalid set will always
> > > > not be satisfiable on the hardware.
> > > 
> > > I agree that 1,2,4 should be a valid set, but how do we avoid the user
> > > attempting to select it on a platform that supports 1,2,3,4 now? Until
> > > the hardware and KVM will allow it, then I don't think we should try
> > > to "support" it in any way. Maybe we should name the property such that
> > > it's clear we need to use truncation when constructing a subset now.
> > > This would allow us to add another, more generally named, property later
> > > when we can select nearly arbitrary subsets.
> > > 
> > > > 
> > > > Ideally, userspace should not preempt this decision, in case the
> > > > architecture becomes more flexible someday in what it can virtualise.
> > > > 
> > > > > > The host must support all lengths in the sequence, which means
> > > > > > that while Host_A supports 32, since it doesn't support 24 and
> > > > > > we can only truncate sequences, we must use either [ 8 ] or
> > > > > > [ 8, 16 ] for a compatible sequence if we intend to migrate
> > > > > > between the hosts.
> > > > > > 
> > > > > > Now to the $SUBJECT question:
> > > > > > 
> > > > > > My feeling is that we should require the sequence to be
> > > > > > provided on the command line as a cpu property. Something
> > > > > > like
> > > > > > 
> > > > > >   -cpu host,sve-vl-list=8:16
> > > > 
> > > > I can't decide whether it's more or less user-friendly to denote VL in
> > > > terms of bytes in this kind of context.
> > > 
> > > I'm not sure either. Switching to my second take at quadwords this should
> > > have been '1:2'. In bytes [ 1, 2 ] would be '16:32'.
> > > 
> > > > 
> > > > Usually when I say "VL" in the kernel ABI I try to mean this.
> > > > (KVM_REG_ARM64_SVE_VLS is a special case: here the lengths are fancily
> > > > encoded rather than being passed as integers, so the precise unit used
> > > > is a more abstract concept here ... or anyway, that's my excuse.
> > > > _SVE_VQS felt odd here.)
> > > 
> > > My thinking is that using the number of quadwords keeps the numbers
> > > smaller. But, if people are mostly going to want to think about vector
> > > lengths in terms of bits, like the specification does, then maybe we
> > > should just let the numbers be bigger: [ 128, 256, 512 ]
> > > 
> > > > 
> > > > For kvmtool I went with quadwords on the command line, but only as a
> > > > quick hack to simplify the parser slightly.
> > > > 
> > > > OTOH, specifying 1,2,4 on the command line is clearly less annoying and
> > > > error-prone than 16,32,64.
> > > > 
> > > > Naming the option with "vq" instead of "vl" is another option, though
> > > > "vq" is Linux terminology not endorsed by the architecture.
> > > 
> > > I think VL is OK as long we document the unit.
> > 
> > As you wish (and that's basically where I ended up for now with kvmtool).
> > 
> > > > 
> > > > For kvmtool I've considered a range / filtering syntax, but I didn't
> > > > implement it yet.
> > > 
> > > I think we want each VL to be explicitly listed on the command line
> > > for the compatibility issue described above. On one host the range
> > > [1,4] can be different than the range [1,4] on anther host, depending
> > > on whether '3' is in there. And syntax like 1-4:-3 would be overly
> > > complicated, IMO. So I'm inclined to just require all of them to be
> > > listed, but of course order wouldn't matter.
> > 
> > Probably reasonable.  For now (and quite likely forever) the length of
> > the list is at most 16 elements.  To avoid overengineering I stuck with
> > a flat comma-separated list for now.
> > 
> > 
> > It's worth nothing that there are two problems to be solved here: one is
> > to specify an exact set unambiguously, which is important for migration
> > scenarios.
> > 
> > The other is to be able to clamp the vector length for user convenience,
> > but without particularly considering migration.  There's no direct way
> > for the user to know the set of vector lengths supported by KVM today,
> > so cooking up the right magic to go on the command line is non-trivial:
> > you have to create a dummy vcpu and read KVM_REG_ARM64_SVE_VLS to find
> > the host's supported set.  Or you just have to know.
> 
> Right, we'll have to query first to know what's available. libvirt will
> do that and tools built on libvirt should help provide the user with
> sane defaults or a relatively painless selection process. The QEMU command
> line is primarily for developers, so they'll likely just know what they
> want.
> 
> > 
> > A separate max-vl option (or whatever) might be a useful alternative.
> 
> Yeah, we could offer this to be nicer to QEMU command line users that
> don't intend to migrate, or don't care if a migration can fail. This
> would be analogous to '-cpu max' which enables all available cpu
> features. Bad for migration, but a nice shorthand if you want your
> guest to have everything.

One that printed out what was available on that host might be nice
(and/or via a qmp query- command) for when you're trying to figure out
what's going.

Dave

> Thanks,
> drew
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK