Re: [Qemu-devel] KVM and variable-endianness guest CPUs

[Victor Kamensky]

On 23 January 2014 02:23, Peter Maydell <address@hidden> wrote:
> On 23 January 2014 00:22, Victor Kamensky <address@hidden> wrote:
>> Peter, could I please ask you a favor. Could you please
>> stop deleting pieces of your and my previous responses
>> when you reply.
>
> No, sorry. It produces excessively long and totally unreadable
> emails for everybody else if people don't trim for context.
> This is standard mailing list practice.

Usually it is OK, but with your choices sometimes you remove
my questions without answering them, where I think that they
are essential for discussion. For example I asked about 'len'
with value that are not power of 2:

In [1] I wrote

"I don't see why you so attached to desire to describe
data part of memory transaction as just one of int
types. If we are talking about bunch of hypothetical
cases imagine such bus that allow transaction with
size of 6 bytes. How do you describe such data in
your ints speak? What endianity you can assign to
sequence of 6 bytes? While note that description of
such transaction as set of 6 byte values at address
$whatever makes perfect sense."

But notice that in your next reply [2] you just dropped it

Similar situation happens with this reply you removed
piece that I had to bring back. Please see below.

>>>> Consider above big endian case (setend be) example,
>>>> but now running in BE KVM host. 0x4 is LSB of CPU
>>>> core register in this case.
>>>
>>> Yes. In this case if we are using the "mmio.data is host
>>> kernel endianness" definition then mmio.data[0] should be
>>> 0x01 (the MSB of the 32 bit data value).
>>
>> If mmio.data[0] is 0x1, mmio.data[] = {0x1, 0x2, 0x3, 0x4},
>> and now KVM host and emulator running in BE mode.
>> But that contradicts to what you said before.
>
> Sorry, I misread the example here (and assumed we were
> writing the same word in both cases, when actually the BE
> code example is writing a different value). mmio.data[0] should
> be 0x4, because:
>  * BE ARM guest, so KVM must byte-swap the register value
>     (giving 0x04030201)
>  * BE host, so it writes the uint32_t in host order (giving
>    0x4 in mmio.data[0])
>
>>>> I believe, but I need to check, that PPC BE setup actually
>>>> acts as the second case in above example  If we have PPC
>>>> BE guest executing the following instructions:
>>>>
>>>> lis     r1,0x102
>>>> ori     r1,r1,0x304
>>>> stw    r1,0(r0)
>>>>
>>>> after first two instructions r1 would contain 0x01020304.
>>>> IMHO It exactly corresponds to above my ARM second case -
>>>> BE guest when it runs under ARM BE KVM host. I believe
>>>> that mmio.data[] in PPC BE case would be {0x1, 0x2, 0x3, 0x4}.
>>>
>>> Yes, assuming a BE PPC host kernel (which is the usual
>>> arrangement).
>>
>> OK, that confirms my understanding how PPC mmio
>> should work.
>>
>>>> But according to you data[0] must be 0x4 in BE host case
>>>
>>> Er, no. The data here is 0x01020304, so for a BE host
>>> data[0] is the big end, ie 0x1. It would only be 0x4 if
>>> mmio.data[] were LE always (or if you were running
>>> your BE PPC guest on an LE PPC host, which I don't
>>> think is supported currently).
>>
>> So do you agree that for all three code snippets cited in this
>> email, we always will have mmio.data[] = {0x1, 0x2,
>> 0x3, 0x4}, for ARM LE qemu/host, for ARM BE qemu/host
>> and for ppc code snippet in PPC BE qemu/host.
>
> No. Also your ARM and PPC examples are not usefully
> comparable, because:
>
>> setend le
>> mov r1, #0x04030201
>> str r1, [r0]
>
> This is an LE guest writing 0x04030201, and that is the
> value that will go out on the bus.
>
>> and
>>
>> setend be
>> mov r1, #0x01020304
>> str r1, [r0]
>
> This is a BE guest writing 0x01020304; as far as the
> code running on the CPU is concerned; the value on the
> bus will be byteswapped.
>
>> lis     r1,0x102
>> ori     r1,r1,0x304
>> stw    r1,0(r0)
>
> This is also a BE guest writing 0x01020304. I'm pretty
> sure that the PPC approach is that for BE guests writing
> a word that word goes out to the bus as is; for LE guests
> (or if the page table is set up to say "this page is LE") the
> CPU swaps it before putting it on the bus. In this regard
> it is the opposite way round to ARM.
>
> So the value you start with in the CPU register is not
> the same in all three cases, and what the hardware
> does is not the same either.

So in what cases h/w does differently? I think we agreed
before that in ARM cases it is the same memory
transaction h/w cannot do anything different. And ARM
'setend be' cases matches PPC BE case in both cases
BE write happens to the same h/w address and value is
0x01020304, why h/w would see it differently? It is
the same write.

I think you missing that in all discussed cases BE-8, byte
invariant CPU memory buses are used. Here is what I wrote
in reply to Alex, it is worth copying it here:

---- start quote from my response to Alex -----
I disagree with Peter's point of view as you saw from our
long thread :). I strongly believe that current mmio.data[]
describes data on the bus perfectly fine with array of bytes.
data[0] goes into phys_addr, data[1] goes into phys_addr + 1,
etc.

Please check "Differences between BE-32 and BE-8 buses"
section in [3]. In modern ARM CPU memory bus is byte invariant (BE-8).
As data lines bytes view concerns, it is the same between LE and
BE-8 that is why IMHO array of bytes view is very good choice.
PPC and MIPS CPUs memory buses are also byte invariant, they
always been that way. I don't think we care about BE-32. So
for all practical purposes, mmio structure is BE-8 bus emulation,
where data signals could be defined by array of bytes. If one
would try to define it as set of other bigger integers
one need to have endianness attribute associated with it. If
such attribute implied by default just through CPU type in order to
work with existing cases it should be different for different CPU
types, which means qemu running in the same endianity but
on different CPU types should acts differently if it emulates
the same device and that is bad IMHO. So I don't see any
value from departing from bytes array view of data on the bus."
---- end quote from my response to Alex -----

We really need to agree that in all three cases that
if the same device attached to memory bus at r0 address,
device sees the same data write. That is really the key.  If you
disagree please stop reading and let's discuss that. If we have
disagreement on this, I will find examples
in Linux kernel with code snippets from the same driver
writing to the same h/w register between considered 3
cases ARM LE, ARM BE, PPC BE. Or if it won't be
the same driver I will find logically very similar cases.

If now you agree that it is the same data seen by h/w
in all three cases let's go back to mmio.data[] content

So to summarize as far as mmio.data[[ concerned
for this three cases according to you:

For LE KVM host/qemu:
ARM 'setend le' case mmio.data[] = {0x1, 0x2, 0x3, 0x4}
ARM 'setend be' case mmio.data[] = {0x1, 0x2, 0x3, 0x4}

For BE KVM host/qemu:
ARM 'setend le' case mmio.data[] = {0x4, 0x3, 0x2, 0x1}
ARM 'setend be' case mmio.data[] = {0x4, 0x3, 0x2, 0x1}
PPC case mmio.data[] = {0x1, 0x2, 0x3, 0x4}

And here is what I asked wrt this in the end of [4]:

> So do you agree that for all three code snippets cited in this
> email, we always will have mmio.data[] = {0x1, 0x2,
> 0x3, 0x4}, for ARM LE qemu/host, for ARM BE qemu/host
> and for ppc code snippet in PPC BE qemu/host.
> I believe it should be this way, because from emulator (i.e
> qemu) code point of view running on ARM BE qemu/host
> or PPC BE and emulating the same h/w device, code
> should not make difference whether it is ARM or PPC.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

just to highlight it again, as far as BE qemu concerned it sees
{0x4, 0x3, 0x2, 0x1} in ARM case and {0x1, 0x2, 0x3, 0x4}
in PPC case but it is the same value h/w write, why qemu
should process it differently depending on CPU type? I
believe it should not be the case and content of mmio.data[]
should be {0x1, 0x2, 0x3, 0x4} which is unambiguous BE-8
description of bus data signals for all possible endianness
cases of KVM host/qemu.

Thanks,
Victor

[1] https://lists.cs.columbia.edu/pipermail/kvmarm/2014-January/008902.html

[2] https://lists.cs.columbia.edu/pipermail/kvmarm/2014-January/008903.html

[3] 
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0290g/ch06s05s01.html

[4] https://lists.cs.columbia.edu/pipermail/kvmarm/2014-January/008906.html

> thanks
> -- PMM