Re: [Qemu-devel] [RFC] TCG unit testing

[Aurelien Jarno]

On Fri, Aug 23, 2013 at 12:47:43PM -0700, Richard Henderson wrote:
> I've been thinking for a while about how to reliably test TCG backends, and
> maybe how to do regression testing on them.  Having to begin the test from a
> guest binary, especially considering the vast cross-compilation problem, is
> pretty much a non-starter.

Given the regular breakages we have with TCG, it's really a gread idea.
Usually most of the targets are working correctly so the problem is not
found immediately.

> I've been thinking of a truly stripped down target for the purpose, with a
> special-purpose machine loop and main to go with it.  I.e. avoid vl.c.

I think we should use as much as possible of the current infrastructure,
having two different main loops means they are going to be
desynchronised at some point.

Ideally it would be nice to have a machine called TCG (maybe with a few
variants for big/little endian, 32 and 64-bits) which loads TCG "code"
from a text file and execute it. Then we can imagine zillions of small
testcases like the GCC ones, and more can be added when a problem is
discovered and fixed.

> My current idea is that the test file consists of 3 sections: guest memory
> layout, raw TBs, and expected results.  Perhaps best explained with some 
> examples:
> 
> (1a) I've split up this test into two TBs to prevent some constant folding.
> (1b) The guest machine should have enough registers to make it easy to perform
> lots of tests at once.  Even better if we can avoid the complication of 
> memory.
> 
> --------------------------------------------
> # Test basic arithmetic, esp with constant arguments
> CODE
> 0:    movi_i32        a0, 0x12345678
>       movi_i32        a1, 0x87654321
>       movi_tl         pc, 1
>       exit_tb         0
> 
> 1:    add_i32         a2, a1, a0
>       sub_i32         a1, a1, a0
>       movi_i32        t0, 1
>       add_i32         a3, a0, t0
>       sub_i32         a4, a0, t0
>       movi_i32        t0, -1
>       add_i32         a5, a0, t0
>       sub_i32         a6, a0, t0
>       movi_i32        t0, -0x5678
>       add_i32         a7, a0, t0
>       sub_i32         a8, a0, t0
>       movi_i32        t0, 0x123456
>       add_i32         a9, a0, t0
>       sub_i32         a10, a0, t0
>       movi_i32        t0, 0x12341234
>       add_i32         a11, a0, t0
>       sub_i32         a12, a0, t0
>       call            done
> 
> RESULT
>       a0      0x12345678
>       a1      0x7530eca9
>       a2      0x99999999
>       a3      0x12345679
>       a4      0x12345677
>       a5      0x12345677
>       a6      0x12345679
>       a7      0x12340000
>       a8      0x1234acf0
>       a9      0x12468ace
>       a10     0x12222222
>       a11     0x246868ac
>       a12     0x4444
> -------------------------------------------
> 
> (2) The ram layout should be flexible enough to test specific scenarios
> 
> -------------------------------------------
> # For 64-bit guest, 32-bit host, make sure we compare high-part of tlb
> MEM
>       ram     0x0_0000_0000, 4096
>       ram     0x1_0000_0000, 4096

Maybe we can provide a few helper in the simple TCG machine to create
and remove TLB entries, or to memset the memory. That way each code that
is run can use a different memory mapping.

> CODE
> 0:    movi_i64        b0, 0x12345678abcdef
>       movi_i64        t1, 0
>       movi_i64        t2, 0x1_0000_0000
>       qemu_st32       b0, t1
>       qemu_ld32       b1, t1
>       qemu_ld32       b2, t2
>       call            done
> 
> RESULT
>       b0      0x12345678abcdef
>       b1      0x12345678abcdef
>       b2      0
> -------------------------------------------
> 
> (3a) If we let "ioram" be defined via memory_region_init_io, with the "/l" and
> "/b" qualifiers controlling the .endianness field of the region, then we can
> test the memory access path all the way through.
> (3b) The final test in the result section would *not* use the device interface
> from memory.h, but rather examine the backing ram directly.
> 
> -------------------------------------------
> MEM
>         ioram/l 0, 4096                         # little-endian device
>       ioram/b 4096, 4096                      # big-endian device
> CODE
> 0:      movi_tl         t0, 0x12345678
>         movi_tl         t1, 0
>         qemu_st32       t0, t1
>       qemu_ld32       a0, t1
>       movi_tl         t1, 4096
>       qemu_st32       t0, t1
>       qemu_ld32       a1, t1
>         call            done
> RESULT
>       a0      0x12345678
>       a1      0x12345678
>         mem/b   0, 0x78, 0x56, 0x34, 0x12       # little-endian bytes
>       mem/b   4096, 0x12, 0x34, 0x56, 0x78    # big-endian bytes
> -------------------------------------------
> 
> Thoughts?  Anything that we ought to be testing that I haven't thought of 
> here,
> and that this sort of structure couldn't support?

As said above this kind of structure might be difficult to maintain
while we increase it with a lot more tests.

I think the call to helpers, with variable number of arguments and
types, should be tested. Probably also access to the host memory through
the non-qemu ld/st functions. And maybe the condition ops, especially
with some corner cases with signed/unsigned comparisons.

The problem I also see is the testing of optional ops.

Also what about user mode? At least the memory access is different.

> It may be some time before I can progress this enough to usefulness, but I
> wanted to get this written down while it is fresh in my head.

That would be really great if it can be implemented at some point.
Thanks for working on that.

-- 
Aurelien Jarno                          GPG: 1024D/F1BCDB73
address@hidden                 http://www.aurel32.net