Re: [RFC 0/9] Add an interVM memory sharing device

[Igor Kotrasiński]

On 10.02.2020 14:01, Igor Kotrasiński wrote:
> On 2/7/20 5:33 PM, Stefan Hajnoczi wrote:
>> On Fri, Feb 07, 2020 at 11:04:03AM +0100, Igor Mammedov wrote:
>>> On Fri, 7 Feb 2020 10:00:50 +0100
>>> Igor Kotrasiński <address@hidden> wrote:
>>>
>>>> On 2/5/20 3:49 PM, Jan Kiszka wrote:
>>>>> On 05.02.20 15:39, Stefan Hajnoczi wrote:
>>>>>> On Tue, Feb 04, 2020 at 12:30:42PM +0100,
>>>>>> address@hidden wrote:
>>>>>>> From: Igor Kotrasinski <address@hidden>
>>>>>>>
>>>>>>> This patchset adds a "memory exposing" device that allows two QEMU
>>>>>>> instances to share arbitrary memory regions. Unlike ivshmem, it 
>>>>>>> does not
>>>>>>> create a new region of memory that's shared between VMs, but instead
>>>>>>> allows one VM to access any memory region of the other VM we 
>>>>>>> choose to
>>>>>>> share.
>>>>>>>
>>>>>>> The motivation for this device is a sort of ARM Trustzone 
>>>>>>> "emulation",
>>>>>>> where a rich system running on one machine (e.g. x86_64 linux) is 
>>>>>>> able
>>>>>>> to perform SMCs to a trusted system running on another (e.g. 
>>>>>>> OpTEE on
>>>>>>> ARM). With a device that allows sharing arbitrary memory between 
>>>>>>> VMs,
>>>>>>> this can be achieved with minimal changes to the trusted system 
>>>>>>> and its
>>>>>>> linux driver while allowing the rich system to run on a speedier x86
>>>>>>> emulator. I prepared additional patches for linux, OpTEE OS and 
>>>>>>> OpTEE
>>>>>>> build system as a PoC that such emulation works and passes OpTEE 
>>>>>>> tests;
>>>>>>> I'm not sure what would be the best way to share them.
>>>>>>>
>>>>>>> This patchset is my first foray into QEMU source code and I'm 
>>>>>>> certain
>>>>>>> it's not yet ready to be merged in. I'm not sure whether memory 
>>>>>>> sharing
>>>>>>> code has any race conditions or breaks rules of working with memory
>>>>>>> regions, or if having VMs communicate synchronously via chardevs 
>>>>>>> is the
>>>>>>> right way to do it. I do believe the basic idea for sharing memory
>>>>>>> regions is sound and that it could be useful for inter-VM 
>>>>>>> communication.
>>>>>>
>>>>>> Hi,
>>>>>> Without having looked into the patches yet, I'm already wondering 
>>>>>> if you
>>>>>> can use the existing -object
>>>>>> memory-backend-file,size=512M,mem-path=/my/shared/mem feature for 
>>>>>> your
>>>>>> use case?
>>>>>>
>>>>>> That's the existing mechanism for fully sharing guest RAM and if you
>>>>>> want to share all of memory then maybe a device is not necessary - 
>>>>>> just
>>>>>> share the memory.
>>>>
>>>> That option adds memory in addition to the memory allocated with the
>>>> '-m' flag, doesn't it? I looked into that option, and it seemed to me
>>>> you can't back all memory this way.
>>> with current QEMU you play with memory sharing using numa workaround
>>>
>>> -m 512 \
>>> -object memory-backend-file,id=mem,size=512M,mem-path=/my/shared/mem 
>>> feature,share=on \
>>> -numa node,memdev=mem
>>>
>>> also on the list there is series that allows to share main ram
>>> without numa workaround, see
>>>    "[PATCH v4 00/80] refactor main RAM allocation to use hostmem 
>>> backend"
>>>
>>> with it applied you can share main RAM with following CLI:
>>>
>>> -object memory-backend-file,id=mem,size=512M,mem-path=/my/shared/mem 
>>> feature,share=on \
>>> -m 512 \
>>> -M virt,memory-backend=mem
>>
>> Nice!  That takes care of memory.
> 
> After a bit of hacking to map the shared RAM instead of communicating 
> via socket I can confirm - I can run OpTEE this way, and it passes 
> tests. My solution is *technically* more accurate since it is aware of 
> memory subregions and completely independent from memory backend setup, 
> but with my use case satisfied already, I don't think it's of use to 
> anyone.
> 

After a long while hacking QEMU to achieve 1-to-1 memory mapping between 
machines, I realized that I wasn't completely right here. I can share 
main memory from both machines, but for the arm machine that's only 
non-secure memory. Secure memory (VIRT_SECURE_MEM) is always allocated 
with memory_region_init_ram(), I don't know if other secure memory 
regions (e.g. VIRT_FLASH) might need to be shared as well.

This can probably be solved by allowing these regions to use file-backed 
memory when configured to do so, then mapping these files in the other 
machine at correct offsets.

Another solution would be memory sharing in this patchset. If we strip 
away interrupts, PCI stuff and io memory region support, it amounts to 
basically the same thing - a mechanism for accessing shareable memory in 
the other machine in a discoverable way.

>>
>> If signalling (e.g. a notification interrupt) is necessary then a
>> mechanism is still needed for that.  I don't know enough about TrustZone
>> to suggest an appropriate way of doing it with existing QEMU features.
>> Maybe Peter understands?
>>
> 
> Any signalling mechanism that can pass data along with it (e.g. ivshmem 
> with its shared memory) will suffice.

Igor