Re: [PATCH v12 00/13] Support blob memory and venus on qemu

[Dmitry Osipenko]

On 5/22/24 12:00, Alex Bennée wrote:
> Dmitry Osipenko <dmitry.osipenko@collabora.com> writes:
> 
>> On 5/21/24 17:57, Alex Bennée wrote:
>>> Alex Bennée <alex.bennee@linaro.org> writes:
>>>
>>>> Dmitry Osipenko <dmitry.osipenko@collabora.com> writes:
>>>>
>>>>> Hello,
>>>>>
>>>>> This series enables Vulkan Venus context support on virtio-gpu.
>>>>>
>>>>> All virglrender and almost all Linux kernel prerequisite changes
>>>>> needed by Venus are already in upstream. For kernel there is a pending
>>>>> KVM patchset that fixes mapping of compound pages needed for DRM drivers
>>>>> using TTM [1], othewrwise hostmem blob mapping will fail with a KVM error
>>>>> from Qemu.
>>>>>
>>>>> [1] 
>>>>> https://lore.kernel.org/kvm/20240229025759.1187910-1-stevensd@google.com/
>>>>>
>>>>> You'll need to use recent Mesa version containing patch that removes
>>>>> dependency on cross-device feature from Venus that isn't supported by
>>>>> Qemu [2].
>>>>>
>>>>> [2] 
>>>>> https://gitlab.freedesktop.org/mesa/mesa/-/commit/087e9a96d13155e26987befae78b6ccbb7ae242b
>>>>>
>>>>> Example Qemu cmdline that enables Venus:
>>>>>
>>>>>   qemu-system-x86_64 -device 
>>>>> virtio-vga-gl,hostmem=4G,blob=true,venus=true \
>>>>>       -machine q35,accel=kvm,memory-backend=mem1 \
>>>>>       -object memory-backend-memfd,id=mem1,size=8G -m 8G
>>>>
>>>> What is the correct device for non-x86 guests? We have virtio-gpu-gl-pci
>>>> but when doing that I get:
>>>>
>>>>   -device virtio-gpu-gl-pci,hostmem=4G,blob=true,venus=true
>>>>   qemu-system-aarch64: -device 
>>>> virtio-gpu-gl-pci,hostmem=4G,blob=true,venus=true: opengl is not available
>>>>
>>>> According to 37f86af087 (virtio-gpu: move virgl realize + properties):
>>>>
>>>>   Drop the virgl property, the virtio-gpu-gl-device has virgl enabled no
>>>>   matter what.  Just use virtio-gpu-device instead if you don't want
>>>>   enable virgl and opengl.  This simplifies the logic and reduces the test
>>>>   matrix.
>>>>
>>>> but that's not a good solution because that needs virtio-mmio and there
>>>> are reasons to have a PCI device (for one thing no ambiguity about
>>>> discovery).
>>>
>>> Oops my mistake forgetting:
>>>
>>>   --display gtk,gl=on
>>>
>>> Although I do see a lot of eglMakeContext failures.
>>
>> Please post the full Qemu cmdline you're using
> 
> With:
> 
>   ./qemu-system-aarch64 \
>            -machine type=virt,virtualization=on,pflash0=rom,pflash1=efivars \
>            -cpu neoverse-n1 \
>            -smp 4 \
>            -accel tcg \
>            -device virtio-net-pci,netdev=unet \
>            -device virtio-scsi-pci \
>            -device scsi-hd,drive=hd \
>            -netdev user,id=unet,hostfwd=tcp::2222-:22 \
>            -blockdev 
> driver=raw,node-name=hd,file.driver=host_device,file.filename=/dev/zen-ssd2/trixie-arm64,discard=unmap
>  \
>            -serial mon:stdio \
>            -blockdev 
> node-name=rom,driver=file,filename=(pwd)/pc-bios/edk2-aarch64-code.fd,read-only=true
>  \
>            -blockdev 
> node-name=efivars,driver=file,filename=$HOME/images/qemu-arm64-efivars \
>            -m 8192 \
>            -object memory-backend-memfd,id=mem,size=8G,share=on \
>            -device virtio-gpu-gl-pci,hostmem=4G,blob=true,venus=true \
>            -display gtk,gl=on,show-cursor=on -vga none \
>            -device qemu-xhci -device usb-kbd -device usb-tablet
> 
> I get a boot up with a lot of:
> 
>                                                                               
>                                                                               
>                  
>   (qemu:1545322): Gdk-WARNING **: 09:26:09.470: eglMakeCurrent failed         
>                                                                               
>                    
> 
>   (qemu:1545322): Gdk-WARNING **: 09:26:09.470: eglMakeCurrent failed         
>                                                                               
>                    
> 
>   (qemu:1545322): Gdk-WARNING **: 09:26:09.470: eglMakeCurrent failed         
>                                                                               
>                    
> 
>   (qemu:1545322): Gdk-WARNING **: 09:26:09.470: eglMakeCurrent failed         
>       
> 
> In the guest I run:
> 
>   meson devenv -C /root/lsrc/graphics/mesa.git/build fish
> 
> to bring in the latest Mesa (with virtio enabled). Running vulkaninfo
> reports two cards:
> 
>   ==========                                                                  
>           
>   VULKANINFO                        
>   ==========                                                                  
>           
> 
>   Vulkan Instance Version: 1.3.280                                            
>           
> 
> 
>   Instance Extensions: count = 14
>   -------------------------------
>   VK_EXT_debug_report                    : extension revision 10
>   VK_EXT_debug_utils                     : extension revision 2
>   VK_EXT_headless_surface                : extension revision 1
>   VK_KHR_device_group_creation           : extension revision 1
>   VK_KHR_external_fence_capabilities     : extension revision 1
>   VK_KHR_external_memory_capabilities    : extension revision 1
>   VK_KHR_external_semaphore_capabilities : extension revision 1
>   VK_KHR_get_physical_device_properties2 : extension revision 2
>   VK_KHR_get_surface_capabilities2       : extension revision 1
>   VK_KHR_portability_enumeration         : extension revision 1
>   VK_KHR_surface                         : extension revision 25
>   VK_KHR_surface_protected_capabilities  : extension revision 1
>   VK_KHR_wayland_surface                 : extension revision 6
>   VK_LUNARG_direct_driver_loading        : extension revision 1
> 
>   Instance Layers: count = 2
>   --------------------------
>   VK_LAYER_MESA_device_select Linux device selection layer 1.3.211  version 1
>   VK_LAYER_MESA_overlay       Mesa Overlay layer           1.3.211
>   version 1
> 
>   Devices:
>   ========
>   GPU0:
>           apiVersion         = 1.3.230
>           driverVersion      = 24.1.99
>           vendorID           = 0x8086
>           deviceID           = 0xa780
>           deviceType         = PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU
>           deviceName         = Virtio-GPU Venus (Intel(R) Graphics (RPL-S))
>           driverID           = DRIVER_ID_MESA_VENUS
>           driverName         = venus
>           driverInfo         = Mesa 24.2.0-devel (git-0b582449f0)
>           conformanceVersion = 1.3.0.0
>           deviceUUID         = 29d2e940-a1a0-3054-0f9a-9f7dec52a084
>           driverUUID         = 3694c390-f245-612a-12ce-7d3a99127622
>   GPU1:
>           apiVersion         = 1.2.0
>           driverVersion      = 24.1.99
>           vendorID           = 0x10005
>           deviceID           = 0x0000
>           deviceType         = PHYSICAL_DEVICE_TYPE_CPU
>           deviceName         = Virtio-GPU Venus (llvmpipe (LLVM 15.0.6, 256 
> bits))
>           driverID           = DRIVER_ID_MESA_VENUS
>           driverName         = venus
>           driverInfo         = Mesa 24.2.0-devel (git-0b582449f0)
>           conformanceVersion = 1.3.0.0
>           deviceUUID         = 5fb5c03f-c537-f0fe-a7e6-9cd5866acb8d
>           driverUUID         = 3694c390-f245-612a-12ce-7d3a99127622
> 
> Running weston and then vkcube-wayland reports its selecting "GPU 0:
> Virtio-GPU Venus (Intel(R) Graphics (RPL-S))" but otherwise produces no
> output.
> 
> If I run with "-display sdl,gl=on,show-cursor=on" and the same other
> command line options the results for vulkaninfo are the same. However
> vkcube-wayland gets a little further and draws the initial cube on the
> screen before locking up with:
> 
>  MESA-VIRTIO: debug: stuck in fence wait with iter at xxxx
> 
> where xxxx grows each time it prints. On shutting down I see some virgl
> errors interspersed with the systemd logs:
> 
>   [drm:virtio_gpu_dequeue_ctrl_func [virtio_gpu]] *ERROR* response 0x1200 
> (command 0x209)
>   [  OK  ] Stopped systemd-logind.service - User Login Management.
>   virtio_gpu_virgl_process_cmd: ctrl 0x209, error 0x1200
>   [  475.257111] [drm:virtio_gpu_dequeue_ctrl_func [virtio_gpu]] *ERROR* 
> response 0x1200 (command 0x209)
>   [drm:virtio_gpu_dequeue_ctrl_func [virtio_gpu]] *ERROR* response 0x1200 
> (command 0x209)
>   [  OK  ] Stopped target network-online.target - Network is Online.
>   [  OK  ] Stopped target remote-fs.target - Remote File Systems.
>   [  OK  ] Stopped NetworkManager-wait-online…vice - Network Manager Wait 
> Online.
>            Stopping avahi-daemon.service - Avahi mDNS/DNS-SD Stack...
>            Stopping cups.service - CUPS Scheduler...
>            Stopping user-runtime-dir@0.servic…er Runtime Directory 
> /run/user/0...
>   [  OK  ] Stopped avahi-daemon.service - Avahi mDNS/DNS-SD Stack.
>   [  OK  ] Stopped cups.service - CUPS Scheduler.
>   virtio_gpu_virgl_process_cmd: ctrl 0x209, error 0x1200
>   [  475.357543] [drm:virtio_gpu_dequeue_ctrl_func [virtio_gpu]] *ERROR* 
> response 0x1200 (command 0x209)
>   [drm:virtio_gpu_dequeue_ctrl_func [virtio_gpu]] *ERROR* response 0x1200 
> (command 0x209)
>   [  OK  ] Stopped target network.target - Network.
>   [  OK  ] Stopped target nss-user-lookup.target - User and Group Name 
> Lookups.
>            Stopping NetworkManager.service - Network Manager...
>            Stopping networking.service - Raise network interfaces...
>            Stopping wpa_supplicant.service - WPA supplicant...
>   [  OK  ] Stopped wpa_supplicant.service - WPA supplicant.
>   virtio_gpu_virgl_process_cmd: ctrl 0x209, error 0x1200
>   [  493.585261] [drm:virtio_gpu_dequeue_ctrl_func [virtio_gpu]] *ERROR* 
> response 0x1200 (command 0x209)
>   [drm:virtio_gpu_dequeue_ctrl_func [virtio_gpu]] *ERROR* response 0x1200 
> (command 0x209)
> 

I've reproduced this with qemu-system-aarch64. Vkcube works for a second
and then stops, Qemu compeltely gets frozen after closing and re-running
vkcube. Doesn't feel like this is a problem with venus, but with arm64.
For now don't know where is the bug, will take a closer look.

-- 
Best regards,
Dmitry