Re: Issue with VDUSE (QSD vduse-blk export) and vhost-vdpa

[Stefano Garzarella]

On Wed, Oct 26, 2022 at 05:39:23PM +0800, Yongji Xie wrote:
> Hi Stefano,
>
> On Wed, Oct 26, 2022 at 5:12 PM Stefano Garzarella <sgarzare@redhat.com> wrote:
> > Hi Xie,
> > I was testing libblkio [1] with QSD vduse-blk export and had some
> > issues.
> >
> > In a nutshell, QSD prints me the following messages when using
> > vhost-vdpa to access the device:
> >
> >   Failed to get vq[0] iova mapping
> >   Failed to update vring for vq[0]
> >
> > This happens only with vhost-vdpa, using virtio-vdpa instead the device
> > works fine.
> > I'm using Linux v6.0 and QEMU master (commit
> > 214a8da23651f2472b296b3293e619fd58d9e212).
> >
> > I haven't had much time to investigate, I hope to do it next week, but
> > maybe it's much faster for you.
> >
> > I saw that ioctl(VDUSE_IOTLB_GET_FD) in libvduse.c returns -1 (EPERM),
> > so IIUC in the kernel vduse_dev_broken() was called, and the device is
> > in a broken state.
> >
> >
> > We will use libblkio in QEMU [2] to access vDPA devices via vhost-vdpa.
> > But I'm doing these tests without QEMU for now, using an example inside
> > the libblkio repo:
> >
> > # Build libblkio and examples
> >     # Fedora/CentOS/RHEL
> >     dnf install -y git meson rust cargo python3-docutils rustfmt
> >     # Debian/Ubuntu
> >     apt-get install -y git meson rustc cargo python3-docutils
> >
> >     git clone https://gitlab.com/libblkio/libblkio.git
> >
> >     cd libblkio
> >     git checkout v1.1.0
> >
> >     meson setup build
> >     meson compile -C build
> >
> >
> > # On terminal 1
> >     modprobe vduse
> >     modprobe vhost-vdpa
> >
> >     qemu-img create -f qcow2 -o preallocation=full /path/to/test.qcow2 1g
> >
> >     qemu-storage-daemon \
> >       --blockdev 
> > file,filename=/path/to/test.qcow2,cache.direct=on,aio=native,node-name=file \
> >       --blockdev qcow2,file=file,node-name=qcow2 \
> >       --object iothread,id=iothread0 \
> >       --export 
> > vduse-blk,id=vduse0,name=vduse0,num-queues=1,node-name=qcow2,writable=on,iothread=iothread0
> >
> >
> > # On terminal 2
> >     vdpa dev add name vduse0 mgmtdev vduse
> >
> >     cd libblkio/build
> >
> >     # blkio-bench executes
> >     ./examples/blkio-bench virtio-blk-vhost-vdpa \
> >       path=/dev/vhost-vdpa-0 --runtime=5 --readwrite=randread
> >
> >     # after this step, QSD (running on terminal 1) prints the following 
> > messages:
> >       Failed to get vq[0] iova mapping
> >       Failed to update vring for vq[0]
> >
> > I don't know if I'm doing something wrong or in libblkio we have some
> > issue, but using vdpa-sim-blk works correctly, so maybe there is
> > something in vduse that is missing.
> >
> > Any help or suggestion is welcome :-)
>
> I'd like to know whether bio-bench uses the shared memory
> (tmpfs/hugetlbfs) as the vdpa memory region. This is what VDUSE needs.

Okay, so IIUC every memory regions should have an associated fd.

The buffers in libblkio are already allocated in this way, but it is not 
true for the virtqueue memory, I'll change it and test.

Thanks for the help :-)
Stefano