[PATCH v2 00/21] export/fuse: Use coroutines and multi-threading

[Hanna Czenczek]

Hi,

This series:
- Fixes some bugs/minor inconveniences,
- Removes libfuse from the request processing path,
- Make the FUSE export use coroutines for request handling,
- Introduces multi-threading into the FUSE export.

More detail on the v1 cover letter:

https://lists.nongnu.org/archive/html/qemu-block/2025-03/msg00359.html


Changes from v1:
- Patch 1: Clarified “polling” to be `aio_poll()`
- Patch 11 (new): Pulled out from patch 13 (prev. 11)
- Patch 12 (new): Suggested by Eric
- Patch 13 (prev. 11):
  - Drop false polling handlers
  - Use qemu_fcntl_addfl() instead of fcntl(F_SETFL) to keep
    pre-existing FD flags
  - Add a note that the buffer returned by read needs to be freed via
    qemu_vfree()
  - Pulled out a variable rename into the new patch 11
- Patch 15 (prev. 13):
  - Simplified the co_read_from_fuse_fd() interface thanks to no longer
    needing to support poll handlers
  - Increment in-flight counter before entering the coroutine to make it
    more obvious how tihs ensures that the export pointer remains valid
    throughout
- Patch 16 (new): Add a common multi-threading interface for exports
  instead of a specific one just for FUSE
- Patch 17 (new): Test cases for this new interface
- Patch 18 (prev. 14):
  - Drop the contrasting with virtio-blk from the commit message;
    explaining the interface is no longer necessary now that it’s
    introduced separately in patch 16.
  - Generally, the interface definition is removed in favor of the new
    one in patch 16.
  - Some rebase conflicts (due to other changes earlier in this series).
- Patch 19 (new): Stefan suggested adding an explicit note for users on
  how multi-threading behaves with FUSE, not least because in the future
  this behavior may depend on the specific implementation features
  chosen (io-uring or not).  Because the actual multi-thread interface
  is now on the common export options, it is no longer obvious where to
  put this implementation note; I decided to put it into the general
  description of the FUSE export options, inside of this dedicated
  patch.
- Patch 20 (new): Simple sanity test for FUSE multi-threading (just test
  that e.g. nothing crashes when running qemu-img bench on top)
- Patch 21 (prev. 15): Rebase conflict due to the changes in patch 15;
  kept Stefan’s R-b anyway


Review notes/suggestions I deliberately did not follow in v2:
- Stefan suggested to make patch 1 simpler and more robust by allocating
  a new buffer for each request.  This is indeed a simple change (for
  patch 1) that I wouldn’t mind, and that I also started to implement.
  However, eventually I decided against it:
  The problem doesn’t disappear with the rest of the series, it
  basically stays the exact same; though instead of an implicit
  aio_poll() leading to nested polling, it turns into an implicit
  coroutine yield doing pretty much the same.
  For performance, it is better not to allocate a new buffer for each
  request; we only really need a bounce buffer for writes, as there is
  no other case where we’d continue to read the request buffer after
  yielding.  Therefore, the final state I would like to have after this
  series is to use a common request buffer for all requests on a single
  queue, only using a bounce buffer for writes.
  With that, I think it’s better to implement exactly that right from
  the start, rather than introducing a new intermediate state.


git-backport-diff from v1:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/21:[0012] [FC] 'fuse: Copy write buffer content before polling'
002/21:[----] [--] 'fuse: Ensure init clean-up even with error_fatal'
003/21:[----] [--] 'fuse: Remove superfluous empty line'
004/21:[----] [--] 'fuse: Explicitly set inode ID to 1'
005/21:[----] [--] 'fuse: Change setup_... to mount_fuse_export()'
006/21:[----] [--] 'fuse: Fix mount options'
007/21:[----] [--] 'fuse: Set direct_io and parallel_direct_writes'
008/21:[----] [--] 'fuse: Introduce fuse_{at,de}tach_handlers()'
009/21:[----] [--] 'fuse: Introduce fuse_{inc,dec}_in_flight()'
010/21:[----] [--] 'fuse: Add halted flag'
011/21:[down] 'fuse: Rename length to blk_len in fuse_write()'
012/21:[down] 'block: Move qemu_fcntl_addfl() into osdep.c'
013/21:[0077] [FC] 'fuse: Manually process requests (without libfuse)'
014/21:[----] [--] 'fuse: Reduce max read size'
015/21:[0061] [FC] 'fuse: Process requests in coroutines'
016/21:[down] 'block/export: Add multi-threading interface'
017/21:[down] 'iotests/307: Test multi-thread export interface'
018/21:[0077] [FC] 'fuse: Implement multi-threading'
019/21:[down] 'qapi/block-export: Document FUSE's multi-threading'
020/21:[down] 'iotests/308: Add multi-threading sanity test'
021/21:[0002] [FC] 'fuse: Increase MAX_WRITE_SIZE with a second buffer'


Hanna Czenczek (21):
  fuse: Copy write buffer content before polling
  fuse: Ensure init clean-up even with error_fatal
  fuse: Remove superfluous empty line
  fuse: Explicitly set inode ID to 1
  fuse: Change setup_... to mount_fuse_export()
  fuse: Fix mount options
  fuse: Set direct_io and parallel_direct_writes
  fuse: Introduce fuse_{at,de}tach_handlers()
  fuse: Introduce fuse_{inc,dec}_in_flight()
  fuse: Add halted flag
  fuse: Rename length to blk_len in fuse_write()
  block: Move qemu_fcntl_addfl() into osdep.c
  fuse: Manually process requests (without libfuse)
  fuse: Reduce max read size
  fuse: Process requests in coroutines
  block/export: Add multi-threading interface
  iotests/307: Test multi-thread export interface
  fuse: Implement multi-threading
  qapi/block-export: Document FUSE's multi-threading
  iotests/308: Add multi-threading sanity test
  fuse: Increase MAX_WRITE_SIZE with a second buffer

 qapi/block-export.json               |   39 +-
 include/block/export.h               |   12 +-
 include/qemu/osdep.h                 |    1 +
 block/export/export.c                |   48 +-
 block/export/fuse.c                  | 1181 ++++++++++++++++++++------
 block/export/vduse-blk.c             |    7 +
 block/export/vhost-user-blk-server.c |    8 +
 block/file-posix.c                   |   17 +-
 nbd/server.c                         |    6 +
 util/osdep.c                         |   18 +
 tests/qemu-iotests/307               |   47 +
 tests/qemu-iotests/307.out           |   18 +
 tests/qemu-iotests/308               |   55 +-
 tests/qemu-iotests/308.out           |   61 +-
 14 files changed, 1213 insertions(+), 305 deletions(-)

-- 
2.49.0