On Tue, Nov 16, 2021 at 11:30 PM Vladislav Yaroshchuk
<yaroshchuk2000@gmail.com> wrote:
>
>
>
> вт, 16 нояб. 2021 г. в 10:23, Jason Wang <jasowang@redhat.com>:
>>
>> On Mon, Nov 15, 2021 at 6:45 PM Vladislav Yaroshchuk
>> <yaroshchuk2000@gmail.com> wrote:
>> >
>> >
>> >
>> > пн, 15 нояб. 2021 г. в 07:53, Jason Wang <jasowang@redhat.com>:
>> >>
>> >> On Fri, Nov 12, 2021 at 5:14 PM Vladislav Yaroshchuk
>> >> <yaroshchuk2000@gmail.com> wrote:
>> >> >
>> >> > macOS provides networking API for VMs called 'vmnet.framework':
>> >> > https://developer.apple.com/documentation/vmnet
>> >> >
>> >> > We can provide its support as the new QEMU network backends which
>> >> > represent three different vmnet.framework interface usage modes:
>> >> >
>> >> > * `vmnet-shared`:
>> >> > allows the guest to communicate with other guests in shared mode and
>> >> > also with external network (Internet) via NAT. Has (macOS-provided)
>> >> > DHCP server; subnet mask and IP range can be configured;
>> >> >
>> >> > * `vmnet-host`:
>> >> > allows the guest to communicate with other guests in host mode.
>> >> > By default has enabled DHCP as `vmnet-shared`, but providing
>> >> > network unique id (uuid) can make `vmnet-host` interfaces isolated
>> >> > from each other and also disables DHCP.
>> >> >
>> >> > * `vmnet-bridged`:
>> >> > bridges the guest with a physical network interface.
>> >> >
>> >> > This backends cannot work on macOS Catalina 10.15 cause we use
>> >> > vmnet.framework API provided only with macOS 11 and newer. Seems
>> >> > that it is not a problem, because QEMU guarantees to work on two most
>> >> > recent versions of macOS which now are Big Sur (11) and Monterey (12).
>> >> >
>> >> > Also, we have one inconvenient restriction: vmnet.framework interfaces
>> >> > can create only privileged user:
>> >> > `$ sudo qemu-system-x86_64 -nic vmnet-shared`
>> >> >
>> >> > Attempt of `vmnet-*` netdev creation being unprivileged user fails with
>> >> > vmnet's 'general failure'.
>> >> >
>> >> > This happens because vmnet.framework requires `com.apple.vm.networking`
>> >> > entitlement which is: "restricted to developers of virtualization software.
>> >> > To request this entitlement, contact your Apple representative." as Apple
>> >> > documentation says:
>> >> > https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_vm_networking
>> >>
>> >> Do you know how multipass work? Looks like it uses vmnet without privileges.
>> >>
>> >
>> > I've just checked this, and they still need root privileges. They have a
>> > `multipassd` daemon which is launched as root by launchd by default.
>> >
>> > ```
>> > bash-5.1$ ps axo ruser,ruid,comm | grep multipass
>> > root 0 /Library/Application Support/com.canonical.multipass/bin/multipassd
>> > root 0 /Library/Application Support/com.canonical.multipass/bin/hyperkit
>> > ```
>> >
>> > That's the reason why it's required to 'enter a password' while multipass installation:
>> > it creates launchd plist (kinda launch rule) and places it to /Library/LaunchDaemons/,
>> > which can be done only with root privileges.
>> >
>> > ```
>> > bash-5.1$ ll /Library/LaunchDaemons | grep multipass
>> > -rw-r--r-- 1 root wheel 1.1K 15 Nov 12:47 com.canonical.multipassd.plist
>> > ```
>> >
>> > And after this launchd launches this multipassd daemon as root every boot.
>> >
>> > So an unprivileged user can launch a multipass VM instance, but actually the
>> > `hyperkit` process which interacts with vmnet is gonna be launched by multipassd
>> > running as root.
>>
>> I wonder how it passes the vmnet object to qemu? Nothing obvious from
>> the qemu command line that multipass launched:
>>
>> -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:52:e8:e4
>>
>> (But I haven't had time to check their qemu codes).
>>
>
> It seems they just use QEMU with patch by Phillip Tennen:
> https://patchew.org/QEMU/20210218134947.1860-1-phillip.ennen@gmail.com/
>
> In that patch he does quite the same as we in this series, the
> difference remains in foreground: he creates one new 'vmnet-macos'
> netdev, and uses 'mode=shared' property to choose vmnet
> operating mode. I decided to create three different netdevs instead
> (vmnet-shared, vmnet-host, vmnet-bridged). Also I've added some
> features related to isolation and ipv6.
Ok.
>
> > I wonder how it passes the vmnet object to qemu
> I hope I clearly described this.
A silly question, how did the 'hyperkit' process pass the vmnet object to qemu?
I think I didn't describe it very well in the previous mail.
The 'hyperkit' and QEMU are two multipass 'drivers'.
hyperkit is an independent hypervisor based on xhyve (bhyve):
So there are many ways to launch multipass VMs:
* VM <-> hyperkit <-> host
* VM <-> QEMU <-> host
* VM <-> virtualbox <-> host
* etc.
So hyperkit doesn't pass anything to QEMU, it's a separate
hypervisor and multipass driver. But it uses vmnet backend
too, the same way as we do:
>
>> >
>> > tl;dr sadly, we can't interact with vmnet.framework without having our binary correctly
>> > signed and being an unprivileged user. Root privileges or special signature with
>> > entitlement is required.
>>
>> This is something similar to what happens in other OS. E.g for the tap
>> backend, it can't be created without privileges. So qemu allows to:
>>
>> 1) the TAP to be created by privileged program like libvirt, and its
>> fd could be passed to qemu via SCM_RIGHTS
>> 2) run a set-uid helper to create and config TAP
>>
>> This is something we need to consider now or in the future probably.
>>
>
> Seems we can do nothing with this if we have qemu-bundled &
> direct vmnet.framework interaction, it always requires privileges
> or entitlement.
> The workaround can be moving vmnet-related things to
> another helper binary running with privileges, and usage of
> this helper somewhere between qemu and vmnet.
Yes, that's my point.
>
> I think for now it's applicable to leave it as is, having qemu
> that requires privileges for vmnet.framework usage.
Right, but we need to consider it for the future.
Btw, if you wish, you can list yourself as the maintainer for this backend.
Ok, thank you! Let me do this in the next patch version, after
we finish our discussion and I fix all the issues you pointed
in your review.
Thanks
>
>>
>> Thanks
>>
>> >
>> >
>> >> Thanks
>> >>
>> >
>> > Thank you for your review, I will check it this week and reply as soon as possible.
>> >
>> >>
>> >> >
>> >> > One more note: we still have quite useful but not supported
>> >> > 'vmnet.framework' features as creating port forwarding rules, IPv6
>> >> > NAT prefix specifying and so on.
>> >> >
>> >> > Nevertheless, new backends work fine and tested within `qemu-system-x86-64`
>> >> > on macOS Bir Sur 11.5.2 host with such nic models:
>> >> > * e1000-82545em
>> >> > * virtio-net-pci
>> >> > * vmxnet3
>> >> >
>> >> > The guests were:
>> >> > * macOS 10.15.7
>> >> > * Ubuntu Bionic (server cloudimg)
>> >> >
>> >> >
>> >> > This series partially reuses patches by Phillip Tennen:
>> >> > https://patchew.org/QEMU/20210218134947.1860-1-phillip.ennen@gmail.com/
>> >> > So I included his signed-off line into one of the commit messages and
>> >> > also here.
>> >> >
>> >> > v1 -> v2:
>> >> > Since v1 minor typos were fixed, patches rebased onto latest master,
>> >> > redundant changes removed (small commits squashed)
>> >> >
>> >> > v2 -> v3:
>> >> > - QAPI style fixes
>> >> > - Typos fixes in comments
>> >> > - `#include`'s updated to be in sync with recent master
>> >> > v3 -> v4:
>> >> > - Support vmnet interfaces isolation feature
>> >> > - Support vmnet-host network uuid setting feature
>> >> > - Refactored sources a bit
>> >> > v4 -> v5:
>> >> > - Missed 6.2 boat, now 7.0 candidate
>> >> > - Fix qapi netdev descriptions and styles
>> >> > (@subnetmask -> @subnet-mask)
>> >> > - Support vmnet-shared IPv6 prefix setting feature
>> >> >
>> >> > Signed-off-by: Phillip Tennen <phillip@axleos.com>
>> >> > Signed-off-by: Vladislav Yaroshchuk <yaroshchuk2000@gmail.com>
>> >> >
>> >> > Vladislav Yaroshchuk (6):
>> >> > net/vmnet: add vmnet dependency and customizable option
>> >> > net/vmnet: add vmnet backends to qapi/net
>> >> > net/vmnet: implement shared mode (vmnet-shared)
>> >> > net/vmnet: implement host mode (vmnet-host)
>> >> > net/vmnet: implement bridged mode (vmnet-bridged)
>> >> > net/vmnet: update qemu-options.hx
>> >> >
>> >> > meson.build | 4 +
>> >> > meson_options.txt | 2 +
>> >> > net/clients.h | 11 ++
>> >> > net/meson.build | 7 +
>> >> > net/net.c | 10 ++
>> >> > net/vmnet-bridged.m | 111 ++++++++++++
>> >> > net/vmnet-common.m | 325 ++++++++++++++++++++++++++++++++++
>> >> > net/vmnet-host.c | 111 ++++++++++++
>> >> > net/vmnet-shared.c | 92 ++++++++++
>> >> > net/vmnet_int.h | 48 +++++
>> >> > qapi/net.json | 127 ++++++++++++-
>> >> > qemu-options.hx | 25 +++
>> >> > scripts/meson-buildoptions.sh | 3 +
>> >> > 13 files changed, 874 insertions(+), 2 deletions(-)
>> >> > create mode 100644 net/vmnet-bridged.m
>> >> > create mode 100644 net/vmnet-common.m
>> >> > create mode 100644 net/vmnet-host.c
>> >> > create mode 100644 net/vmnet-shared.c
>> >> > create mode 100644 net/vmnet_int.h
>> >> >
>> >> > --
>> >> > 2.23.0
>> >> >
>> >>
>> >
>> >
>> > --
>> > Best Regards,
>> >
>> > Vladislav Yaroshchuk
>>
>
>
> --
> Best Regards,
>
> Vladislav Yaroshchuk