Re: [Qemu-devel] [libvirt] [RFC 0/7] Live Migration with Pass-through Devices proposal

[Laine Stump]

On 04/23/2015 04:34 AM, Chen Fan wrote:
>
> On 04/20/2015 06:29 AM, Laine Stump wrote:
>> On 04/17/2015 04:53 AM, Chen Fan wrote:
>>> -  on destination side, check whether need to hotplug new NIC
>>> according to specified XML.
>>>     usually, we use migrate "--xml" command option to specify the
>>> destination host NIC mac
>>>     address to hotplug a new NIC, because source side passthrough
>>> NIC mac address is different,
>>>     then hotplug the deivce according to the destination XML
>>> configuration.

>> Why does the MAC address need to be different? Are you suggesting doing
>> this with passed-through non-SRIOV NICs? An SRIOV virtual function gets
>> its MAC address from the libvirt config, so it's very simple to use the
>> same MAC address across the migration. Any network card that would be
>> able to do this on any sort of useful scale will be SRIOV-capable (or
>> should be replaced with one that is - some of them are not that
>> expensive).

> Hi Laine,
>
> I think SRIOV virtual NIC to support migration is good idea,
> but I also think some passthrough NIC without SRIOV-capable. for
> these NIC devices we only able to use <hostdev> to specify the
> passthrough
> function, so for these NIC I think we should support too.

As I think you've already discovered, passing through non-SRIOV NICS is
problematic. It is completely impossible for the host to change their
MAC address before assigning them to the guest - the guest's driver sees
standard netdev hardware and resets it, which resets the MAC address to
the original value burned into the firmware. This makes management more
complicated, especially when you get into scenarios such as what we're
discussing (i.e. migration) where the actual hardware (and thus MAC
address) may be different from one run to the next.

Since libvirt's <interface> element requires a fixed MAC address in the
XML, it's not possible to have an <interface> that gets the actual
device from a network pool (without some serious hacking to that code),
and there is no support for plain (non-network) <hostdev> device pools;
there would need to be a separate (nonexistent) driver for that. Since
the <hostdev> element relies on the PCI address of the device (in the
<source> subelement, which also must be fixed) to determine which device
to passthrough, a domain config with a <hostdev> that could be run on
two different machines would require the device to reside at exactly the
same PCI address on both machines, which is a very serious limitation to
have in an environment large enough that migrating domains is a requirement.

Also, non-SRIOV NICs are limited to a single device per physical port,
meaning probably at most 4 devices per physical host PCIe slot, and this
results in a greatly reduced density on the host (and even more so on
the switch that connects to the host!) compared to even the old Intel
82576 cards, which have 14 VFs (7VFs x 2 ethernet ports). Think about it
- with an 82576, you can get 14 guests into 1 PCIe slot and 2 switch
ports, while the same number of guests with non-SRIOV would take 4 PCIe
slots and 14(!) switch ports. The difference is even more striking when
comparing to chips like the 82599 (64 VFs per port x 2), or a Mellanox
(also 64?) or SolarFlare (128?) card. And don't forget that, because you
don't have pools of devices to be automatically chosen from, that each
guest domain that will be migrated requires a reserved NIC on *every*
machine it will be migrated to (no other domain can be configured to use
that NIC, in order to avoid conflicts).

Of course you could complicate the software by adding a driver that
manages pools of generic hostdevs, and coordinates MAC address changes
with the guest (part of what you're suggesting), but all that extra
complexity not only takes a lot of time and effort to develop, it also
creates more code that needs to be maintained and tested for regressions
at each release.

The alternative is to just spend $130 per host for an 82576 or Intel
I350 card (these are the cheapest SRIOV options I'm aware of). When
compared to the total cost of any hardware installation large enough to
support migration and have performance requirements high enough that NIC
passthrough is needed, this is a trivial amount.

I guess the bottom line of all this is that (in my opinion, of course
:-) supporting useful migration of domains that used passed-through
non-SRIOV NICs would be an interesting experiment, but I don't see much
utility to it, other than "scratching an intellectual itch", and I'm
concerned that it would create more long term maintenance cost than it
was worth.