Re: [Qemu-devel] [RFC/PATCH] Fix guest OS panic when 64bit BAR is present

[Avi Kivity]

On 01/31/2012 11:40 AM, Avi Kivity wrote:
> On 01/27/2012 06:42 AM, Alexey Korolev wrote:
> > On 27/01/12 04:12, Avi Kivity wrote:
> > > On 01/26/2012 04:36 PM, Michael S. Tsirkin wrote:
> > >> On Thu, Jan 26, 2012 at 03:52:27PM +0200, Avi Kivity wrote:
> > >>> On 01/26/2012 11:14 AM, Michael S. Tsirkin wrote:
> > >>>> On Wed, Jan 25, 2012 at 06:46:03PM +1300, Alexey Korolev wrote:
> > >>>>> Hi, 
> > >>>>> In this post
> > >>>>> http://lists.gnu.org/archive/html/qemu-devel/2011-12/msg03171.html 
> > >>>>> I've
> > >>>>> mentioned about the issues when 64Bit PCI BAR is present and 32bit
> > >>>>> address range is selected for it.
> > >>>>> The issue affects all recent qemu releases and all
> > >>>>> old and recent guest Linux kernel versions.
> > >>>>>
> > >>>>> We've done some investigations. Let me explain what happens.
> > >>>>> Assume we have 64bit BAR with size 32MB mapped at [0xF0000000 -
> > >>>>> 0xF2000000]
> > >>>>>
> > >>>>> When Linux guest starts it does PCI bus enumeration.
> > >>>>> The OS enumerates 64BIT bars using the following procedure.
> > >>>>> 1. Write all FF's to lower half of 64bit BAR
> > >>>>> 2. Write address back to lower half of 64bit BAR
> > >>>>> 3. Write all FF's to higher half of 64bit BAR
> > >>>>> 4. Write address back to higher half of 64bit BAR
> > >>>>>
> > >>>>> Linux code is here: 
> > >>>>> http://lxr.linux.no/#linux+v3.2.1/drivers/pci/probe.c#L149
> > >>>>>
> > >>>>> What does it mean for qemu?
> > >>>>>
> > >>>>> At step 1. qemu pci_default_write_config() recevies all FFs for lower
> > >>>>> part of the 64bit BAR. Then it applies the mask and converts the value
> > >>>>> to "All FF's - size + 1" (FE000000 if size is 32MB).
> > >>>>> Then pci_bar_address() checks if BAR address is valid. Since it is a
> > >>>>> 64bit bar it reads 0x00000000FE000000 - this address is valid. So qemu
> > >>>>> updates topology and sends request to update mappings in KVM with new
> > >>>>> range for the 64bit BAR FE000000 - 0xFFFFFFFF. This usually means 
> > >>>>> kernel
> > >>>>> panic on boot, if there is another mapping in the FE000000 - 
> > >>>>> 0xFFFFFFFF
> > >>>>> range, which is quite common.
> > >>>> Do you know why does it panic? As far as I can see
> > >>>> from code at
> > >>>> http://lxr.linux.no/#linux+v2.6.35.9/drivers/pci/probe.c#L162
> > >>>>
> > >>>>  171        pci_read_config_dword(dev, pos, &l);
> > >>>>  172        pci_write_config_dword(dev, pos, l | mask);
> > >>>>  173        pci_read_config_dword(dev, pos, &sz);
> > >>>>  174        pci_write_config_dword(dev, pos, l);
> > >>>>
> > >>>> BAR is restored: what triggers an access between lines 172 and 174?
> > >>> Random interrupt reading the time, likely.
> > >> Weird, what the backtrace shows is init, unrelated
> > >> to interrupts.
> > >>
> > > It's a bug then.  qemu doesn't undo the mapping correctly.
> > >
> > > If you have clear instructions, I'll try to reproduce it.
> > >
> > Well the easiest way to reproduce this is:
> >
> >
> > 1. Get kernel bzImage (version < 2.6.36)
> > 2. Apply patch to ivshmem.c
> >
> > ---
> > diff --git a/hw/ivshmem.c b/hw/ivshmem.c
> > index 1aa9e3b..71f8c21 100644
> > --- a/hw/ivshmem.c
> > +++ b/hw/ivshmem.c
> > @@ -341,7 +341,7 @@ static void create_shared_memory_BAR(IVShmemState *s, 
> > int fd) {
> >      memory_region_add_subregion(&s->bar, 0, &s->ivshmem);
> >  
> >      /* region for shared memory */
> > -    pci_register_bar(&s->dev, 2, PCI_BASE_ADDRESS_SPACE_MEMORY, &s->bar);
> > +    pci_register_bar(&s->dev, 2, 
> > PCI_BASE_ADDRESS_SPACE_MEMORY|PCI_BASE_ADDRESS_MEM_TYPE_64, &s->bar)
> >  }
> >  
> >  static void close_guest_eventfds(IVShmemState *s, int posn)
> > ---
> >
> > 3. Launch qemu with a command like that
> >
> > /usr/bin/qemu-system-x86_64 -M pc-0.14 -enable-kvm -m 2048 -smp 
> > 1,socket=1,cores=1,threads=1 -name centos54 -uuid
> > d37daefd-75bd-4387-cee1-7f0b153ee2af -nodefconfig -nodefaults -chardev
> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/centos54.monitor,server,nowait
> >  -mon chardev=charmonitor,id=monitor,mode=readline -rtc
> > base=utc -drive 
> > file=/dev/dock200-1/centos54,if=none,id=drive-ide0-0-0,format=raw -device
> > ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 
> > -drive
> > file=/data/CentOS-5.4-x86_64-bin-DVD.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
> >  -device
> > ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -chardev 
> > file,id=charserial0,path=/home/alexey/cent54.log -device
> > isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -k en-us 
> > -vga cirrus -device
> > virtio-balloon-pci,id=balloon0,bus=pci.0,multifunction=on,addr=0x4.0x0 
> > --device ivshmem,size=32,shm="shm" -kernel bzImage -append
> > "root=/dev/hda1 console=ttyS0,115200n8 console=tty0"
> >
> > in other words add: --device ivshmem,size=32,shm="shm"
> >
> > That is all.
> >
> > Note: it won't necessary cause panic message on some kernels it just hangs 
> > or reboots.
> >
>
> In fact qemu segfaults for me, since registering a ram region not on a
> page boundary is broken.  This happens when the ivshmem bar is split by
> the hpet region, which is less than page long.
>

Happens only with qemu-kvm for some reason.  Two separate bugs.

-- 
error compiling committee.c: too many arguments to function