Re: [Qemu-devel] [PATCH v2 03/17] memory: add ref/unref calls

[Alexey Kardashevskiy]

Hi.

Ok. Back to the bug with this patch. The initial problem with this patch is
that "make check" fails.

Please help with subpages.

It turned out that tests use MALLOC_PERTURB_ which is normally off. Who
does not know - this is a way to tell glibc to fill released memory with
some value and then debug accesses to released memory. Some bright mind
made it random what confuses a lot (and btw valgrind found nothing :-/ ).
So I spend some time before figured out how to reproduce it outside of the
qtest thingy.

The tree is qemu.org/master "bd5c51e Michael Roth qemu-char: don't issue
CHR_EVENT_OPEN in a BH" + replayed patches till the one from $subj on top
of it. QEMU is configured as "configure --target-list=x86_64-softmmu".

The magic is:

export MALLOC_PERTURB_=123
nc -l -U ~/qtest-16318.sock &
nc -l -U ~/qtest-16318.qmp &
x86_64-softmmu/qemu-system-x86_64  \
        -qtest unix:/home/alexey/qtest-16318.sock,nowait \
        -qtest-log /dev/null \
        -qmp unix:/home/alexey/qtest-16318.qmp,nowait \
        -pidfile ~/qtest-16318.pid -machine accel=qtest -vnc none

Immediate crash at (the very last backtrace in this mail is that crash).

x86_cpu_apic_realize() creates a subpage for IO:


#0  aik_dbg_start (address@hidden "subpage_init",
address@hidden, address@hidden)
    at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc.c:1297
#1  0x0000555555774299 in subpage_init (base=0x0, as=0x5555564a9260) at
/home/alexey/pcipassthru/qemu-impreza/exec.c:1696
#2  register_subpage (address@hidden,
address@hidden)
    at /home/alexey/pcipassthru/qemu-impreza/exec.c:845
#3  0x000055555577447d in mem_add (listener=0x555556523d08,
section=<optimized out>)
    at /home/alexey/pcipassthru/qemu-impreza/exec.c:881
#4  0x00005555557c2d69 in address_space_update_topology_pass
(address@hidden, address@hidden, old_view=...,
    new_view=...) at /home/alexey/pcipassthru/qemu-impreza/memory.c:751
#5  0x00005555557c64b8 in address_space_update_topology (as=0x5555564a9260)
at /home/alexey/pcipassthru/qemu-impreza/memory.c:766
#6  memory_region_transaction_commit () at
/home/alexey/pcipassthru/qemu-impreza/memory.c:790
#7  0x00005555557c79cd in memory_region_add_subregion_common
(mr=0x555556523c30, address@hidden,
    address@hidden) at
/home/alexey/pcipassthru/qemu-impreza/memory.c:1518
#8  0x00005555557c7ae8 in memory_region_add_subregion (mr=<optimized out>,
address@hidden,
    address@hidden) at
/home/alexey/pcipassthru/qemu-impreza/memory.c:1527
#9  0x0000555555663995 in sysbus_add_io (address@hidden,
address@hidden, address@hidden)
    at /home/alexey/pcipassthru/qemu-impreza/hw/core/sysbus.c:242
#10 0x000055555579cfce in vapic_init (dev=0x55555654e700) at
/home/alexey/pcipassthru/qemu-impreza/hw/i386/kvmvapic.c:707
#11 0x0000555555661651 in device_realize (dev=0x55555654e700,
err=0x7fffffffda40)
    at /home/alexey/pcipassthru/qemu-impreza/hw/core/qdev.c:178
#12 0x0000555555662cf3 in device_set_realized (obj=0x55555654e700,
value=0x1, err=0x7fffffffdb50)
    at /home/alexey/pcipassthru/qemu-impreza/hw/core/qdev.c:699
#13 0x000055555573358e in property_set_bool (obj=0x55555654e700,
v=<optimized out>, opaque=0x55555653c1f0, name=<optimized out>,
    errp=0x7fffffffdb50) at
/home/alexey/pcipassthru/qemu-impreza/qom/object.c:1301
#14 0x0000555555736445 in object_property_set_qobject (obj=0x55555654e700,
value=<optimized out>, name=0x555555896553 "realized",
    errp=0x7fffffffdb50) at
/home/alexey/pcipassthru/qemu-impreza/qom/qom-qobject.c:24
#15 0x000055555573525e in object_property_set_bool
(address@hidden, address@hidden,
    address@hidden "realized", address@hidden)
    at /home/alexey/pcipassthru/qemu-impreza/qom/object.c:852
#16 0x0000555555661c3a in qdev_init (address@hidden) at
/home/alexey/pcipassthru/qemu-impreza/hw/core/qdev.c:163
#17 0x0000555555661e91 in qdev_init_nofail (address@hidden)
    at /home/alexey/pcipassthru/qemu-impreza/hw/core/qdev.c:277
#18 0x0000555555663789 in sysbus_create_varargs
(address@hidden "kvmvapic", address@hidden)
    at /home/alexey/pcipassthru/qemu-impreza/hw/core/sysbus.c:157
#19 0x00005555557a4ead in sysbus_create_simple (irq=0x0,
addr=0xffffffffffffffff, name=0x5555558c73a1 "kvmvapic")
    at /home/alexey/pcipassthru/qemu-impreza/include/hw/sysbus.h:75
#20 apic_init_common (dev=0x555556535350) at
/home/alexey/pcipassthru/qemu-impreza/hw/intc/apic_common.c:311
#21 0x0000555555790fb6 in icc_device_realize (dev=0x555556535350,
errp=0x7fffffffdc80)
    at /home/alexey/pcipassthru/qemu-impreza/hw/cpu/icc_bus.c:50
#22 0x0000555555662cf3 in device_set_realized (obj=0x555556535350,
value=0x1, err=0x7fffffffdd90)
    at /home/alexey/pcipassthru/qemu-impreza/hw/core/qdev.c:699
#23 0x000055555573358e in property_set_bool (obj=0x555556535350,
v=<optimized out>, opaque=0x555556535610, name=<optimized out>,
    errp=0x7fffffffdd90) at
/home/alexey/pcipassthru/qemu-impreza/qom/object.c:1301
#24 0x0000555555736445 in object_property_set_qobject (obj=0x555556535350,
value=<optimized out>, name=0x555555896553 "realized",
    errp=0x7fffffffdd90) at
/home/alexey/pcipassthru/qemu-impreza/qom/qom-qobject.c:24
#25 0x000055555573525e in object_property_set_bool
(address@hidden, address@hidden,
    address@hidden "realized", address@hidden)
    at /home/alexey/pcipassthru/qemu-impreza/qom/object.c:852
#26 0x0000555555661c3a in qdev_init (dev=0x555556535350) at
/home/alexey/pcipassthru/qemu-impreza/hw/core/qdev.c:163
#27 0x00005555557d9a7c in x86_cpu_apic_realize (errp=0x7fffffffddd0,
cpu=0x55555653df50)
    at /home/alexey/pcipassthru/qemu-impreza/target-i386/cpu.c:2327
#28 x86_cpu_realizefn (dev=0x55555653df50, errp=0x7fffffffde20) at
/home/alexey/pcipassthru/qemu-impreza/target-i386/cpu.c:2397
#29 0x0000555555662cf3 in device_set_realized (obj=0x55555653df50,
value=0x1, err=0x7fffffffdf30)
    at /home/alexey/pcipassthru/qemu-impreza/hw/core/qdev.c:699
#30 0x000055555573358e in property_set_bool (obj=0x55555653df50,
v=<optimized out>, opaque=0x55555652e390, name=<optimized out>,
    errp=0x7fffffffdf30) at
/home/alexey/pcipassthru/qemu-impreza/qom/object.c:1301
---Type <return> to continue, or q <return> to quit---
#31 0x0000555555736445 in object_property_set_qobject (obj=0x55555653df50,
value=<optimized out>, name=0x555555896553 "realized",
    errp=0x7fffffffdf30) at
/home/alexey/pcipassthru/qemu-impreza/qom/qom-qobject.c:24
#32 0x000055555573525e in object_property_set_bool (obj=0x55555653df50,
value=<optimized out>, name=0x555555896553 "realized",
    errp=0x7fffffffdf30) at
/home/alexey/pcipassthru/qemu-impreza/qom/object.c:852
#33 0x000055555579f3b0 in pc_new_cpu (cpu_model=<optimized out>,
apic_id=0x0, icc_bridge=<optimized out>, errp=0x7fffffffdf70)
    at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc.c:911
#34 0x00005555557a0fc1 in pc_cpus_init (cpu_model=0x5555558c7a2d "qemu64",
address@hidden,
    address@hidden) at
/home/alexey/pcipassthru/qemu-impreza/hw/i386/pc.c:964
#35 0x00005555557a129f in pc_init1 (system_memory=0x555556522e60,
system_io=0x555556523c30, address@hidden,
    address@hidden "cad",
address@hidden,
    address@hidden "",
address@hidden,
    address@hidden, address@hidden,
address@hidden)
    at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc_piix.c:98
#36 0x00005555557a1aea in pc_init_pci (args=<optimized out>) at
/home/alexey/pcipassthru/qemu-impreza/hw/i386/pc_piix.c:242
#37 0x00005555555dcea0 in main (argc=<optimized out>, argv=<optimized out>,
envp=<optimized out>)
    at /home/alexey/pcipassthru/qemu-impreza/vl.c:4307




This subpage is released later due to some magic which I do not understand:


(gdb) bt
#0  aik_dbg (address@hidden "destroy_page_desc", address@hidden)
    at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc.c:1284
#1  0x0000555555773d48 in destroy_page_desc (section_index=<optimized out>)
at /home/alexey/pcipassthru/qemu-impreza/exec.c:773
#2  destroy_l2_mapping (level=0x0, lp=0x555556559e10) at
/home/alexey/pcipassthru/qemu-impreza/exec.c:791
#3  destroy_l2_mapping (lp=0x555556559e10, level=0x0) at
/home/alexey/pcipassthru/qemu-impreza/exec.c:777
#4  0x0000555555773c88 in destroy_l2_mapping (level=0x1, lp=0x555556559610)
at /home/alexey/pcipassthru/qemu-impreza/exec.c:789
#5  destroy_l2_mapping (lp=0x555556559610, level=0x1) at
/home/alexey/pcipassthru/qemu-impreza/exec.c:777
#6  0x0000555555773c88 in destroy_l2_mapping (level=0x2, lp=0x555556558e10)
at /home/alexey/pcipassthru/qemu-impreza/exec.c:789
#7  destroy_l2_mapping (lp=0x555556558e10, level=0x2) at
/home/alexey/pcipassthru/qemu-impreza/exec.c:777
#8  0x0000555555773c88 in destroy_l2_mapping (level=0x3, lp=0x555556523d00)
at /home/alexey/pcipassthru/qemu-impreza/exec.c:789
#9  destroy_l2_mapping (lp=0x555556523d00, level=0x3) at
/home/alexey/pcipassthru/qemu-impreza/exec.c:777
#10 0x0000555555773df8 in destroy_all_mappings (d=0x555556523d00) at
/home/alexey/pcipassthru/qemu-impreza/exec.c:800
#11 mem_begin (listener=0x555556523d08) at
/home/alexey/pcipassthru/qemu-impreza/exec.c:1732
#12 0x00005555557c6168 in memory_region_transaction_commit () at
/home/alexey/pcipassthru/qemu-impreza/memory.c:787
#13 0x00005555557c79cd in memory_region_add_subregion_common
(address@hidden, address@hidden,
    address@hidden) at
/home/alexey/pcipassthru/qemu-impreza/memory.c:1518
#14 0x00005555557c7a72 in memory_region_add_subregion_overlap
(mr=0x555556522e60, offset=0xfee00000, subregion=0x55555652d7b8,
    priority=<optimized out>) at
/home/alexey/pcipassthru/qemu-impreza/memory.c:1537
#15 0x00005555557a1038 in pc_cpus_init (cpu_model=0x5555558c7a2d "qemu64",
address@hidden,
    address@hidden) at
/home/alexey/pcipassthru/qemu-impreza/hw/i386/pc.c:976
#16 0x00005555557a129f in pc_init1 (system_memory=0x555556522e60,
system_io=0x555556523c30, address@hidden,
    address@hidden "cad",
address@hidden,
    address@hidden "",
address@hidden,
    address@hidden, address@hidden,
address@hidden)
    at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc_piix.c:98
#17 0x00005555557a1aea in pc_init_pci (args=<optimized out>) at
/home/alexey/pcipassthru/qemu-impreza/hw/i386/pc_piix.c:242
#18 0x00005555555dcea0 in main (argc=<optimized out>, argv=<optimized out>,
envp=<optimized out>)
    at /home/alexey/pcipassthru/qemu-impreza/vl.c:4307
(gdb)



And - crash:


#0  object_unref (obj=0xa7a7a7a7a7a7a7a7) at
/home/alexey/pcipassthru/qemu-impreza/qom/object.c:691
#1  0x00005555557c505c in memory_region_unref (mr=<optimized out>) at
/home/alexey/pcipassthru/qemu-impreza/memory.c:1172
#2  0x0000555555775953 in phys_sections_clear () at
/home/alexey/pcipassthru/qemu-impreza/exec.c:826
#3  0x0000555555775999 in core_begin (listener=<optimized out>) at
/home/alexey/pcipassthru/qemu-impreza/exec.c:1738
#4  0x00005555557c6168 in memory_region_transaction_commit () at
/home/alexey/pcipassthru/qemu-impreza/memory.c:787
#5  0x00005555557c79cd in memory_region_add_subregion_common
(address@hidden, address@hidden,
    address@hidden) at
/home/alexey/pcipassthru/qemu-impreza/memory.c:1518
#6  0x00005555557c7a72 in memory_region_add_subregion_overlap
(mr=0x555556522e60, offset=0xfee00000, subregion=0x55555652d7b8,
    priority=<optimized out>) at
/home/alexey/pcipassthru/qemu-impreza/memory.c:1537
#7  0x00005555557a1038 in pc_cpus_init (cpu_model=0x5555558c7a2d "qemu64",
address@hidden,
    address@hidden) at
/home/alexey/pcipassthru/qemu-impreza/hw/i386/pc.c:976
#8  0x00005555557a129f in pc_init1 (system_memory=0x555556522e60,
system_io=0x555556523c30, address@hidden,
    address@hidden "cad",
address@hidden,
    address@hidden "",
address@hidden,
    address@hidden, address@hidden,
address@hidden)
    at /home/alexey/pcipassthru/qemu-impreza/hw/i386/pc_piix.c:98
#9  0x00005555557a1aea in pc_init_pci (args=<optimized out>) at
/home/alexey/pcipassthru/qemu-impreza/hw/i386/pc_piix.c:242
#10 0x00005555555dcea0 in main (argc=<optimized out>, argv=<optimized out>,
envp=<optimized out>)
    at /home/alexey/pcipassthru/qemu-impreza/vl.c:4307
(gdb)





On 06/13/2013 07:02 PM, Alexey Kardashevskiy wrote:
> Fails on qtest_init() in tests/libqtest.c, "Broken pipe". I cannot easily
> see what is wrong here with this patch but it is 100% reproducible on x86_64
>  :(
> 
> 
> On 06/13/2013 04:28 PM, Alexey Kardashevskiy wrote:
>> Hi!
>>
>> I do not know how (yet) but this patch breaks qtest on x86 (I bisected it):
>>
>>
>> make check-qtest V=1
>> QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64
>> MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((RANDOM % 255 + 1))} gtester -k
>> --verbose -m=quick tests/fdc-test tests/ide-test tests/hd-geo-test
>> tests/rtc-test tests/i440fx-test tests/fw_cfg-test
>> TEST: tests/fdc-test... (pid=13049)
>> Broken pipe
>> FAIL: tests/fdc-test
>> TEST: tests/ide-test... (pid=13053)
>>   /x86_64/ide/identify:
>> Broken pipe
>> FAIL
>> GTester: last random seed: R02S2f8a8fd53ff256765db44cefb0a920ce
>> (pid=13057)
>>   /x86_64/ide/bmdma/setup:
>> Broken pipe
>> FAIL
>> GTester: last random seed: R02S0cec5d222cfd196e6e839e06d7ddde89
>> (pid=13061)
>>   /x86_64/ide/bmdma/simple_rw:                                         FAIL
>> GTester: last random seed: R02S46a30a1ccd33dc104919118330810a85
>> (pid=13062)
>>   /x86_64/ide/bmdma/short_prdt:                                        FAIL
>> GTester: last random seed: R02S19fdcc95895b870371ed5ddcc8b77eda
>> (pid=13063)
>>
>> [...]
>>
>>
>> On 06/04/2013 10:13 PM, Paolo Bonzini wrote:
>>> Add ref/unref calls at the following places:
>>>
>>> - places where memory regions are stashed by a listener and
>>>   used outside the BQL (including in Xen or KVM).
>>>
>>> - memory_region_find callsites
>>>
>>> - creation of aliases and containers (only the aliased/contained
>>>   region gets a reference to avoid loops)
>>>
>>> - around calls to del_subregion/add_subregion, where the region
>>>   could disappear after the first call
>>>
>>> Signed-off-by: Paolo Bonzini <address@hidden>
>>> ---
>>>  exec.c                                |  6 +++++-
>>>  hw/core/loader.c                      |  1 +
>>>  hw/display/exynos4210_fimd.c          |  6 ++++++
>>>  hw/display/framebuffer.c              | 12 +++++++-----
>>>  hw/i386/kvmvapic.c                    |  1 +
>>>  hw/misc/vfio.c                        |  2 ++
>>>  hw/virtio/dataplane/hostmem.c         |  7 +++++++
>>>  hw/virtio/vhost.c                     |  2 ++
>>>  hw/virtio/virtio-balloon.c            |  1 +
>>>  hw/xen/xen_pt.c                       |  4 ++++
>>>  include/hw/virtio/dataplane/hostmem.h |  1 +
>>>  kvm-all.c                             |  2 ++
>>>  memory.c                              | 20 ++++++++++++++++++++
>>>  target-arm/kvm.c                      |  2 ++
>>>  target-sparc/mmu_helper.c             |  1 +
>>>  xen-all.c                             |  2 ++
>>>  16 files changed, 64 insertions(+), 6 deletions(-)
> 
> 


-- 
Alexey