Re: [Qemu-devel] [PATCH v1 2/2] reduce qemu's heap Rss size from 12252kB to 2752KB

[Xu, Anthony]

> -----Original Message-----
> From: Paolo Bonzini [mailto:address@hidden
> Sent: Tuesday, March 14, 2017 3:15 AM
> To: Xu, Anthony <address@hidden>
> Cc: Zhong, Yang <address@hidden>; address@hidden; Peng,
> Chao P <address@hidden>
> Subject: Re: [Qemu-devel] [PATCH v1 2/2] reduce qemu's heap Rss size from
> 12252kB to 2752KB
> 
> 
> 
> On 14/03/2017 06:14, Xu, Anthony wrote:
> > Below functions are registered in RCU thread
> > address_space_dispatch_free,
> > do_address_space_destroy
> > flatview_unref
> > reclaim_ramblock,
> > qht_map_destroy,
> > migration_bitmap_free
> >
> > first three are address space related, should work without global lock per
> above analysis.
> > The rest are very simple, seems doesn't need global lock.
> 
> flatview_unref can call object_unref and thus reach:

Okay,  flatview_unref is the one you worried about,

Flatview_unref is registered as a RCU callback only in 
address_space_update_topology,
Strangely, it is registered as a RCU callback, and is called directly in this 
function. 
Basially flatview_unref is called twice for old view. There are some comments, 
but it is not clear to me, is this a bug or by design? Is flatview_destroy 
called in current thread 
or RCU thread?


static void address_space_update_topology(AddressSpace *as)
{
    FlatView *old_view = address_space_get_flatview(as);
    FlatView *new_view = generate_memory_topology(as->root);

    address_space_update_topology_pass(as, old_view, new_view, false);
    address_space_update_topology_pass(as, old_view, new_view, true);

    /* Writes are protected by the BQL.  */
    atomic_rcu_set(&as->current_map, new_view);
    call_rcu(old_view, flatview_unref, rcu);

    /* Note that all the old MemoryRegions are still alive up to this
     * point.  This relieves most MemoryListeners from the need to
     * ref/unref the MemoryRegions they get---unless they use them
     * outside the iothread mutex, in which case precise reference
     * counting is necessary.
     */
    flatview_unref(old_view);

    address_space_update_ioeventfds(as);
}


Let me split the patch, Do you think below patch is correct?

--- a/memory.c
+++ b/memory.c
@@ -1503,15 +1503,9 @@ static void memory_region_finalize(Object *obj)
      * and cause an infinite loop.
      */
     mr->enabled = false;
-    memory_region_transaction_begin();
-    while (!QTAILQ_EMPTY(&mr->subregions)) {
-        MemoryRegion *subregion = QTAILQ_FIRST(&mr->subregions);
-        memory_region_del_subregion(mr, subregion);
-    }
-    memory_region_transaction_commit();
-
+    assert(QTAILQ_EMPTY(&mr->subregions));
     mr->destructor(mr);
-    memory_region_clear_coalescing(m
+    assert(QTAILQ_EMPTY(&mr->coalesced));
     g_free((char *)mr->name);
     g_free(mr->ioeventfds);
 }


Then let us take a look at flatview_unref, memory_region_unref may call any QOM 
finalize through 
Object_unref(mr->owner), that's your concern, I got it. It is way too 
complicated to look at each QOM 
object release callbacks and each QOM property release callbacks.  I gave up 
this path.

How about fall back to synchronize_rcu?
As for address space, the RCU read lock is used to protect PhysPageMap, but not 
the regular MemoryRegions, 
because that are not allocated in PhysPageMap build. Subpage MemoryRegion is an 
exception, because that is 
allocated in PhysPageMap build.

The MemoryRegions returned from address_space_translate are regular 
MemoryRegions, so 
address_space_write_continue and address_space_read_continue don't need RCU 
read lock.

Below patch reclaim address space in synchronized way, it reduces heap size 
from ~12MB to ~3MB.
You need to apply this patch with above patch. And when 
address_space_dispatch_free is called 
it already holds the global lock, we don't need to acquire the global lock in 
address_space_dispatch_free.
Please review this patch.

Thanks,
Anthony


diff --git a/cputlb.c b/cputlb.c
index 6c39927..98bd21f 100644
--- a/cputlb.c
+++ b/cputlb.c
@@ -347,6 +347,7 @@ void tlb_set_page_with_attrs(CPUState *cpu, target_ulong 
vaddr,
         tlb_add_large_page(env, vaddr, size);
     }

+    rcu_read_lock();
     sz = size;
     section = address_space_translate_for_iotlb(cpu, asidx, paddr, &xlat, &sz);
     assert(sz >= TARGET_PAGE_SIZE);
@@ -406,6 +407,7 @@ void tlb_set_page_with_attrs(CPUState *cpu, target_ulong 
vaddr,
     } else {
         te->addr_write = -1;
     }
+    rcu_read_unlock();
 }

 /* Add a new TLB entry, but without specifying the memory
diff --git a/exec.c b/exec.c
index 6fa337b..446d622 100644
--- a/exec.c
+++ b/exec.c
@@ -455,7 +455,7 @@ IOMMUTLBEntry address_space_get_iotlb_entry(AddressSpace 
*as, hwaddr addr,
     IOMMUTLBEntry iotlb = {0};
     MemoryRegionSection *section;
     MemoryRegion *mr;
-
+    rcu_read_lock();
     for (;;) {
         AddressSpaceDispatch *d = atomic_rcu_read(&as->dispatch);
         section = address_space_lookup_region(d, addr, false);
@@ -477,7 +477,7 @@ IOMMUTLBEntry address_space_get_iotlb_entry(AddressSpace 
*as, hwaddr addr,
                 | (addr & iotlb.addr_mask));
         as = iotlb.target_as;
     }
-
+    rcu_read_unlock();
     return iotlb;
 }

@@ -490,6 +490,7 @@ MemoryRegion *address_space_translate(AddressSpace *as, 
hwaddr addr,
     MemoryRegionSection *section;
     MemoryRegion *mr;

+    rcu_read_lock();
     for (;;) {
         AddressSpaceDispatch *d = atomic_rcu_read(&as->dispatch);
         section = address_space_translate_internal(d, addr, &addr, plen, true);
@@ -517,6 +518,7 @@ MemoryRegion *address_space_translate(AddressSpace *as, 
hwaddr addr,
     }

     *xlat = addr;
+    rcu_read_unlock();
     return mr;
 }

@@ -2413,7 +2415,8 @@ static void mem_commit(MemoryListener *listener)

     atomic_rcu_set(&as->dispatch, next);
     if (cur) {
-        call_rcu(cur, address_space_dispatch_free, rcu);
+        synchronize_rcu();
+        address_space_dispatch_free(cur);
     }
 }

@@ -2459,7 +2462,8 @@ void address_space_destroy_dispatch(AddressSpace *as)

     atomic_rcu_set(&as->dispatch, NULL);
     if (d) {
-        call_rcu(d, address_space_dispatch_free, rcu);
+        synchronize_rcu();
+        address_space_dispatch_free(d);
     }
 }

@@ -2686,12 +2690,10 @@ MemTxResult address_space_write(AddressSpace *as, 
hwaddr addr, MemTxAttrs attrs,
     MemTxResult result = MEMTX_OK;

     if (len > 0) {
-        rcu_read_lock();
         l = len;
         mr = address_space_translate(as, addr, &addr1, &l, true);
         result = address_space_write_continue(as, addr, attrs, buf, len,
                                               addr1, l, mr);
-        rcu_read_unlock();
     }

     return result;
@@ -2776,12 +2778,10 @@ MemTxResult address_space_read_full(AddressSpace *as, 
hwaddr addr,
     MemTxResult result = MEMTX_OK;

     if (len > 0) {
-        rcu_read_lock();
         l = len;
         mr = address_space_translate(as, addr, &addr1, &l, false);
         result = address_space_read_continue(as, addr, attrs, buf, len,
                                              addr1, l, mr);
-        rcu_read_unlock();
     }

     return result;
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index febe519..3557ac5 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -914,8 +914,6 @@ void vhost_device_iotlb_miss(struct vhost_dev *dev, 
uint64_t iova, int write)
     IOMMUTLBEntry iotlb;
     uint64_t uaddr, len;

-    rcu_read_lock();
-
     iotlb = address_space_get_iotlb_entry(dev->vdev->dma_as,
                                           iova, write);
     if (iotlb.target_as != NULL) {
@@ -936,7 +934,7 @@ void vhost_device_iotlb_miss(struct vhost_dev *dev, 
uint64_t iova, int write)
         }
     }
 out:
-    rcu_read_unlock();
+    return;
 }

 static int vhost_virtqueue_start(struct vhost_dev *dev,













> 
> - all QOM instance_finalize callbacks
> 
> - all QOM property release callbacks
> 
> In turn, of QOM property release callbacks the more important ones are
> release_drive (which calls blockdev_auto_del and blk_detach_dev) and
> release_chr (which calls qemu_chr_fe_deinit).
> 
> Your patch is incorrect, sorry.  If it were that simple, it would have
> been done already...
> 
> Paolo