Re: [Qemu-devel] [Qemu-block] [PATCH 0/3] mirror: Fix guest responsiveness during bitmap scan

[Alexandre DERUMIER]

Hi again,

I have redone a lot of tests,

with raw on nfs
---------------

Patch 3/3, fix my problem (not apply patch 1/3 and patch 2/3).

without patch 3/3, I'm seeing a lot of lseek, can take some minutes with guest 
hang.

with patch 3/3, it almost take no time to generate the bitmap, no guest hang . 
(I have tested with differents raw files, different sizes)


with raw on ext3
----------------
Here, this hurt a lot. Not with every disk, but with some disks is totaly 
hanging the guest

-----------------------
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND    
 4387 root      20   0 8985788 288900  12532 R  99.3  1.8   2:50.82 qemu    



  10.27%  [kernel]                  [k] ext4_es_find_delayed_extent_range
   9.64%  [kernel]                  [k] _raw_read_lock
   9.17%  [kernel]                  [k] ext4_es_lookup_extent
   6.63%  [kernel]                  [k] down_read
   5.49%  [kernel]                  [k] ext4_map_blocks
   4.82%  [kernel]                  [k] up_read
   3.09%  [kernel]                  [k] ext4_ind_map_blocks
   2.58%  [kernel]                  [k] ext4_block_to_path.isra.9
   2.18%  [kernel]                  [k] kvm_arch_vcpu_ioctl_run
   1.99%  [kernel]                  [k] vmx_get_segment
   1.99%  [kernel]                  [k] ext4_llseek
   1.62%  [kernel]                  [k] ext4_get_branch
   1.43%  [kernel]                  [k] 
vmx_segment_access_rights.isra.28.part.29
   1.23%  [kernel]                  [k] x86_decode_insn
   1.00%  [kernel]                  [k] copy_user_generic_string
   0.83%  [kernel]                  [k] x86_emulate_instruction
   0.81%  [kernel]                  [k] rmode_segment_valid
   0.78%  [kernel]                  [k] gfn_to_hva_prot
   0.74%  [kernel]                  [k] __es_tree_search
   0.73%  [kernel]                  [k] do_insn_fetch
   0.70%  [kernel]                  [k] vmx_vcpu_run
   0.69%  [kernel]                  [k] vmx_read_guest_seg_selector
   0.67%  [kernel]                  [k] vmcs_writel
   0.64%  [kernel]                  [k] init_emulate_ctxt
   0.62%  [kernel]                  [k] native_write_msr_safe
   0.61%  [kernel]                  [k] kvm_apic_has_interrupt
   0.57%  [kernel]                  [k] vmx_handle_external_intr
   0.57%  [kernel]                  [k] x86_emulate_insn
   0.56%  [kernel]                  [k] vmx_handle_exit
   0.50%  [kernel]                  [k] native_read_tsc
   0.47%  [kernel]                  [k] _cond_resched
   0.46%  [kernel]                  [k] decode_operand
   0.44%  [kernel]                  [k] __srcu_read_lock
   0.41%  [kernel]                  [k] __linearize.isra.25
   0.41%  [kernel]                  [k] vmx_read_l1_tsc
   0.39%  [kernel]                  [k] clear_page_c
   0.39%  [kernel]                  [k] vmx_get_interrupt_shadow
   0.36%  [kernel]                  [k] vmx_set_interrupt_shadow


Seem to be a known kernel bug:

http://qemu.11.n7.nabble.com/Re-2-Strange-problems-with-lseek-in-qemu-img-map-td334414.html

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=14516bb7bb6ffbd49f35389f9ece3b2045ba5815




So, I think patch 3/3 is enough. (could be great to have it for qemu 2.4)

Thanks again Fam for your hard work.

Regards,

Alexandre


----- Mail original -----
De: "aderumier" <address@hidden>
À: "Fam Zheng" <address@hidden>
Cc: "Kevin Wolf" <address@hidden>, "Stefan Hajnoczi" <address@hidden>, 
"qemu-devel" <address@hidden>, address@hidden
Envoyé: Vendredi 10 Juillet 2015 12:36:40
Objet: Re: [Qemu-devel] [Qemu-block] [PATCH 0/3] mirror: Fix guest 
responsiveness during bitmap scan

>>Does it completely hang? 
yes, can't ping network, and vnc console is frozen. 


>>What does "perf top" show? 
I'll do test today . (I'm going to vacation this night,I'll try to send results 
before that) 

----- Mail original ----- 
De: "Fam Zheng" <address@hidden> 
À: "aderumier" <address@hidden> 
Cc: "Stefan Hajnoczi" <address@hidden>, "Kevin Wolf" <address@hidden>, 
"qemu-devel" <address@hidden>, address@hidden 
Envoyé: Vendredi 10 Juillet 2015 09:13:33 
Objet: Re: [Qemu-devel] [Qemu-block] [PATCH 0/3] mirror: Fix guest 
responsiveness during bitmap scan 

On Fri, 07/10 08:54, Alexandre DERUMIER wrote: 
> >>Thinking about this again, I doubt 
> >>that lengthening the duration with a hardcoded value benifits everyone; and 
> >>before Alexandre's reply on old server/slow disks 
> 
> With 1ms sleep, I can reproduce the hang 100% with a fast cpu (xeon e5 v3 
> 3,1ghz) and source raw file on nfs. 

Does it completely hang? I can't reproduce this with my machine mirroring from 
nfs to local: the guest runs smoothly. What does "perf top" show? 

Fam 

> 
> 
> ----- Mail original ----- 
> De: "Fam Zheng" <address@hidden> 
> À: "Stefan Hajnoczi" <address@hidden> 
> Cc: "Kevin Wolf" <address@hidden>, "qemu-devel" <address@hidden>, 
> address@hidden 
> Envoyé: Vendredi 10 Juillet 2015 08:43:50 
> Objet: Re: [Qemu-devel] [Qemu-block] [PATCH 0/3] mirror: Fix guest 
> responsiveness during bitmap scan 
> 
> On Thu, 07/09 14:02, Stefan Hajnoczi wrote: 
> > This patch only converts the mirror block job to use the new relax 
> > function. The other block jobs (stream, backup, commit) are still using 
> > a 0 ns delay and are therefore broken. They should probably be 
> > converted in the same series. 
> 
> That's because they all can be perfectly mitigated by setting a reasonable 
> "speed" that matchs the host horsepower. Thinking about this again, I doubt 
> that lengthening the duration with a hardcoded value benifits everyone; and 
> before Alexandre's reply on old server/slow disks, I don't recall any report, 
> because the coroutines would already yield often enough, when waiting for IO 
> to 
> complete. So I am not sure whether they're broken in practice. 
> 
> Fam