Re: [Qemu-devel] O_DIRECT on glusterfs (was Re: [PATCH v9 4/4] block: Support GlusterFS as a QEMU block backend)

[Paolo Bonzini]

Il 03/10/2012 20:17, Anand Avati ha scritto:
> On 10/03/2012 11:17 AM, Paolo Bonzini wrote:
>> Il 03/10/2012 19:58, Anand Avati ha scritto:
>>>>
>>>> I think these 3 lines should be removed.  We're bypassing the host
>>>> buffer cache just by virtue of using a userspace driver, and that's
>>>> what
>>>> cache=none cares about.
>>>
>>> O_DIRECT also has an effect on the behavior of the "client side" (the
>>> part within the qemu) of Gluster stack as well. I presume the intention
>>> of O_DIRECT is to minimize use of memory (whether as host' page cache or
>>> buffered data in user space). To that end it is a good idea to leave
>>> O_DIRECT flag set.
>>>
>>> The behavior of whether gluster bricks need to get the O_DIRECT
>>> propagated or not is a different issue. We are exploring the possibility
>>> of not sending O_DIRECT flag over the wire to mimic NFS behavior. That
>>> would be independent of the qemu block driver setting the open flag.
>>
>> What is the effect of O_DIRECT on the client exactly?
> 
> To avoid caching in the io-cache module, disable read-ahead etc (if
> those translators are loaded). The behavior in write-behind is tunable.
> You could either disable write-behind entirely (which will happen once
> libgfapi supports 0-copy/RDMA) or perform a sliding-window like
> size-limited write-behind (defaults to 1MB).

Thanks for the information.

I think we need to benchmark it and see what is the actual difference in
memory usage vs. performance.  The usual reasons for cache=none (enable
Linux native AIO + allow migration with shared non-coherent storage) do
not apply to gluster.

Paolo