gluster-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Gluster-devel] Performance report and some issues


From: Jordi Moles
Subject: Re: [Gluster-devel] Performance report and some issues
Date: Thu, 06 Mar 2008 16:43:11 +0100
User-agent: Thunderbird 2.0.0.12 (X11/20080213)

Hi,

thanks for the info, but... where should i add the write-behind code? under afr? over afr?

something like this...

***clients****

volume espa1
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.1.204
 option remote-subvolume espai
end-volume

volume espai1
 type performance/write-behind
 option aggregate-size 1MB
 option flush-behind on
 subvolumes espa1
end-volume

volume espa2
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.1.205
 option remote-subvolume espai
end-volume

volume espai1
 type performance/write-behind
 option aggregate-size 1MB
 option flush-behind on
 subvolumes espa1
end-volume

.......

.......

volume namespace1
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.1.204
 option remote-subvolume nm
end-volume

volume namespace2
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.1.205
 option remote-subvolume nm
end-volume

volume grup1
 type cluster/afr
 subvolumes espai1 espai2
end-volume

volume grup2
 type cluster/afr
 subvolumes espai3 espai4
end-volume

volume grup3
 type cluster/afr
 subvolumes espai5 espai6
end-volume

volume nm
 type cluster/afr
 subvolumes namespace1 namespace2
end-volume

volume ultim
 type cluster/unify
 subvolumes grup1 grup2 grup3
 option scheduler rr
 option namespace nm
end-volume

************



or maybe like this:




***clients****

volume espai1
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.1.204
 option remote-subvolume espai
end-volume

volume espai2
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.1.205
 option remote-subvolume espai
end-volume

.......

........

volume namespace1
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.1.204
 option remote-subvolume nm
end-volume

volume namespace2
 type protocol/client
 option transport-type tcp/client
 option remote-host 192.168.1.205
 option remote-subvolume nm
end-volume

volume gru1
 type cluster/afr
 subvolumes espai1 espai2
end-volume

volume grup1
 type performance/write-behind
 option aggregate-size 1MB
 option flush-behind on
 subvolumes gru1
end-volume

volume gru2
 type cluster/afr
 subvolumes espai3 espai4
end-volume

volume grup2
 type performance/write-behind
 option aggregate-size 1MB
 option flush-behind on
 subvolumes gru2
end-volume

...


...

volume ultim
 type cluster/unify
 subvolumes grup1 grup2 grup3
 option scheduler rr
 option namespace nm
end-volume

************



En/na Amar S. Tumballi ha escrit:
> Hi Jordi,
> I see no performance translators on client side. You can load write-behind > and read-ahead/io-cache on client side. Without write-behind loaded, write
> performance will be *very* less.
>
> Regards,
> Amar
> On Thu, Mar 6, 2008 at 4:58 AM, Jordi Moles <address@hidden> wrote:
>
>> Hi,
>>
>> I want to report back the performance issues i've had so far with
>> glusterfs mainline 2.5, patch 690 and fuse-2.7.2glfs8.
>>
>> I'm setting a mail system, which is all ran by Xen 3.2.0 and every
>> "actual" piece of the mail system is a virtual machine from xen.
>>
>> Anyway... the virtual machines accessing glusterfs are 6 dovecots and 4
>> postfixs.  There are also 6 nodes, which share their own disk to the
>> gluster filesystem. Two of the nodes, share 2 disks, one for the
>> glusterfs, and the other for the namespace
>>
>> these are the conf files:
>>
>> ****nodes with namespace****
>>
>> volume esp
>>    type storage/posix
>>    option directory /mnt/compartit
>> end-volume
>>
>> volume espa
>>    type features/posix-locks
>>    subvolumes esp
>> end-volume
>>
>> volume espai
>>   type performance/io-threads
>>   option thread-count 15
>>   option cache-size 512MB
>>   subvolumes espa
>> end-volume
>>
>> volume nm
>>    type storage/posix
>>    option directory /mnt/namespace
>> end-volume
>>
>> volume ultim
>>    type protocol/server
>>    subvolumes espai nm
>>    option transport-type tcp/server
>>    option auth.ip.espai.allow *
>>    option auth.ip.nm.allow *
>> end-volume
>>
>> *************
>>
>>
>> ***nodes without namespace*****
>>
>> volume esp
>>    type storage/posix
>>    option directory /mnt/compartit
>> end-volume
>>
>> volume espa
>>    type features/posix-locks
>>    subvolumes esp
>> end-volume
>>
>> volume espai
>>   type performance/io-threads
>>   option thread-count 15
>>   option cache-size 512MB
>>   subvolumes espa
>> end-volume
>>
>> volume ultim
>>    type protocol/server
>>    subvolumes espai
>>    option transport-type tcp/server
>>    option auth.ip.espai.allow *
>> end-volume
>>
>> *****************************
>>
>>
>> ***clients****
>>
>> volume espai1
>>    type protocol/client
>>    option transport-type tcp/client
>>    option remote-host 192.168.1.204
>>    option remote-subvolume espai
>> end-volume
>>
>> volume espai2
>>    type protocol/client
>>    option transport-type tcp/client
>>    option remote-host 192.168.1.205
>>    option remote-subvolume espai
>> end-volume
>>
>> volume espai3
>>    type protocol/client
>>    option transport-type tcp/client
>>    option remote-host 192.168.1.206
>>    option remote-subvolume espai
>> end-volume
>>
>> volume espai4
>>    type protocol/client
>>    option transport-type tcp/client
>>    option remote-host 192.168.1.207
>>    option remote-subvolume espai
>> end-volume
>>
>> volume espai5
>>    type protocol/client
>>    option transport-type tcp/client
>>    option remote-host 192.168.1.213
>>    option remote-subvolume espai
>> end-volume
>>
>> volume espai6
>>    type protocol/client
>>    option transport-type tcp/client
>>    option remote-host 192.168.1.214
>>    option remote-subvolume espai
>> end-volume
>>
>> volume namespace1
>>    type protocol/client
>>    option transport-type tcp/client
>>    option remote-host 192.168.1.204
>>    option remote-subvolume nm
>> end-volume
>>
>> volume namespace2
>>    type protocol/client
>>    option transport-type tcp/client
>>    option remote-host 192.168.1.205
>>    option remote-subvolume nm
>> end-volume
>>
>> volume grup1
>>    type cluster/afr
>>    subvolumes espai1 espai2
>> end-volume
>>
>> volume grup2
>>    type cluster/afr
>>    subvolumes espai3 espai4
>> end-volume
>>
>> volume grup3
>>    type cluster/afr
>>    subvolumes espai5 espai6
>> end-volume
>>
>> volume nm
>>    type cluster/afr
>>    subvolumes namespace1 namespace2
>> end-volume
>>
>> volume ultim
>>    type cluster/unify
>>    subvolumes grup1 grup2 grup3
>>    option scheduler rr
>>    option namespace nm
>> end-volume
>>
>> ************
>>
>> The thing is that in earlier patches, the whole system used to hang,
>> with many different error messages.
>>
>> Right now, it's been on for days without any hang at all, but i'm facing
>> serious performance issues.
>>
>> By only running an "ls" command, it can take like 3 seconds to show
>> something when the system is "under load". It doesn't happen at all when >> there's no activity, so i don't thing has anything to do with xen. Well,
>> actually, "under load" can mean 3 mails arriving per second. I'm
>> monitoring everything, and no virtual machine is using more than 20% of
>> cpu or so.
>>
>> First, i had log level on both nodes and clients set to DEBUG, but now
>> is just WARNING, and i've restarted everything so many times.
>>
>> I was suggested to use "type performance/io-threads" on the node side.
>> It actually worked, before that, it wasn't 3 seconds, but 5 or more.
>> I've set the "thread-count" value to different values and also
>> "cache-size"
>>
>> The system is supposed to handle a big amount of traffic, far more than
>> 3 mails a second.
>>
>> What do you think about the whole set up? Should i keep using namespace? >> Should i use new nodes for namespaces? Should i use different values for
>> iothread?
>>
>> One last thing... i'm using reiserfs on the "storage devices" that nodes
>> share. Should i be using XFS or something else?
>>
>> Logs don't show any kind of error now... i don't have a clue of what is
>> failing now....
>>
>> I would be pleased if you could give some ideas.
>>
>> Thank you.
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
>
>
>






reply via email to

[Prev in Thread] Current Thread [Next in Thread]