[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Gluster-devel] Can I bring a development idea to Dev's attention?

From: Ed W
Subject: Re: [Gluster-devel] Can I bring a development idea to Dev's attention?
Date: Fri, 24 Sep 2010 11:07:20 +0100
User-agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv: Gecko/20100915 Lightning/1.0b2 Thunderbird/3.1.4

 On 24/09/2010 08:25, Shehjar Tikoo wrote:
Thanks, we have similar locking improvements in mind but cannot promise a date when these will be available. Some of the challenges that we'll need to think about are how to map any such locking scheme to standard locking behaviour for posix/nfsv3/v4/cifs.

Hi, thanks for replying

Whilst I can see that there is some optimisation to be had by combining brick level locking with filesystem level locking, I just want to clarify that my proposal was really about intra-brick locks, and not really about the toplevel filesystem level locking?

Just to clarify (and apologies if I'm trying to teach filesystem experts really obvious stuff...)

- The goal of any fileserver is to take async requests from lots of clients and arrange to serialise that access - Up until recently such fileservers have been on a single machine, but involving multiple async clients connecting - Even in a single server solution the bottleneck becomes that each client cannot cache *any* data since it's not known if the server copy has changed since we accessed it (even a microsecond earlier) - The solution which has become popular (see CIFS, NFSV4 (?), GFS2, etc) was to offer clients an "optimistic lock", ie the client can acquire a token which while it's held means that it can cache data locked by that token and even offer writeback optimisations on that data (obviously subject to whatever the application tolerates for unsync'ed data) - This "optimistic lock" means that we effectively push the file locking to the client, hence once a lock is acquired then further access by the client is no longer bounded by the network access latency, under many circumstances this leads to massive speedups - Clearly when a second client comes along and demands access to the same data then we need a process to break the lock and inform the first client that they need to reacquire the lock (or revert to a kind of "write-through" access system while waiting)

So this process clearly benefits situations where there is serialised access by single clients at a time. Excluding databases however, this access pattern seems quite common for lots of applications

So with regards to Gluster I would see that we need this same type of locking implemented at the brick level. Hence if you re-read the description above, then each *gluster server* would be the possible clients (think of the lower level being bricks talking to each other, and the upper level being clients talking to bricks). ie yes, posix locking needs to serialise access to every end client that connects to every brick, but we can also benefit from locking to serialise access between bricks (if 3,000 clients hammer one brick for a single file, then we care that our single brick is allowed to read/write that file freely because it informed the other bricks that it now holds a lock, it's a separate problem to serialise all the clients talking to the one brick)

So compared with traditional fileservers we actually need two levels of locking to serialise access. At one level we need to serialise clients access to the filesystem, and lower down we need to serialise access between bricks

I think an alternative way of looking (and perhaps implementing) the situation could be something like:

- Consider two bricks with files replicated between them
- Client 1 accesses Brick 1 and requests File A
- Brick 1 contacts the other replicas and requests to become the "master replica" of that file. All future accesses to that file must now go through only Brick 1 while it remains in that "role" - If Client 2 accesses Brick 1 and tries to do something with File A, then the normal filesystem locking must arrange for serialisation between Client 1 and Client 2, however, Brick 1 need not contact any other brick and there is no network latency penalty serving that file to Client 2 (obviously at some point one client will write data and we need to sync that, but read access incurs no network access)

- OK, now the trick is what happens when Client 3 accesses Brick 2 and requests File A... Somehow we need to wrest control back from Brick 1 and inform it that it's no longer the "master". A really simple solution to this (at least conceptually) is to proxy all access requests from Brick 2 back to Brick 1. This satisfies our requirement that accesses are serialised across bricks and effectively there is still a "master" brick remaining in control. - We can see that this setup is conceptually similar to having a traditional lock server arbitrating brick access to a given file, but in example above we have implemented a distributed lock server, the lock server effectively becoming the same server as what we hope is the "hot server", so that we aren't incuring network latency to contact the lock server all the time. - A further improvement would clearly be to have some kind of process where the "master brick" can move about, ie in the case above if Client 3 starts to bash away at Brick 2 for File A, then Brick 2 is migrated to become the "master" and hold the lock, now any access through Brick 1 must effectively proxy requests back to Brick 2 or re-acquire it's lock (ie become the master)

OK, so the above is a very simple example of optimistic locking and could be trivially implemented using an external lock server which tracks which brick currently holds the lock for a given file (ie can read/write freely without first checking if other bricks have modified the file). A given brick which doesn't hold a lock on a file must first do kind of what it does already and contact the lock server to see if another brick holds the lock. If not it can acquire the lock itself. If the lock is held elsewhere we either need to break the lock (or proxy access requests to the server holding the lock).

Really this is not so different to what is there today, but it's simply an efficiency improvement because we don't need to touch *every* brick for *every* file access, instead we make some network requests on first access to a file and then can continue to touch that file for a period afterwards without needing further network access with other bricks

However, whilst some kind of implementation of the above could offer a huge performance speedup for many of the situations which come up on the mailing list, the issue is that the lock server becomes a) a bottleneck and b) point of failure. So the chain of thought almost certainly goes something like:

- Make the gluster bricks become the lock servers, ie they negotiate amongst themselves. Really this is roughly what happens right now, only it's on every access, rather than access being "sticky" once acquired - Now analyse all the corner cases that bricks go down holding locks, or get segmented while holding/acquiring locks and discover some tricky issues...

Paxos seems like a clever way of dealing with the locking going distributed, yet not necessarily having a 100% consistent view of who owns which lock. By introducing a voting method it can show robustness in the face of failed machines and new machines can be added without needing to store reliable state information (or at least this is true with the improvements described in the articles)

Does that make sense? Apologies if the above is long winded, but the point is really that the performance improvements come from pushing locks between bricks, and probably this is distinct from client level locking such as nfs/cifs/posix, etc locking

For advanced cluster filesystems such as GFS2, the general "optimistic locking" technique appears to show massive speed improvements (for many access patterns) and it's also likely to do so in Gluster. Really my original email jumped two steps and suggested an improved form of distributed locking, which itself could be used as the actual implementation, but other forms of distributed locking between bricks would be highly desirable also.

Thanks for listening

Ed W

reply via email to

[Prev in Thread] Current Thread [Next in Thread]