[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Gluster-devel] Client side AFR race conditions?

From: Martin Fick
Subject: Re: [Gluster-devel] Client side AFR race conditions?
Date: Tue, 6 May 2008 09:59:48 -0700 (PDT)

--- Anand Babu Periasamy <address@hidden> wrote:

> I really want to understand the issue and help you
> out. We always have heated discussions even in our 
> labs. We only take it positively :) Your feedback is

> very valuable to us.

No prob!  I appreciate it.

> Martin Fick wrote:
> In other words, what prevents conflicts when 
> client A & B both write to the same file?  Could 
> A's write to subvolume A succeed before B's write
> to subvolume A, and at the same time B's write to
> subvolume B succeed before A's write to subvolume
> B? 

OK, just for giggles I created a test script to
attempt to replicate this theoretical problem.  I was
in fact able to do so fairly easily, in fact, more
easily than I might have hoped!

What I wrote was a simple shell script which attempts
to write to a file at the same time from two different
processes.  Since this is actually hard to time with a
shell script, I did not expect very good results.  A C
program is likely to create much greater contention
possibilities.  The script simply writes once from
each process to a file and then increments the
filename counter and starts over with the next file. 
This should perform one 20 character write only from
each process to each file.  I have attached the test
script (stress).  I run the script with the following

  ./stress /mnt -d /mnt2 -c 100

This tells it to perform the test on 100 files and
specifies the 2 different mount points.

As for my glusterfs setup, I use two client afr mounts
on the same machine /mnt and /mnt2.  As noted above,
the script is run so that each process points to a
different client mount.  The server runs on the same
machine and maps the client subvolumes to /export/a
and /export/b, configs below.

Here are the split brain counts for a few successive
runs of 100 file writes: 
4,8,6,7,6,10,18,7,0,5,4,9,10,12.  Not good!  That
means that it is actually very easy to create this
problem, at least in the single digits percentage

I use this simple command to get those results:

  diff -q /export/a /export/b |wc -l

This command compares the two different subvolumes.  I
have looked at the file themselves, and yes they are
different, either lots of AAAs or lots of BBBs.

I hope this helps you debug this race condition a


I am using debian, packages:  

  glusterfs client and server 1.3.8-0pre
  fuse-utils and libfuse  2.7.2-glfs
  linux kernel 2.6.17-1-vserver-686 


volume a
  type storage/posix
  option directory /export/a

volume b
  type storage/posix
  option directory /export/b

volume afr
  type cluster/afr
  subvolumes a b

volume server
  type protocol/server
  option transport-type tcp/server
  option bind-address

  subvolumes a b

  option auth.ip.a.allow *
  option auth.ip.b.allow *


volume ca
  type protocol/client
  option transport-type tcp/client
  option remote-host
  option remote-subvolume a

volume cb
  type protocol/client
  option transport-type tcp/client
  option remote-host
  option remote-subvolume b

volume afr
  type cluster/afr
  subvolumes ca cb



trap clean HUP INT QUIT TERM

  kill $A $B 2>/dev/null
  wait $A $B
  rm "$A2B" "$B2A" "$P" 2>/dev/null
  exit 1

{  # dir val rec snd
  typeset  dir="$1" val="$2" rec="$3" snd="$4"
  typeset -i i=0

  while  read i < "$rec" ; do
    [ ! -z "$SW_P" ]  && read  < "$P"
    echo "$val" > "$dir/$i"  # we follow
    i=$(($i +1))

    [ ! -z "$SW_S" ]  && sleep 1

    [ ! -z "$C"  -a  $i -gt $C ] && return
    echo $i > "$snd"
    [ ! -z "$C"  -a  $i -eq $C ] && return
    echo "$val" > "$dir/$i"  # we initiate


COM=/tmp/$(basename $0)
mkfifo "$A2B" "$B2A"


while [ $# -gt 0 ] ; do
  case "$1" in
    -p|--prompt) SW_P="-p" ; mkfifo "$P" ;;
    -s|--sleep)  SW_S="-s" ;;

    -m|--manual) shift ; M="$1" ;;
    -d) shift ; D2="$1" ;;
    -c|--count) shift ; C="$1" ;;
    -a) shift ; VA="$1" ;;
    -b) shift ; VB="$1" ;;
shift ; done

writei "$D" "$VA" "$B2A$M" "$A2B" &

writei "$D2" "$VB" "$A2B$M" "$B2A" &

echo 0 > "$B2A"

if [ ! -z "$SW_P" ] ; then
  while true ; do  read ; echo > "$P" ; done

wait $A $B

Be a better friend, newshound, and 
know-it-all with Yahoo! Mobile.  Try it now.;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

reply via email to

[Prev in Thread] Current Thread [Next in Thread]