Re: [Gluster-devel] GlusterFS 1.3 mainlin 2.5-patch-317 crashes when second afr-node goes down

[Urban Loesch]

Hi Avati,

still the Problem when I shut down the second server.
Here the Log
2007-07-20 18:51:15 C [tcp.c:81:tcp_disconnect] server: connection 
disconnected
2007-07-20 18:51:15 E [client-protocol.c:336:client_protocol_xfer] 
data1-gluster2-ds: transport_submit failed
2007-07-20 18:51:15 W [client-protocol.c:4196:client_protocol_cleanup] 
data1-gluster2-ds: cleaning up state in transport object 0x8052bd8
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] 
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: 
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] 
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: 
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] 
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: 
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] 
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: 
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] 
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: 
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] 
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: 
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] 
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: 
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] 
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: 
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] 
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: 
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 C [common-utils.c:210:gf_print_trace] 
debug-backtrace: Got signal (11), printing backtrace
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] 
debug-backtrace: /usr/lib/libglusterfs.so.0(gf_print_trace+0x2d) 
[0xb7edc32d]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] 
debug-backtrace: /lib/i686/cmov/libc.so.6 [0xb7d98208]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] 
debug-backtrace: /usr/lib/libglusterfs.so.0(dict_ref+0x29) [0xb7ed5989]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] 
debug-backtrace: 
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so [0xb752b018]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] 
debug-backtrace: 
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so [0xb752c185]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] 
debug-backtrace: 
/usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so [0xb7538b1f]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] 
debug-backtrace: /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so 
[0xb753d1d5]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] 
debug-backtrace: /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so 
[0xb754bef3]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] 
debug-backtrace: 
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so [0xb756203f]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] 
debug-backtrace: 
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so(notify+0x46b) 
[0xb756072b]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] 
debug-backtrace: /usr/lib/libglusterfs.so.0(transport_notify+0x37) 
[0xb7edd837]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] 
debug-backtrace: /usr/lib/libglusterfs.so.0(sys_epoll_iteration+0xd9) 
[0xb7ede359]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] 
debug-backtrace: /usr/lib/libglusterfs.so.0(poll_iteration+0x1d) 
[0xb7edd90d]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] 
debug-backtrace: [glusterfsd] [0x80493ee]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] 
debug-backtrace: /lib/i686/cmov/libc.so.6(__libc_start_main+0xe0) 
[0xb7d84030]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] 
debug-backtrace: [glusterfsd] [0x8048d81]

And the backtrace:
gluster1:/# gdb glusterfsd -c core.15573
GNU gdb 6.4.90-debian
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i486-linux-gnu"...(no debugging symbols found)
Using host libthread_db library "/lib/i686/cmov/libthread_db.so.1".

Reading symbols from /usr/lib/libglusterfs.so.0...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/libglusterfs.so.0
Reading symbols from /lib/i686/cmov/libdl.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib/i686/cmov/libdl.so.2
Reading symbols from /lib/i686/cmov/libpthread.so.0...(no debugging 
symbols found)...done.
Loaded symbols for /lib/i686/cmov/libpthread.so.0
Reading symbols from /lib/i686/cmov/libc.so.6...(no debugging symbols 
found)...done.
Loaded symbols for /lib/i686/cmov/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from 
/usr/lib/glusterfs/1.3.tla-322/xlator/storage/posix.so...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/xlator/storage/posix.so
Reading symbols from 
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so
Reading symbols from 
/usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so
Reading symbols from 
/usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so
Reading symbols from 
/usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so...(no 
debugging symbols found)...done.
Loaded symbols for 
/usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so
Reading symbols from 
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so
Reading symbols from 
/usr/lib/glusterfs/1.3.tla-322/transport/tcp/client.so...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/transport/tcp/client.so
Reading symbols from 
/usr/lib/glusterfs/1.3.tla-322/scheduler/rr.so...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/scheduler/rr.so
Reading symbols from 
/usr/lib/glusterfs/1.3.tla-322/transport/tcp/server.so...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/transport/tcp/server.so
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols 
found)...done.
Loaded symbols for /lib/libgcc_s.so.1

Core was generated by 
`[glusterfsd]                                                                  
'.
Program terminated with signal 11, Segmentation fault.
#0  0xb7ebc430 in pthread_mutex_lock () from /lib/i686/cmov/libpthread.so.0
(gdb) backtrace
#0  0xb7ebc430 in pthread_mutex_lock () from /lib/i686/cmov/libpthread.so.0
#1  0xb7ed5989 in dict_ref () from /usr/lib/libglusterfs.so.0
#2  0xb752b018 in fini () from 
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so
#3  0xb752c185 in mop_lock () from 
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so
#4  0xb7538b1f in ?? () from 
/usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so
#5  0x081f1924 in ?? ()
#6  0x081e2a08 in ?? ()
#7  0x08051970 in ?? ()
#8  0x00020000 in ?? ()
#9  0x00000022 in ?? ()
#10 0xbfadbdf0 in ?? ()
#11 0xbfadbd48 in ?? ()
#12 0xb753d1d5 in ?? () from 
/usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so
#13 0x081e2a08 in ?? ()
#14 0x081ec2c8 in ?? ()
#15 0x080518f8 in ?? ()
#16 0x00020000 in ?? ()
#17 0x00000022 in ?? ()
#18 0xbfadbdf0 in ?? ()
#19 0xbfadbd88 in ?? ()
#20 0xb754bef3 in fini () from 
/usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so
Previous frame inner to this frame (corrupt stack?)

Thanks and regards
Urban