Re: cfservd thrushes, nodes fail to get anything

[Yaroslav Halchenko]

I've found the reason and probably that would be benefitial to adjust
cfservd to don't get into such situation again:

I had a leftover file 
/tmp/__db.testDATABASEcache

so strace revealed me infinite loop of

28731 stat64("/tmp/testDATABASEcache", 0xb7c57350) = -1 ENOENT (No such file or 
directory)
28731 open("/tmp/__db.testDATABASEcache", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 
0644) = -1 EEXIST (File exists)
28731 open("/tmp/__db.testDATABASEcache", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 
0644) = -1 EEXIST (File exists)
28731 open("/tmp/__db.testDATABASEcache", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 
0644) = -1 EEXIST (File exists)


Version installed  (debian unstable)
cfengine2              2.1.14-1 

--
Yarik


On Sat, May 07, 2005 at 11:50:59AM -0400, Yaroslav Halchenko wrote:
> Dear All,

> Yesterday one of the users filled up /tmp on a main node with junk and it 
> rendered
> cfengine unusable. First it reported

> daemon.log:May  6 21:11:23 ravana cfservd[16657]:  Couldn't open checksum 
> database /tmp/testDATABASEcache 
> daemon.log:May  6 21:11:23 ravana cfservd[16657]:  db_open: No space left on 
> device

> and seems after that whenever any node connects to it - cfservd
> becomes extremely busy and then finally fails with next message being
> reported by the nodes

> cfengine:node20: Received signal 13 (SIGPIPE) while doing [no_active_lock]
> cfengine:node20: Logical start time Fri May  6 23:51:10 2005
> cfengine:node20: This sub-task started really at Fri May  6 23:51:10 2005

> or actually now for some reason without a node name

> cfengine:: Received signal 13 (SIGPIPE) while doing [pre-lock-state]
> cfengine:: Logical start time Sat May  7 11:00:33 2005
> cfengine:: This sub-task started really at Sat May  7 11:00:33 2005

> and then another stating refusal for copying

> cfengine:: Transmission refused or failed statting 
> /etc/cfengine/inputs/CVS/Repository
> Got:
> cfengine:: Received signal 13 (SIGPIPE) while doing 
> [lock.cfagent_conf.node2.copy.copy_3343]
> cfengine:: Logical start time Sat May  7 04:30:29 2005
> cfengine:: This sub-task started really at Sat May  7 04:30:29 2005

> I've tried restarting cfengine parts on both ends - doesn't help.
> running cfservd with -d2 gave next: while trying to run update script
> (copy /etc/cfengine/input files across the nodes into /etc/cfengine)

> ----------------------------------------
> ...
> Access privileges - match found
> cfservd: Host node2.ravana.rutgers.edu granted access to 
> /etc/cfengine/inputs/CVS/Root
> Clocks were off by 0
> StatFile(/etc/cfengine/inputs/CVS/Root)
> OK: type=0
>  mode=644
>  lmode=0
>  uid=0
>  gid=0
>  size=10
>  atime=1115477605
>  mtime=1067285389
> Transaction Send[t 65][Packed text]
> Attempting to send 73 bytes
> SendSocketStream, sent 73
> Transaction Send[t 3][Packed text]
> Attempting to send 11 bytes
> SendSocketStream, sent 11
> RecvSocketStream(8)
>     (Concatenated 8 from stream)
> Transaction Receive [t 51][]
> RecvSocketStream(51)
>     (Concatenated 51 from stream)
> Received: [MD5 /etc/cfengine/inputs/CVS/Root] on socket 5
> CompareLocalChecksums(/etc/cfengine/inputs/CVS/Root,MD5=05e8d918529f204488a626792c4f8a6f)
> ChecksumChanged: key /etc/cfengine/inputs/CVS/Root with data 
> MD5=05e8d918529f204488a626792c4f8a6f

> <At this point it stalls for a minute or two although cfservd running
> busy>

> IPV4 address
> sockaddr_ntop(10.0.0.2)
> Obtained IP address of 10.0.0.2 on socket 7 from accept

> FuzzyItemIn(LIST,10.0.0.2)
> Purging Old Connections...
> Done purging

> FuzzyItemIn(LIST,10.0.0.2)
> cfservd: Denying repeated connection from 10.0.0.2
> ----------------------------------------

> from client (cfagent) side it looks like

> ----------------------------------------
> Compare binary sums on ravana:/etc/cfengine/inputs/CVS/Root & 
> /var/lib/cfengine2/inputs/CVS/Root
> Using network md5 checksum instead
> ChecksumFile(m,/var/lib/cfengine2/inputs/CVS/Root)
> Send digest of /var/lib/cfengine2/inputs/CVS/Root to server, 
> MD5=05e8d918529f204488a626792c4f8a6f
> Transaction Send[t 51][Packed text]
> Attempting to send 59 bytes
> SendSocketStream, sent 59
> RecvSocketStream(8)
> <STALLS HERE and I got bored waiting till it dies... may be it never
> dies this time>

> ----------------------------------------

> So here are the questions:

> 1. how to fix current situation?  
>    clearly there is something broken in a current state, so may be I can
>    clean out cfengine state so as to start from a clean one - I wouldn't
>    mind if it takes longer to run for the first time ;-) Sure I can
>    completely reinstall and then it should work I believe but...


> 2. what would be a nice policy to enforce over /tmp so I don't
> remove anything valuable (like ssh-agent sockets and some other staff
> opened by running programs). I'm thinking about smth like files and
> directories large in size should be forbidden (>1M) if they are older
> than an hour. I'm not sure if I can discard data solely on age, so
> age+size sounds good to me..
-- 
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
        101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07105
Student  Ph.D. @ CS Dept. NJIT