Problem with followVoice line

[Kim Bastin]

In the attached extract, I want a 'followVoice' line connecting beats 3 
and 4, and no others. Depending on where I place "\set followVoice = 
##t" I get either additional unwanted lines between other beats (bars 2 
and 3), or no line at all (bar 4). What's going on? Is this intended 
behaviour?
%charset=utf-8
\version "2.11.45"
\include "english.ly"

UpperStaff = \relative c' { 
  \time 2/2 
  \clef treble 
  s1*4
}

LowerStaff = \relative c {
  \time 2/2 \clef bass 
  r1 | 
        \change Staff = up r4 \set followVoice = ##t \stemDown cs' c'! \change 
Staff = down \stemNeutral d,,, \set followVoice = ##f | 
        \change Staff = up r \stemDown cs'' \set followVoice = ##t c'! \change 
Staff = down \stemNeutral d,,, \set followVoice = ##f | 
        \change Staff = up r \stemDown cs'' c'! \set followVoice = ##t \change 
Staff = down \stemNeutral d,,, \set followVoice = ##f | 
}

\new PianoStaff {
  <<
  \new Staff = "up"   { \UpperStaff }
  \new Staff = "down" { \LowerStaff }
  >>
}