Re: [Fab-user] EC2 host keys

[Christian Vest Hansen]

On Sat, May 9, 2009 at 7:42 PM, Patrick J McNerthney
<address@hidden> wrote:
> Jeff,
>
> Thanks for the response.
>
> Sorry, I did use IP address in my original message, but probably should have
> used DNS name.  It actually does not matter, the problem exists for both.
>  You are correct that I could disable the additional checking of IP address
> conflicts, but I have the same exact problem with the DNS name.
>
> The problem is that there is an entry in my known_hosts file for a
> particular DNS name that has a key that does not match the DNS name being
> used.  This is caused by the recycling of both DNS names and IP address by
> Amazon's EC2.

Out of curiosity: do the servers get new IPs while they are running,
or do the recycling only happen when servers are rebooted?

Also, is there a way to partition the servers? I mean, you may have
two kinds of servers, like front-end and back-end, and then you say
all front-ends are in this IP range and all back-ends are in that
range. Is that possible?

Where this all leads is; I'm wondering if it might make sense if all
servers of a kind shared the same key pair, so it wouldn't make any
difference if they swapped IP addresses amongst one another.

This way, I think, the servers could swap IPs all they want and we
could still connect to them in a manner that is both reliable and
resistant to DNS cache poisoning (once their fingerprints are in
known_hosts, that is).


>
> To replicate, I did the following:
>
> 1.  removed both my known_hosts file and my .ssh config file.
> 2.  ssh'ed into ServerA, then exited.  There are now two lines in the
> known_hosts file, one for the dns name and one for the ip address.
> 3.  ssh'ed into ServerB, then exited.  There are now four lines in the
> known_hosts file.
> 4.  modified the known_hosts file so that ServerA's entries contains
> ServerB's key.
> 5.  attempted to ssh to ServerA and was rejected because the host key has
> changed.
> 6.  attempted to ssh to ServerA with "StrictHostKeyChecking no", this
> worked.
>
> So it is with this setup, where the known_hosts file contains the wrong key
> for a server, I am unable to use Fabric against such a server.  What I want
> to do is replicate the effect of "StrictHostKeyChecking no" in Fabric.
>
> In this scenario, this is a case of an invalid server key, not a missing
> server key.  The missing host key policy never gets called.  This is why the
> env.reject_unknown_keys currently has no effect.
>
> Any clearer?
>
> Pat
>
>
> Jeff Forcier wrote:
>>
>> Hi Pat,
>>
>> First, I think this partially falls under an existing TODO item, which
>> I plan to have in place for 1.0 and hopefully 0.9: honoring any
>> .ssh/config options that we have functionality for. In this case,
>> Fabric would probably check your StrictHostKeyChecking option and use
>> that to override the default value of env.reject_unknown_keys.
>>
>> Secondly, I've read over your use case and I may be missing something:
>> as things currently are, why isn't setting env.reject_unknown_keys to
>> False good enough? Simply loading the host key list is not what drives
>> the reject/don't-reject decision: that's driven by the policy given to
>> set_missing_host_key_policy.
>>
>> I just double checked this by tweaking a (static) IP in my known_hosts
>> file so that it was incorrect, then ensuring my fabfile had
>> env.reject_unknown_keys = False. When connecting to that server, the
>> "new" IP was added as a new entry to my known_hosts and the connection
>> was created without issue.
>>
>> Is the problem that you don't want the new IPs added to your host list, or
>> what?
>>
>> Best,
>> Jeff
>>
>> P.S. In checking "man ssh_config" I found that there's an even more
>> specific setting, CheckHostIP, which sounds like it fits your
>> situation better than StrictHostKeyChecking. Unfortunately, Paramiko
>> doesn't appear to support that level of granularity, so we're out of
>> luck with that for now. Wanted to mention it anyways, though, in case
>> you weren't aware of it, for non-Fabric use.
>>
>> On Sat, May 9, 2009 at 11:10 AM, Patrick J McNerthney
>> <address@hidden> wrote:
>>
>>>
>>> I have an issue with Amazon EC2 instances where ssh host keys have been
>>> saved in .ssh/known_hosts but are incompatible with an EC2 instance ip
>>> address.  This occurs when the ip address has been reassigned to a new
>>> EC2
>>> instance.  So the basic sequence of events are:
>>>
>>> o Start an EC2 instance which is assigned an ip address.
>>> o ssh to that ip address and that server's ssh key is associated with
>>> that
>>> ip address in the known_hosts file.
>>> o Terminate that EC2 instance.
>>> o A new EC2 instance is started and it happens to get assigned the same
>>> ip
>>> address.
>>>
>>> At this point, if I first ssh to it, I have ssh configured with
>>> StrictHostKeyChecking set to no, so ssh will emit a warning about this ip
>>> address having a new key, but still allows me to continue.
>>>
>>> However, if at this point I try to use Fabric to execute some commands,
>>> it
>>> always will fail.  This is because the SSHClient.load_system_host_keys is
>>> always called, causing the connection to fail if there is an
>>> incompatibility
>>> between the ip address and the server key.
>>>
>>> I have addressed this in my own fork here:
>>>
>>>  
>>> http://github.com/iciclespider/fabric/commit/08ad1c491e5643990c2a35e865784d2b61aa742f
>>>
>>> What this does is replace this:
>>>
>>>  client.load_system_host_keys()
>>>  if not env.reject_unknown_keys:
>>>       client.set_missing_host_key_policy(ssh.AutoAddPolicy())
>>>
>>> with this:
>>>
>>>  if env.reject_unknown_keys:
>>>      client.load_system_host_keys()
>>>  else:
>>>       client.set_missing_host_key_policy(ssh.AutoAddPolicy())
>>>
>>> I also considered using another env setting value to control this, but my
>>> conclusion that this behavior is in fact in line with the implied
>>> behavior
>>> of the "reject_unknown_keys" name.  In other words, the list of known
>>> keys
>>> should only be loaded if the intention is to reject those keys that are
>>> not
>>> known.
>>>
>>> Pat McNerthney
>>> ClearPoint Metrics, Inc.
>>>
>>>
>>> _______________________________________________
>>> Fab-user mailing list
>>> address@hidden
>>> http://lists.nongnu.org/mailman/listinfo/fab-user
>>>
>>>
>
>
>
> _______________________________________________
> Fab-user mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/fab-user
>



-- 
Venlig hilsen / Kind regards,
Christian Vest Hansen.