Re: [lmi] Conflicting ssh keys [Was: Using git to manage CVS webpages repository]

[Greg Chicares]

On 2018-01-17 23:19, Vadim Zeitlin wrote:
> On Wed, 17 Jan 2018 22:40:22 +0000 Greg Chicares <address@hidden> wrote:
[...]
> But ssh key management is the next level of complexity, so I'll just do my
> best...

Thanks: with your help I believe I've understood and solved the problem.
Feel free to skip everything below, which serves only to document the
steps I went through in case I ever need to repeat them.

> GC> On 2018-01-08 17:52, Greg Chicares wrote:
[...]
> GC> /opt/lmi/src/lmi[0]$git push
> GC> Warning: the RSA host key for 'git.sv.gnu.org' differs from the key for 
> the IP address '208.118.235.201'
> GC> Offending key for IP in /home/greg/.ssh/known_hosts:7
> GC> Matching host key in /home/greg/.ssh/known_hosts:1
> GC> Are you sure you want to continue connecting (yes/no)? n

First of all, I had misinterpreted the diagnostics, focusing on the word
"offending" and guessing that it meant something like "invalid". Rereading
the "warning" carefully, I see that it really means that git.sv.gnu.org
resolves to 208.118.235.201, but they have distinct keys that are not
identical. (That's hard to see in a hashed 'known_hosts'--see below.)
Presumably this came about because I've applied Trust On First Use
(after validating fingerprints), so keys that might have been merged
were not. But you do merge these alias hostnames, manually:

>  My first problem is that I just can't reproduce this at all. Maybe it's
> due to a specific format of my ~/.ssh/known_hosts key which I (sometimes)
> edit manually to make it more compact and share it more easily between
> different machines (which I do using Git, of course, BTW). For me it looks
> like this:
> 
>       % fgrep sv.gnu.org ~/.ssh/known_hosts
>       
> git.sv.gnu.org,svn.savannah.nongnu.org,savannah.nongnu.org,cvs.sv.gnu.org,svn.sv.gnu.org,208.118.235.201
>  ssh-rsa 
> AAAAB3NzaC1yc2EAAAABIwAAAIEAzFQovi+67xa+wymRz9u3plx0ntQnELBoNU4SCl3RkwSFZkrZsRTC0fTpOKatQNs1r/BLFoVt21oVFwIXVevGQwB+Lf0Z+5w9qwVAQNu/YUAFHBPTqBze4wYK/gSWqQOLoj7rOhZk0xtAS6USqcfKdzMdRWgeuZ550P6gSzEHfv0=
> 
> And connecting to both {git,cvs}.sv.gnu.org using either OpenSSL 6.7 (from
> Jessie) or 7.6 (from Sid) works just fine, without any warnings. Just to be
> clear, by "just fine" I mean that I can do "ssh -v" to either of these
> hosts and see the login screen followed by an error about not being allowed
> shell acess to the server, but this still means that there are no problems
> with key verification anywhere.

Copying the screen for attempts to 'ssh -v' into both equivalent hostnames
into files, and applying 'diff -U0', yields:

@@ -1 +1 @@
-/home/greg[0]$ssh -v git.sv.gnu.org
+/home/greg[255]$ssh -v 208.118.235.201
@@ -6 +6 @@
-debug1: Connecting to git.sv.gnu.org [208.118.235.201] port 22.
+debug1: Connecting to 208.118.235.201 [208.118.235.201] port 22.
@@ -26 +26 @@
-debug1: Authenticating to git.sv.gnu.org:22 as REDACTED
+debug1: Authenticating to 208.118.235.201:22 as REDACTED
@@ -35,2 +35,2 @@
-debug1: Host 'git.sv.gnu.org' is known and matches the RSA host key.
-debug1: Found key in /home/greg/.ssh/known_hosts:1
+debug1: Host '208.118.235.201' is known and matches the RSA host key.
+debug1: Found key in /home/greg/.ssh/known_hosts:6
@@ -51 +51 @@
address@hidden's password: 
address@hidden's password: 

which clearly supports the analysis above. Apparently the keys are
looked up according to the literal way the host is specified, and
only later does ssh discern that two distinct keys of different
encryption types were looked up for equivalent hosts, leading to
the "Warning" displayed above with 'git push'.

>  So maybe you could just copy the line above to your ~/.ssh/known_hosts
> (verifying that the keys match, of course), delete all the other entries
> for these hosts and this would solve the problem? Of course, you'd still be
> using RSA key and not ECDSA one.

Maybe I can accomplish something like that, but using the ECDSA password,
as follows:

rm ~/.ssh/known_hosts*
ssh -v 208.118.235.201

modify configuration:
/opt/lmi/src/lmi[0]$cat ~/.ssh/config     
Protocol 2
HashKnownHosts no

The authenticity of host '208.118.235.201 (208.118.235.201)' can't be 
established.
ECDSA key fingerprint is SHA256:qRLLJ4w/GAeiDyYnbx4yWJbZXwGiYYxgNty7lAfUyuM.
Are you sure you want to continue connecting (yes/no)? yes
  [I verified that fingerprint]
Warning: Permanently added '208.118.235.201' (ECDSA) to the list of known hosts.

Now
  ssh -vvvvv 208.118.235.201
works as expected. So far so good.

Then I edited the known_hosts file, replacing '208.118.235.201' with:

  
git.sv.gnu.org,svn.savannah.nongnu.org,savannah.nongnu.org,cvs.sv.gnu.org,svn.sv.gnu.org,208.118.235.201

as in your file, above; but when I tried
  ssh -vvvvv 208.118.235.201
  ssh -v git.sv.gnu.org
repeatedly, I was refused...even when, instead of running in my
chroot as above (where I had modified ~/.ssh), I tried those
commands in my host (non-chroot) system with untouched ~/.ssh:

/home/greg[0]$ssh -v 208.118.235.201
OpenSSH_7.4p1 Debian-10+deb9u2, OpenSSL 1.0.2l  25 May 2017
debug1: Reading configuration data /home/greg/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 208.118.235.201 [208.118.235.201] port 22.
debug1: connect to address 208.118.235.201 port 22: Connection refused
ssh: connect to host 208.118.235.201 port 22: Connection refused

I gather that this rules out a problem on my machine and implies
that I had tried to ssh into gnu.org too many times and was blocked.

An hour later, I tried again, and everything seemed to work (as I
had hoped) in a simpler way (which surprised me): as above, I did
  rm ~/.ssh/known_hosts*
  ssh -v 208.118.235.201
and accepted the key (verifying its fingerprint); then I tried
'git push', and it prompted me to Trust On First Use again, but
subsequent use of the same command elicits no diagnostic:

/opt/lmi/src/lmi[0]$git push
Enter passphrase for key '/home/greg/.ssh/id_rsa': 
Everything up-to-date

Then I did:
  ssh -v git.sv.gnu.org
and that worked as expected, too. Now 'known_hosts' contains only:

git.sv.gnu.org,208.118.235.201 ecdsa-sha2-nistp256 
AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9c1Z2f4OHxymvLxqxQ/hY1g0ol0/iiXUrVFGZBBq4h5gD05c7Gw9rRrcrvF9XvumBvOghOQzDSZZLRWvFGocA=

As if by magic, the aliases have been combined on one line. Perhaps
ssh combines them iff it's configured with "HashKnownHosts no".
This seems so magical that, although I was very careful about
copying and pasting commands here before executing each one, maybe
I forgot some step where I edited that file. Or maybe ssh consulted
the backup copy that it makes in known_hosts.old, found a single
combined line there, and moved it into known_hosts...perhaps only
after I had gotten locked out for issuing 'ssh -v' too often.

But no, I must have made some mistake. In order to make sure that
CVS will work next January when I might have to use it again, I did:
  ssh -v cvs.sv.gnu.org
and trusted the key when its fingerprint matched, and now I have
two lines in known_hosts:

git.sv.gnu.org,208.118.235.201 ecdsa-sha2-nistp256 
AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9c1Z2f4OHxymvLxqxQ/hY1g0ol0/iiXUrVFGZBBq4h5gD05c7Gw9rRrcrvF9XvumBvOghOQzDSZZLRWvFGocA=
cvs.sv.gnu.org ecdsa-sha2-nistp256 
AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9c1Z2f4OHxymvLxqxQ/hY1g0ol0/iiXUrVFGZBBq4h5gD05c7Gw9rRrcrvF9XvumBvOghOQzDSZZLRWvFGocA=

I hesitate to repeat these experiments because it was kind of scary
to get locked out without understanding why. But let's back that
file up carefully and edit it to combine all three aliases on one
single line...okay, that worked this time:

/opt/lmi/src/lmi[0]$git push
Enter passphrase for key '/home/greg/.ssh/id_rsa': 
Everything up-to-date

ssh -v cvs.sv.gnu.org
...
debug1: Server host key: ecdsa-sha2-nistp256 
SHA256:qRLLJ4w/GAeiDyYnbx4yWJbZXwGiYYxgNty7lAfUyuM
debug1: Host 'cvs.sv.gnu.org' is known and matches the ECDSA host key.
debug1: Found key in /home/greg/.ssh/known_hosts:1

> GC> The diagnostics I've copied above seem to me to suggest that gnu.org's
> GC> distinct VCS and web servers have incompatible ssh-key requirements.
> 
>  But they're not distinct, this is one and the same machine,

I'm sure I've read in the past that they're different physical boxes.
Either they've been migrated onto the same box, or the IP address is
shared by two boxes. But that turns out to be of no consequence here.

> so the only
> explanation for being able to connect to it under one name and not the
> other I see is having wrong/outdated entries in ~/.ssh/known_hosts -- yet
> you don't have any.

My hypothesis is that I formerly had a valid RSA key for one host alias,
and a valid ECDSA key for another distinct-but-equivalent alias; and
ssh was just warning me that such a situation is weird but workable.
That warning was just a nuisance, which could be considered bogus or
not depending on POV...

>  So my only remaining hypothesis is that there is a bug in this OpenSSH
> version which results in a bogus warning when a host has both RSA and ECDSA
> key and that this bug was fixed in 7.6 that I'm testing with. If this is
> indeed the case, you can either try updating to 7.6 from testing or
> removing the RSA key because OpenSSH certainly shouldn't complain about
> mismatching keys if there is only one of them.

...but now I have in essence removed the RSA key and everything works
fine as far as I can tell.

>  BTW, I hate this host obfuscation so much that I have "HashKnownHosts no"
> in my ~/.ssh/config and the readability of the lines above does nothing to
> change my mind.

Thanks, I've adopted that suggestion. The only keys in my known_hosts
file are for gnu.org, as anyone reading this public mailing list might
easily have guessed anyway, so hashing them accomplishes nothing.