#751 can't SSH into Duffy nodes anymore
Closed: Fixed with Explanation 2 years ago by arrfab. Opened 2 years ago by evgeni.

Ohai,

for some reason, we can't SSH into Duffy nodes anymore:

[foreman@slave01 ~]$ cico node get
Starting new HTTP connection (1): admin.ci.centos.org
Resetting dropped connection: admin.ci.centos.org
+---------+-----------+--------------+---------+------------+---------------+----------+--------+------+----------------+--------------+-----------+--------------+--------+
| host_id | hostname  | ip_address   | chassis | used_count | current_state | comment  | distro | rel  | centos_version | architecture | node_pool | console_port | flavor |
+---------+-----------+--------------+---------+------------+---------------+----------+--------+------+----------------+--------------+-----------+--------------+--------+
|     106 | n42.pufty | 172.19.3.106 | pufty   |       6077 | Deployed      | 0ea2469f | None   | None | 7              | x86_64       |         1 |         2410 | None   |
+---------+-----------+--------------+---------+------------+---------------+----------+--------+------+----------------+--------------+-----------+--------------+--------+

[Prod - slave01.ci.centos.org]
[foreman@slave01 ~]$ ssh -v root@172.19.3.106
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 55: Applying options for *
debug1: Connecting to 172.19.3.106 [172.19.3.106] port 22.
debug1: Connection established.
debug1: identity file /home/foreman/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/foreman/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/foreman/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/foreman/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/foreman/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/foreman/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/foreman/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/foreman/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.19.3.106:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:4Z7oHNEORQAiVq7l75oKRzqhto+dc7yznjqwVcSG4cg
debug1: Host '172.19.3.106' is known and matches the ECDSA host key.
debug1: Found key in /home/foreman/.ssh/known_hosts:3
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1013)

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1013)

debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/foreman/.ssh/id_rsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /home/foreman/.ssh/id_dsa
debug1: Trying private key: /home/foreman/.ssh/id_ecdsa
debug1: Trying private key: /home/foreman/.ssh/id_ed25519
debug1: Next authentication method: password
root@172.19.3.106's password: 

The last (known) successful run was this morning:
https://ci.centos.org/view/Foreman/job/foreman-pipeline-foreman-nightly-ubuntu2004-upgrade/
(started at 07:50 UTC)


Metadata Update from @arrfab:
- Issue assigned to arrfab

2 years ago

Metadata Update from @arrfab:
- Issue marked as depending on: #728
- Issue tagged with: centos-ci-infra, high-gain, medium-trouble

2 years ago

It seems related to #728 : I see that a new ssh keypair was created to reinitialize your jenkins pod on openshift (and so a new secret) but it was also updated (instead of being added) in the duffy DB, in the userkeys table.
As I see that you're still using the legacy jenkins env in parallel (just by looking at output) , that one suddenly couldn't connect anymore as new nodes are contextualized with new pub key.
You have now a copy of the needed priv key on that node too, so it should normally work now.

Can you give it a try, and close this ticket if that works for you ?

Metadata Update from @arrfab:
- Issue priority set to: Waiting on Reporter (was: Needs Review)

2 years ago

Yepp, this works, now, but I can't close the ticket for some reason.

Metadata Update from @arrfab:
- Issue unmarked as depending on: #728

2 years ago

Metadata Update from @arrfab:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata
Boards 1
CentOS CI Infra Status: Backlog