Ohai,
for some reason, we can't SSH into Duffy nodes anymore:
[foreman@slave01 ~]$ cico node get Starting new HTTP connection (1): admin.ci.centos.org Resetting dropped connection: admin.ci.centos.org +---------+-----------+--------------+---------+------------+---------------+----------+--------+------+----------------+--------------+-----------+--------------+--------+ | host_id | hostname | ip_address | chassis | used_count | current_state | comment | distro | rel | centos_version | architecture | node_pool | console_port | flavor | +---------+-----------+--------------+---------+------------+---------------+----------+--------+------+----------------+--------------+-----------+--------------+--------+ | 106 | n42.pufty | 172.19.3.106 | pufty | 6077 | Deployed | 0ea2469f | None | None | 7 | x86_64 | 1 | 2410 | None | +---------+-----------+--------------+---------+------------+---------------+----------+--------+------+----------------+--------------+-----------+--------------+--------+ [Prod - slave01.ci.centos.org] [foreman@slave01 ~]$ ssh -v root@172.19.3.106 OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 55: Applying options for * debug1: Connecting to 172.19.3.106 [172.19.3.106] port 22. debug1: Connection established. debug1: identity file /home/foreman/.ssh/id_rsa type 1 debug1: key_load_public: No such file or directory debug1: identity file /home/foreman/.ssh/id_rsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/foreman/.ssh/id_dsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/foreman/.ssh/id_dsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/foreman/.ssh/id_ecdsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/foreman/.ssh/id_ecdsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/foreman/.ssh/id_ed25519 type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/foreman/.ssh/id_ed25519-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_7.4 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4 debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000 debug1: Authenticating to 172.19.3.106:22 as 'root' debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: curve25519-sha256 debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none debug1: kex: curve25519-sha256 need=64 dh_need=64 debug1: kex: curve25519-sha256 need=64 dh_need=64 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: ecdsa-sha2-nistp256 SHA256:4Z7oHNEORQAiVq7l75oKRzqhto+dc7yznjqwVcSG4cg debug1: Host '172.19.3.106' is known and matches the ECDSA host key. debug1: Found key in /home/foreman/.ssh/known_hosts:3 debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey after 134217728 blocks debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512> debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password debug1: Next authentication method: gssapi-keyex debug1: No valid Key exchange context debug1: Next authentication method: gssapi-with-mic debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available (default cache: KEYRING:persistent:1013) debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available (default cache: KEYRING:persistent:1013) debug1: Next authentication method: publickey debug1: Offering RSA public key: /home/foreman/.ssh/id_rsa debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password debug1: Trying private key: /home/foreman/.ssh/id_dsa debug1: Trying private key: /home/foreman/.ssh/id_ecdsa debug1: Trying private key: /home/foreman/.ssh/id_ed25519 debug1: Next authentication method: password root@172.19.3.106's password:
The last (known) successful run was this morning: https://ci.centos.org/view/Foreman/job/foreman-pipeline-foreman-nightly-ubuntu2004-upgrade/ (started at 07:50 UTC)
Metadata Update from @arrfab: - Issue assigned to arrfab
Metadata Update from @arrfab: - Issue marked as depending on: #728 - Issue tagged with: centos-ci-infra, high-gain, medium-trouble
It seems related to #728 : I see that a new ssh keypair was created to reinitialize your jenkins pod on openshift (and so a new secret) but it was also updated (instead of being added) in the duffy DB, in the userkeys table. As I see that you're still using the legacy jenkins env in parallel (just by looking at output) , that one suddenly couldn't connect anymore as new nodes are contextualized with new pub key. You have now a copy of the needed priv key on that node too, so it should normally work now.
userkeys
Can you give it a try, and close this ticket if that works for you ?
Metadata Update from @arrfab: - Issue priority set to: Waiting on Reporter (was: Needs Review)
Yepp, this works, now, but I can't close the ticket for some reason.
Metadata Update from @arrfab: - Issue unmarked as depending on: #728
Metadata Update from @arrfab: - Issue close_status updated to: Fixed with Explanation - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.