After koschei-{backend,web}-stg machines were respawned as Fedora 27, I cannot ssh there anymore.
msimacek ~ $ ssh -v koschei-backend-stg OpenSSH_7.5p1, OpenSSL 1.1.0g-fips 2 Nov 2017 debug1: Reading configuration data /home/msimacek/.ssh/config debug1: /home/msimacek/.ssh/config line 57: Applying options for koschei-backend-stg debug1: Reading configuration data /etc/ssh/ssh_config debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for * debug1: Executing proxy command: exec ssh -W koschei-backend01.stg.phx2.fedoraproject.org:22 bastion.fedoraproject.org debug1: permanently_drop_suid: 1000 debug1: identity file /home/msimacek/.ssh/id_rsa type 1 debug1: key_load_public: No such file or directory debug1: identity file /home/msimacek/.ssh/id_rsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/msimacek/.ssh/id_dsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/msimacek/.ssh/id_dsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/msimacek/.ssh/id_ecdsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/msimacek/.ssh/id_ecdsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/msimacek/.ssh/id_ed25519 type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/msimacek/.ssh/id_ed25519-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_7.5 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.5 debug1: match: OpenSSH_7.5 pat OpenSSH* compat 0x04000000 debug1: Authenticating to koschei-backend01.stg.phx2.fedoraproject.org:22 as 'msimacek' debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: curve25519-sha256@libssh.org debug1: kex: host key algorithm: ssh-rsa-cert-v01@openssh.com debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32 debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host certificate: ssh-rsa-cert-v01@openssh.com SHA256:mbRtw8J0G9UEdav2oI9C97XFzPF6h+DJuEa86bL1lHw, serial 1512067161 ID "koschei-backend01.stg.phx2.fedoraproject.org" CA ssh-rsa SHA256:pmw8O+j5VVfgpVIjMvWG+u7DizO2dyzrvxHWrLUg02s valid from 2017-11-30T19:39:21 to 2018-11-29T20:39:21 debug1: No matching CA found. Retry with plain key debug1: Host 'koschei-backend01.stg.phx2.fedoraproject.org' is known and matches the RSA host key. debug1: Found key in /home/msimacek/.ssh/known_hosts:60 debug1: rekey after 4294967296 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey after 4294967296 blocks debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,null> debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Offering RSA public key: /home/msimacek/.ssh/id_rsa debug1: Server accepts key: pkalg rsa-sha2-512 blen 279 Authentication failed.
It used to work and connecting to prod still works, so it shouldn't be misconfiguration on my side.
CC @mizdebsk
Oh yeah, this is due to: https://bugzilla.redhat.com/show_bug.cgi?id=1513704 (selinux policy issue with f27 and nssdb).
I have put them both in permissve mode now, so you should be able to login.
Hopefully they will fix the above soon...
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
It started to happen again after reboot (part of phx2 maintenance), so I've made the SELinux workaround persistent. I hope that's fine.
Thats fine. If we don't get a real fix soon we can push our own selinux policy for it.