fedora-infrastructure

#10397 basessh role issues with signed host-keys after birthday=true
Closed: Fixed by praiskup. Opened by praiskup.

Closed: Fixed
Reopen Issue

When running copr-be-dev playbook, we are constantly asked by ssh:

The authenticity of host 'copr-be-dev.aws.fedoraproject.org (18.208.10.131)' can't be established.                                                                                                                                          
ED25519 key fingerprint is SHA256:xhS550Upa1Zf7ECMceq87+rZfkB6FTGYMKpTKRekEEw.                                                                                                                                                              
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

I believe this is the reason:

https://pagure.io/fedora-infra/ansible/blob/19b8f82f5b5552ee59eb539f2187639e65925c5e/f/roles/basessh/tasks/main.yml#_250-254

Which expects that the birthday=True actions like these work fine, and no host is needed:
https://pagure.io/fedora-infra/ansible/blob/19b8f82f5b5552ee59eb539f2187639e65925c5e/f/roles/basessh/tasks/main.yml#_200-212

This doesn't have the effect though, the signed host key is not accepted.

Per our chat with @mizdebsk, we aren't sure what is going on here, perhaps it
has something to do with stage/production nunaces...:

<praiskup> there are actually two weird things ...
<praiskup> 1) the 'rsa' keys (as they used to work) don't work now ... the playbook refuses to continue when we add non edc* key
<praiskup> 2) when edc* key is used temporarily added, then the new generated key is not accepted by batcave
<mizdebsk> i see what is going on
<mizdebsk> praiskup, FrostyX: copr-be-dev.aws host is staging host, it has has env=staging, so key is signed with staging CA
<mizdebsk> but its domain name does imply it is a staging host
<mizdebsk> ssh client on batcave does not authorize copr-be-dev.aws key because the CA it was signed with is not authorized for non-staging hosts, which it believes the host is
<mizdebsk> or not...; that was the case on my local system, but batcave seems to use different ssh_known_hosts
<mizdebsk> so after all i don't know what is wrong and i don't have more time for debugging this

cc @schlupov, @frostyx

We had to work-around the 1) problem and switch to ssh-ed25519 host keys,
otherwise the birthday=True doesn't work at all against Fedora 35. Not sure
what is going on here -- but seems like the remote host prefers ed* keys over
rsa, and it is not enough to have the rsa key in known_hosts (even though both
should be accepted there).

But the 2) problem (not sure if it is the direct effect of 1) is still not
resolved. We add and remove the known_hosts entries with each playbook run.

So, this was caused by the basessh playbook being run when the hostname was copr-be-dev-temp.aws.fedoraproject.org, so it signed certs to that effect. Then later the name changed, but the certs still had the old name.

I did a 'rm -f /etc/ssh/*cert.pub' on the host and re-ran the basessh role to sign the current certs.

I think there's a few ways to fix this:

  • We could add a thing in basessh to also rm any old certs that are on the host in basessh
  • as part of bringing up a -temp host rm those cert files
  • just call it good and add to workflow to remove those certs at the right time.

Thoughts?

We could add a thing in basessh to also rm any old certs that are on the host in basessh

I guess this way too specific case so it doesn't make sense to touch the basessh role.

as part of bringing up a -temp host rm those cert files

I think we can do this ourselves :-) Thank you for taking a look.

just call it good and add to workflow to remove those certs at the right time.

Good to know there's a manual step, but I think we'll try to automatize this so we
don't have to think about it (and document it).

Thanks!

Metadata Update from @praiskup:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Implemented with:
https://pagure.io/fedora-infra/ansible/blob/053b703e4b68f7c8f30deb1d41995fd018f54366/f/inventory/group_vars/copr_all_instances_aws#_10-14
Metadata
None
None
None
None
Lowest
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.