As CentOS and Fedora are using shared IPA backend for authentication, I'd request that nothing touching enrolled centos machines in IPA would be done (manually or through scripts) I had to just waste my time this morning investigating why ipsilon (https://id.stg.centos.org) wasn't allowing anyone to auth (and so no openidc for services using our ipsilon instance)
Someone (who ? or a script ?) removed ipsilon.stg.iad2.centos.org from the ipsilon HBAC rule, denying so all auth requests .
ipsilon.stg.iad2.centos.org
Can you identify the root cause and ensure it wouldn't happen again please ?
Thanks a lot
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: Needs investigation, high-gain, ops
seems related to https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_92 ...
Metadata Update from @arrfab: - Issue untagged with: Needs investigation, high-gain, ops - Issue priority set to: Needs Review (was: Waiting on Assignee)
The change you are referring to happened 4 years ago. So I assume that didn't caused the machine to be removed.
Metadata Update from @kevin: - Issue assigned to kevin
It's actually https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_101
It was using the wrong hostname... but that was set in 2021? b8e6754f97c (Aurélien Bompard 2021-03-22 17:07:45 +0100 101) host: "{{ (env == 'production')|ternary('ipsilon.iad2.centos.org', 'centos-ipa-client02.stg.iad2.fedoraproject.org') }}"
anyhow, I changed it to ipsilon.stg.iad2.centos.org
If you can confirm it's fixed / working?
forgot to give feedback (PTO) but yes, now working again ... Is there a way to ensure that ansible would not remove hosts not managed by itself ? Otherwise, just ensure that it's documented somewhere as that means that on my centos ansible infra side I'll never be autonomous to deploy/enroll a machine as then Fedora ansible would remove it :/
Well, I mean we could remove that task from our playbooks and you could depend on yourself to create that ?
Otherwise ansible is going to setup the thing thats defined.
Do you want us to remove that setup from our side?
@kevin : even if you remove that block, I guess that the other one about Fedora host would then remove the existing centos machines again .. so isn't there a parameter for that ipahbacrule to not purge things not declared ? if not then we'll have to still hard-code centos machines there but with dc move and so multiple machines that will need to be enrolled, I'd like that to be documented at least :)
ipahbacrule
well, my reading / understanding of it is that:
https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_85 creates the 'ipsilon' hbac rule and
https://pagure.io/fedora-infra/ansible/blob/main/f/playbooks/groups/ipsilon.yml#_101 has action: member, so it adds those members/hosts.
So, I guess we could change the first one to just confirm/add the fedora ipsilon servers and then it wouldn't recreate the rule... but then the rule would have to be... manually created or something?
Would this be high trouble, medium trouble, low trouble or a timeboxed investigation?
What if we do this:
diff --git a/playbooks/groups/ipsilon.yml b/playbooks/groups/ipsilon.yml index 8c95c4344a..1f4636ece3 100644 --- a/playbooks/groups/ipsilon.yml +++ b/playbooks/groups/ipsilon.yml @@ -89,6 +89,14 @@ hbacsvc: - ipsilon usercategory: all + ipaadmin_password: "{{ ipa_admin_password }}" + tags: + - ipsilon + + - name: Allow login through ipsilon for the Fedora instance + ipahbacrule: + name: ipsilon + action: member host: "{{ (env == 'production')|ternary(groups['ipsilon'], groups['ipsilon_stg']) }}" ipaadmin_password: "{{ ipa_admin_password }}" tags:
ie, we remove the host call from the one that makes the hbrbac rule, so it should just create it if it doesn't exist. Then we populate the fedora hosts in a seperate host level task.
Log in to comment on this ticket.