Might be differences between the bodhi pungi.rpm config and the rawhide/branched pungi-fedora config? but then, why did it fail only on the 5th? it was using unified-core before then?

Quick look at the pungi configs revealed that we don't have unified_core=true in pungi config in Bodhi.

The config in pungi-fedora is in:
- Rawhide: https://pagure.io/pungi-fedora/blob/main/f/fedora.conf#_724
- f39: https://pagure.io/pungi-fedora/blob/f39/f/fedora.conf#_724

Where is the bodhi config?

Maybe related to https://pagure.io/fedora-infrastructure/issue/10779 ?

Found https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/backend/files but nothing appears to be duplicated.

Ah, looks like it's https://pagure.io/fedora-infra/ansible/blob/main/f/roles/bodhi2/backend/templates/pungi.rpm.conf.j2

PR in https://pagure.io/fedora-infra/ansible/pull-request/1697

Commit 2c399368 fixes this issue

Looks like this fixed building with unified core (https://kojipkgs.fedoraproject.org/compose/updates/Fedora-39-updates-20231213.0/logs/x86_64/Everything/ostree-1/create-ostree-repo.log) but we still don't have updated commits in the repo. So it looks like something else is failing.

I see:

[WARNING ] Failed to invoke notification script.

in the compose logs. This is similar to what we saw a while back... it being unable to run the notification scripts. Those are needed to interface with robosignatory for signing, and when they don't work it just times out, never signs them or syncs them. ;(

I am not sure what the issue is with the notification script. I am running the playbook now for bodhi-backend... might be able to look more later, but that might also fix it if it's somehow a perm issue.

I looked at permissions and tweaked some things, but it turns out all the file perms are just fine.

So, the problem is this:

https://kojipkgs.fedoraproject.org/compose/updates/Fedora-39-updates-20231214.0/logs/global/notifications/notification-2023-12-14_00-15-48.log

fedora_messaging.exceptions.PublishForbidden: (403, "ACCESS_REFUSED - access to topic 'org.fedoraproject.prod.pungi.compose.status.change' in exchange 'amq.topic' in vhost '/pubsub' refused for user 'bodhi'")

but:

TASK [rabbit/user : debug] *******************************************************************
Thursday 14 December 2023  01:26:15 +0000 (0:00:00.117)       0:00:15.589 ***** 
Thursday 14 December 2023  01:26:15 +0000 (0:00:00.117)       0:00:15.589 *****              
ok: [bodhi-backend01.iad2.fedoraproject.org] => {                                             
    "msg": "Topic permissions: [{'vhost': '/pubsub', 'read_priv': '.*', 'write_priv': '^org\\\
\.fedoraproject\\\\.prod\\\\.(bodhi|pungi)\\\\..*'}]"                                         
}   

so it seems like it's setting it... and indeed I can see the correct perms that should allow it there.

@abompard ... any ideas?

I'll have a look, is there a way for me to trigger those messages again? Maybe a playbook?

Thanks for the investigation.

PR in https://pagure.io/fedora-infra/ansible/pull-request/1697

I've realized that this PR was incomplete so I've made a followup in https://pagure.io/fedora-infra/ansible/pull-request/1699

I'll have a look, is there a way for me to trigger those messages again? Maybe a playbook?

Well, we could do an updates push? @jnsamyak or @humaton should be able to do one?
You might be able to just run the notification scripts manually as 'apache' user on bodhi-backend01 too...

Found why, the bodhi allowed topics list is defined in playbooks/groups/bodhi-backend.yml, in playbooks/openshift-apps/bodhi.yml and in roles/bodhi2/base/tasks/main.yml, with different values. Only the bodhi-backend playbook allows pungi. I'll fix this and apply.

Alright this should be fixed by https://pagure.io/fedora-infra/ansible/pull-request/1700.
It brings a "new" variable files location into the repo so I'm not merging it right away.
In the meantime I've set the proper permissions manually and unless someone runs the playbooks/openshift-apps/bodhi.yml or the playbooks/manual/upgrade/bodhi.yml playbooks they should stay as-is.

Thanks for the investigation and the fix!
Could we trigger an update today in order to not have to wait for tomorrow to verify this is fixed?
Thanks!

I started a f39 updates push. fingers crossed.

I wonder why we didn't hit this before? ;(

Looks like it's working. :)

Awesome! Thanks!

I received Silverblue 39.20231215.1, yay

Looks good now! Thanks!

silverblue  38               39               rawhide
x86_64      38.20231216.0    39.20231215.1    Rawhide.20231216.n.0
aarch64     38.20231216.0    39.20231215.1    Rawhide.20231216.n.0
ppc64le     38.20231216.0    39.20231215.1    Rawhide.20231216.n.0

kinoite     38               39               rawhide
x86_64      38.20231216.0    39.20231215.1    Rawhide.20231216.n.0
aarch64     38.20231216.0    39.20231215.1    Rawhide.20231216.n.0
ppc64le     38.20231216.0    39.20231215.1    Rawhide.20231216.n.0

sericea     38               39               rawhide
x86_64      38.20231216.0    39.20231215.1    Rawhide.20231216.n.0
aarch64     38.20231216.0    39.20231215.1    Rawhide.20231215.n.0

onyx        38               39               rawhide
x86_64                       39.20231215.1    Rawhide.20231216.n.0