This seems to be another instance of a pattern: https://pagure.io/releng/issue/13037, https://pagure.io/releng/issue/13018,https://pagure.io/releng/issue/13009.

I also just noticed that rust-async_http_range_reader has the same problem. I assume there are others…

I don't see the toddler getting the commit where it was dead.package'd... but historical logs on pkgs are anoying to sift though. ;(

Do we have any more recent case to look at?

I also just noticed that rust-async_http_range_reader has the same problem. I assume there are others…

Oops, I orphaned rust-async_http_range_reader but did not retire it, so there is no problem with it after all.

Do we have any more recent case to look at?

Hmm, I don’t exactly retire packages daily, but I think we’re in luck. I retired the brand-new compat package rust-gif0.13 in Rawhide without ever building it, about ten hours ago. (I only needed it in F42, F43, and EPEL9.) So far, koji list-pkgs --package rust-gif0.13 --show-blocked doesn’t show it blocked in f44. Does thaat help?

yeah, thanks.

https://apps.fedoraproject.org/datagrepper/v2/id?id=657cc1ba-4d70-41b3-85b8-d2fc073eb960&is_raw=true&size=extra-large

is the message bus messaage, so it was emitted.

and yet, I see no sign that the toddler processed it. ;(

CC: @abompard any ideas?

Another instance:
https://src.fedoraproject.org/rpms/rust-atspi-common

Yeah, no errors in the poddler pod around then either. ;(

@lenkaseg or @zlopez any idea why the toddler wouldn't be seeing these?

yeah, thanks.

https://apps.fedoraproject.org/datagrepper/v2/id?id=657cc1ba-4d70-41b3-85b8-d2fc073eb960&is_raw=true&size=extra-large

is the message bus messaage, so it was emitted.

and yet, I see no sign that the toddler processed it. ;(

CC: @abompard any ideas?

First off the topic consumed by koji-block-retired toddler is org.fedoraproject.prod.git.receive and not org.fedoraproject.prod.pagure.git.receive

And looking at the datagrepper we are probably consuming wrong messages as the rust-atspi-common retire commit message is completely missing in
https://apps.fedoraproject.org/datagrepper/v2/search?topic=org.fedoraproject.prod.git.receive&end=1768385400, but could be find in https://apps.fedoraproject.org/datagrepper/v2/search?topic=org.fedoraproject.prod.pagure.git.receive&end=1768385400 (https://apps.fedoraproject.org/datagrepper/v2/id?id=75744d1c-78a1-4b4b-ae76-07f3d527624f&is_raw=true&size=extra-large)

I'm not sure what is the difference between these two queues, but the org.fedoraproject.prod.git.receive is missing the correct commit messages.

Thanks for looking at that @zlopez!

Seems that adding a topic org.fedoraproject.*.pagure.git.receive to consumed topics would solve the issue?

yes, we have two git hooks. ;)

I cannot seem to find the ticket about it, but we have pagure emitting a message and also a manual git hook.
We tried to retire one of them, but it broke something so we backed it out.

So, it sounds like that second hook isn't working right then? We can't just add the topic, we would also need to adjust the toddler to be able to process the new messages, and I think they do not actually contain enough information for it to work. Perhaps thats the blocker we hit.

As a simple thing I am going to run the playbook on pkgs and restart httpd and pagure* and see if it's just something thats stuck.

Can the affected packages (not sure how to query for "retired in git but not blocked in koji" effectively) be blocked manually for now? Or would the "fix" also "fix" things retroactively?

(I seem to remember that there used to be a periodic (daily?) check that ran that would catch retirements that were not processed asynchronously based on the messages - is this no longer working either, or was that decommissioned?)

@kevin We should add both topics to toddler, so we don't miss anything. This needs somebody to look at it more closely, as I'm not sure how different the messages are.

@decathorpe We will need to do it manually for those that were missed as the toddler will just start consuming new messages and will not go through old ones. I'm not aware about the automation that is checking that, but it's possible that there is/was something like that and it either doesn't work anymore or it's no longer running.

There is playtime call for koji_block_retired toddler that runs every day at 5:30 AM UTC that finds and blocks all retired packages that are not blocked: https://pagure.io/fedora-infra/toddlers/blob/main/f/toddlers/plugins/koji_block_retired.py#_88 So yeah, it should cover for the missed messages.
Just checking the logs, seems it's the playtime call is checking only on epel branches, ignoring the rest. Something might have changed with the APIs? Checking

I think I might see the problem. The toddler is querying bodhi to get active releases and is fetching some fields which might have changed and now the script does not get full list of active branches.

Here is a PR: https://pagure.io/fedora-infra/toddlers/pull-request/386

It looks reasonable to me... @zlopez can you review and deploy?

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/infra/tickets/issues/13020

Please continue any further discussion there.