Not sure what caused this, but there's a few dozen bodhi updates that keep getting ejected from the f39-stable pushes because some of their builds are mis-tagged.
Almost all the updates on page 2 and 3 when querying for "Fedora 39 updates that are requested → stable" are affected:
https://bodhi.fedoraproject.org/updates/?search=&releases=F39&request=stable
Not sure what happens when another update is submitted for one of the affected packages - I don't think updates that have "request=stable" are obsoleted? It might happen that multiple updates for the same package are in-flight (but I have not checked).
Soon-ish would be great. This is blocking bugfix updates (including infra components like pungi) and other work.
N/A (Fedora 39 EOL I guess?)
~3 dozen Fedora 39 updates remain stuck in "testing" state forever.
I'm not sure what happened here... a failed push? but they all need to be tagged f39-updates-testing (they didnt seem to actually be tagged in testing.)
I seem to remember yes ... I think there was a failed f39 push with the error message something like "failed to move builds 123, 456, 789, ... "
Metadata Update from @phsmoura: - Issue tagged with: medium-gain, medium-trouble, ops
I think I have tagged all of them into f39-updates-testing, so they should go stable correctly later today.
We can keep this open and check tomorrow.
Looks like it didn't work right. ;(
CC: @mattia
I am about to quit for my dayjob, so can't look at that throughly at the moment. However, most of those pending updates started to fail 2 days ago, so, at least, that was before yesterday deployment...
In Bodhi 8.0 a fix for https://github.com/fedora-infra/bodhi/issues/5396 has been deployed, so that the automatic task that checks stuck updates will try to recover the right tagging state after 48h. However, that applies for pending -> testing updates, not testing -> stable. Also, I see that many (all?) of those stuck updates have builds tagged into f39-updates, so they're likely already in the final compose, just the update hasn't been finalized.
f39-updates
Will take a more close look in ~10hs.
Not sure how this happened, but looking at this update: https://bodhi.fedoraproject.org/updates/FEDORA-2024-185b8ba0b7
The first three builds have already moved from f39-updates-testing to f39-updates, while the last three are still tagged f39-updates-testing. This is now causing build failures of dependent packages in Fedora 39, since those builds were part of the same update for a reason :(
Both "normal" and "from side tag" updates are affected - for single-package updates, they are broken if their one build is mis-tagged (already tagged as f39-updates even if the update state is still "testing -> stable"), and for multi-build updates, they can't be pushed "testing -> stable" by bodhi if any of their builds are already tagged f39-updates (and other builds still tagged f39-updates-testing).
Yeah, well, regarding that particular side-tag update, those first three builds are still not tagged in f39-updates, so they will cause the update to be ejected from the next compose again. We need someone from releng to add the tag on those three builds and hope that this is enough to fix the synch between Bodhi update and Koji tags. A self made solution would be to unpush the update and push it again, but: 1) never tested with a side-tag update and 2) would mean to start again the testing period.
But it's not looking for f39-updates is it? according to the message...
FEDORA-2024-185b8ba0b7 ejected from the push because "Cannot find relevant tag for rust-wayland-backend-0.3.3-1.fc39. None of ['f39-updates', 'f39-updates-pending', 'f39-build-side-82523'] are in ['epel9-next-testing', 'epel7-testing', 'eln-updates-testing', 'epel8-testing', 'epel9-testing', 'epel8-next-testing', 'f40-container-updates-testing', 'f38-modular-updates-testing', 'f38-flatpak-updates-testing', 'f40-updates-testing', 'f38-updates-testing', 'f38-container-updates-testing', 'f39-updates-testing', 'f39-container-updates-testing', 'f39-flatpak-updates-testing']."
So, it has 'f39-updates', 'f39-updates-pending', 'f39-build-side-82523' but none of those are in the list. It wants f39-updates-testing...
So, I had tagged everything into f39-updates-testing (and I just made sure they all are).
Is the message wrong? Or I am confused? Or both! ;)
But it's not looking for f39-updates is it? according to the message... FEDORA-2024-185b8ba0b7 ejected from the push because "Cannot find relevant tag for rust-wayland-backend-0.3.3-1.fc39. None of ['f39-updates', 'f39-updates-pending', 'f39-build-side-82523'] are in ['epel9-next-testing', 'epel7-testing', 'eln-updates-testing', 'epel8-testing', 'epel9-testing', 'epel8-next-testing', 'f40-container-updates-testing', 'f38-modular-updates-testing', 'f38-flatpak-updates-testing', 'f40-updates-testing', 'f38-updates-testing', 'f38-container-updates-testing', 'f39-updates-testing', 'f39-container-updates-testing', 'f39-flatpak-updates-testing']." So, it has 'f39-updates', 'f39-updates-pending', 'f39-build-side-82523' but none of those are in the list. It wants f39-updates-testing... So, I had tagged everything into f39-updates-testing (and I just made sure they all are). Is the message wrong? Or I am confused? Or both! ;)
Yep, I should have wrote f39-updates-testing. And indeed the update is now in correct state.
f39-updates-testing
Thanks, FEDORA-2024-185b8ba0b7 looks good now.
My other affected f39 updates were ejected from the push again, though, for example
https://bodhi.fedoraproject.org/updates/FEDORA-2024-752ed62f21 https://bodhi.fedoraproject.org/updates/FEDORA-2024-50cd2b2d81
I tagged both of those.
I think this has been fixed - I see some packages in F39+testing state in bodhi that are really old, but they are stuck in "testing" for other reasons than builds being mis-tagged (i.e. gating failures or packager error). So from my point of view, feel free to close this ticket.
Thanks!
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.