#10882 Update in weird state
Closed: Fixed 2 years ago by kevin. Opened 2 years ago by gotmax23.

Describe the issue

We are having some problems with https://bodhi.fedoraproject.org/updates/FEDORA-2022-fe8d1879bc and https://bodhi.fedoraproject.org/updates/FEDORA-2022-ba365d3703. These updates have languished in testing for a couple weeks now, as the gating tests failed and they were not dealt with.

We'd like to untag graphviz from the rawhide update, as it's since been updated in rawhide, untag singularity from the f36 one, and then rerun the gating tests for both. We were not able to do that, though. It seems that the sidetags were deleted and now the builds are only tagged with f37-build-side-54566-testing-pending for the rawhide one and just f36-updates-testing for the f36 one. Currently, it's impossible to add or remove builds from the update, as the Bodhi interface only allows refreshing the non-existent side tag. We did consider canceling the update, creating a new side-tag and koji move-building the packages, and creating a new update, but I figured I'd check with you all to avoid messing things up any further.

I am attaching a screenshot from @qulogic of the Bodhi edit update page for the rawhide update. I don't have permission to access it.

When do you need this? (YYYY/MM/DD)

Sooner rather than later. This update fixes a CVE.

When is this no longer needed or useful? (YYYY/MM/DD)

N/A

If we cannot complete your request, what is the impact?

The updates don't get pushed and the CVEs don't get fixed.


The merging of the side tags won't cause issues if another package has been built in the meantime, normally it just get dropped.

I waived the tests again.

It looks like the f36 update moved to testing > stable, but the rawhide one failed with This update cannot be pushed to stable. These builds cadvisor-0.44.1-2.fc37, containerd-1.6.6-3.fc37, golang-github-theupdateframework-notary-0.7.0-5.fc37, grafana-7.5.15-3.fc37, graphviz-4.0.0-6.fc37 have a more recent build in koji's f37 tag.

The merging of the side tags won't cause issues if another package has been built in the meantime, normally it just get dropped.

This has not been the case in my experience. For instance, there was an issue with the packages that I rebuilt where a newer version of snapd was pushed to stable before the sidetag was merged, but the older version that was merged from the sidetag replaced the newer one on the mirrors.

Metadata Update from @phsmoura:
- Issue tagged with: medium-gain, medium-trouble, ops

2 years ago

On Tue, 12 Jul 2022, 16:48 Maxwell G, pagure@pagure.io wrote:

gotmax23 added a new comment to an issue you are following:
``

The merging of the side tags won't cause issues if another package has
been built in the meantime, normally it just get dropped.

This has not been the case in my experience. For instance, there was an
issue with the packages that I rebuilt where a newer version of snapd was
pushed to stable before the sidetag was merged, but the older version that
was merged from the sidetag replaced the newer one on the mirrors.
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/10882

Yes because the changes you made in dist-git should be before the one for
new version in Rawhide. This is only a problem if you're trying to build
against a specific package in a side tag. I think.

You need to edit it and remove:

This update cannot be pushed to stable. These builds cadvisor-0.44.1-2.fc37, containerd-1.6.6-3.fc37, golang-github-theupdateframework-notary-0.7.0-5.fc37, grafana-7.5.15-3.fc37, graphviz-4.0.0-6.fc37 have a more recent build in koji's f37 tag.

bodhi has a simple upgrade check and won't let you push an older package if one is already in the dest tag.

If you remove those builds the rest should be able to go.

Yes, but we can't edit it, because the side tag has been deleted, and it only allows us to refresh the non-existent side tag.

ugh, yeah. I can see 3 options, all bad for various reasons. ;(

  1. Make another tag with the same sidetag name, tag builds into it, refresh and hope bodhi handles that ok.

  2. edit the update in the database to remove those builds and see if bodhi will push it then. But it may not realize it needs to look again at it.

  3. Just abandon the update and rebuild everything again. ;(

ok, we managed to get this working by some trickery...

I created a new sidetag, tagged all the builds into it except the ones that were already newer in f37 tag, then edited the bodhi db to associate the new sidetag with it instead of the old one. Then, I could refresh from the sidetag and the update went through. (There was a few stops and starts in there too, but that was the basic process that worked).

To avoid this moving forward perhaps we should have koji-gc ignore sidetags...

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata
Boards 1
Ops Status: Backlog
Attachments 1
Attached 2 years ago View Comment