Learn more about these different git repos.
Other Git URLs
The zlib package was downgraded in rawhide. On the 30th it was
On the 1st it was:
zlib-1.2.11-31.fc36.x86_64.rpm
When do you need this? (YYYY/MM/DD)
Would be nice to get it fixed
They were deliberately untagged.
Jun 30 06:36:15 <ljavorsk> Hi guys, could you please help me untag two new zlib builds from Fedo ra Rawhide? I've submitted them yesterday with the vision of fully functional release, but unfortunat elly some packages (like ruby) started to having issues. I don't want to block any package so I want to untag the new releases and fix the zlib in the private copr package that I'll provide for testing later. Jun 30 06:36:44 <ljavorsk> However, the first priority right now is to untag it until it becomes a bigger issue Jun 30 06:37:13 <ljavorsk> The builds are these two: https://koji.fedoraproject.org/koji/buildin fo?buildID=1994960 and https://koji.fedoraproject.org/koji/buildinfo?buildID=1994960 Jun 30 06:37:16 <jednorozec> ljavorsk, nvr? Jun 30 06:37:18 <jednorozec> ah Jun 30 06:37:33 <ljavorsk> Sorry wrong links Jun 30 06:37:38 <ljavorsk> zlib-1.2.12-2 Jun 30 06:37:44 <ljavorsk> And zlib-1.2.12-1 Jun 30 06:38:27 <ljavorsk> After untagging those two build the zlib-1.2.11-31 should be the newe st one Jun 30 06:39:40 <ljavorsk> It also looks like they were propagated to the ELN so could you pleas e untag them for ELN as well? Jun 30 06:41:22 <jednorozec> ljavorsk, done Jun 30 06:44:31 <ljavorsk> jednorozec: Thank you so much
Normally we don't untag things that have already gone out, but fesco approved a clause allowing releng to do so if they found it was needed.
So, that explains things. Is there something we could do to improve this flow?
Hmm. I guess it's unexpected that a package NEVRA will go backwards. I'm OK with untagging things but there is no way to communicate with the users of the distro that this was expected. I can think of a few ways to improve things:
Going from memory of sitting in releng for years, I think that these reversions in rawhide do happen a lot more than people realize. Could you outline on why this caused an issue for your group so it is known for the future?
The issue mainly stems from our build automation that warns us (Fedora CoreOS) when package versions go down. Maybe we should treat rawhide differently than we do say Fedora 35/36.
Though it is interesting to think about from the user's perspective too. What happens at this point if a user had upgraded (has the new zlib) and the package maintainer doesn't do another build for months. The user is stuck on the bad package. I know stuff like this can be OK for rawhide. Just mentioning it for discussion.
but fesco approved a clause allowing releng to do so if they found it was needed.
Do you recall where this happened?
My understanding always was that untagging can only happen if the build was not yet composed. Untagging broken builds that made it to the compose should only be allowed in emergencies (such as where nothing else can even build -- but even that can usually be solved via a side tag) and such cases should have a limited time of existence. We should not allow epoch-less downgrades in between composes.
So, before this spirals...
It is absolutely not at all common. It's very rare. I don't know if @humaton even knew the package was pushed out in a compose or not. It's also unclear what the failure was in the package.
@ljavorsk can you expand on what is going on in those packages?
The thing I was thinking of was:
"Once a package has been added to the compose and that compose has finished and synced to the master mirrors, it will normally not be untagged. This is required to allow others to depend on the build once it has become visible. In exceptional cases, releng may untag packages."
The last sentence was added 2 years ago in https://pagure.io/fesco/fesco-docs/c/997bd212d760f223ecaa98cbc5e020ad35754b01?branch=master
Perhaps we should send a devel or devel-announce notice when this happens? Or just have a ticket? Open to improving the process...
I would say send the notice on devel list when this happens in future. It's more likely to be noticed by people affected than a ticket would be
Perhaps we should send a devel or devel-announce notice when this happens? Or just have a ticket? Open to improving the process... I would say send the notice on devel list when this happens in future. It's more likely to be noticed by people affected than a ticket would be
Additionally could we add a note to the build in bodhi. It's a great place to go see feedback about a particular build. We could give it a thumbs down and a comment on why it was untagged. If that had existed I never would have opened this ticket.
For example: https://bodhi.fedoraproject.org/updates/FEDORA-2022-ae1ef61f24#comment-2606950
I suppose it's all about what we understand as an "exceptional case".
Hello guys,
I apologize if I did something wrong, this was my first untag in Fedora Rawhide. I was in a hurry because the ruby package has reported an issue (https://github.com/ruby/spec/issues/932#event-6933754688) with the new zlib version (this rebase was quite a big update), so I got scared that it could break other packages as well and I wanted to prevent this by untagging the new release as soon as possible and find out the core of this issue.
After reading these comments I can see that I should have done things differently. However, there are multiple better solutions for this, could someone please wrap them up, so I know how to do it right the next time I have this type of problem?
Should I bump the Epoch and build an old package with bumped epoch without untagging the "new" build?
Thank you for clarifying and again, I'm very sorry for the inconvenience.
I don't think you did anything wrong here @ljavorsk :) It's just that we don't have a real process for these things (yet).
I'm fine with doing a post to devel list on it, adding a note to the bodhi update, and (if desired) a releng ticket...
You could bump the Epoch and do this yourself, but that has some drawbacks as well. (If there's things that BuildRequire/Require your thing by version, all of them have to update to include the Epoch too) and the Epoch stays forever.
Metadata Update from @phsmoura: - Issue tagged with: low-gain, low-trouble, ops
The patched zlib release is now pushed to stable: https://bodhi.fedoraproject.org/updates/FEDORA-2022-1bd6fd68f5
This release fixes the issue with ruby.
Indeed. You didn't do anything wrong @ljavorsk. I opened this ticket originally because I thought there was some bug in our system that accidentally made the package version go down.
This lead to a discussion about our lack of process here and what the potential process should be. Overall it was productive.
There was some productive discussion here. Releng will always track untagged builds in pagure tickets and keep track of them.
Metadata Update from @humaton: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.