#10875 zlib downgraded in rawhide 1.2.12-2.fc37 -> 1.2.11-31.fc36
Closed: Fixed 2 years ago by humaton. Opened 2 years ago by dustymabe.

  • Describe the issue

The zlib package was downgraded in rawhide. On the 30th it was

On the 1st it was:

Would be nice to get it fixed


They were deliberately untagged.

Jun 30 06:36:15 <ljavorsk>      Hi guys, could you please help me untag two new zlib builds from Fedo
ra Rawhide? I've submitted them yesterday with the vision of fully functional release, but unfortunat
elly some packages (like ruby) started to having issues. I don't want to block any package so I want 
to untag the new releases and fix the zlib in the private copr package that I'll provide for testing 
later.
Jun 30 06:36:44 <ljavorsk>      However, the first priority right now is to untag it until it becomes
 a bigger issue
Jun 30 06:37:13 <ljavorsk>      The builds are these two: https://koji.fedoraproject.org/koji/buildin
fo?buildID=1994960 and https://koji.fedoraproject.org/koji/buildinfo?buildID=1994960
Jun 30 06:37:16 <jednorozec>    ljavorsk, nvr?
Jun 30 06:37:18 <jednorozec>    ah
Jun 30 06:37:33 <ljavorsk>      Sorry wrong links
Jun 30 06:37:38 <ljavorsk>      zlib-1.2.12-2
Jun 30 06:37:44 <ljavorsk>      And zlib-1.2.12-1
Jun 30 06:38:27 <ljavorsk>      After untagging those two build the zlib-1.2.11-31 should be the newe
st one
Jun 30 06:39:40 <ljavorsk>      It also looks like they were propagated to the ELN so could you pleas
e untag them for ELN as well?
Jun 30 06:41:22 <jednorozec>    ljavorsk, done
Jun 30 06:44:31 <ljavorsk>      jednorozec: Thank you so much

Normally we don't untag things that have already gone out, but fesco approved a clause allowing releng to do so if they found it was needed.

So, that explains things. Is there something we could do to improve this flow?

Hmm. I guess it's unexpected that a package NEVRA will go backwards. I'm OK with untagging things but there is no way to communicate with the users of the distro that this was expected. I can think of a few ways to improve things:

Going from memory of sitting in releng for years, I think that these reversions in rawhide do happen a lot more than people realize. Could you outline on why this caused an issue for your group so it is known for the future?

The issue mainly stems from our build automation that warns us (Fedora CoreOS) when package versions go down. Maybe we should treat rawhide differently than we do say Fedora 35/36.

Though it is interesting to think about from the user's perspective too. What happens at this point if a user had upgraded (has the new zlib) and the package maintainer doesn't do another build for months. The user is stuck on the bad package. I know stuff like this can be OK for rawhide. Just mentioning it for discussion.

but fesco approved a clause allowing releng to do so if they found it was needed.

Do you recall where this happened?

My understanding always was that untagging can only happen if the build was not yet composed. Untagging broken builds that made it to the compose should only be allowed in emergencies (such as where nothing else can even build -- but even that can usually be solved via a side tag) and such cases should have a limited time of existence. We should not allow epoch-less downgrades in between composes.

So, before this spirals...

It is absolutely not at all common. It's very rare.
I don't know if @humaton even knew the package was pushed out in a compose or not.
It's also unclear what the failure was in the package.

@ljavorsk can you expand on what is going on in those packages?

The thing I was thinking of was:

"Once a package has been added to the compose and that compose has finished and synced to the master mirrors, it will normally not be untagged. This is required to allow others to depend on the build once it has become visible. In exceptional cases, releng may untag packages."

The last sentence was added 2 years ago in https://pagure.io/fesco/fesco-docs/c/997bd212d760f223ecaa98cbc5e020ad35754b01?branch=master

Perhaps we should send a devel or devel-announce notice when this happens?
Or just have a ticket?
Open to improving the process...

Perhaps we should send a devel or devel-announce notice when this happens?
Or just have a ticket?
Open to improving the process...

I would say send the notice on devel list when this happens in future. It's more likely to be noticed by people affected than a ticket would be

Perhaps we should send a devel or devel-announce notice when this happens?
Or just have a ticket?
Open to improving the process...

I would say send the notice on devel list when this happens in future. It's more likely to be noticed by people affected than a ticket would be

Additionally could we add a note to the build in bodhi. It's a great place to go see feedback about a particular build. We could give it a thumbs down and a comment on why it was untagged. If that had existed I never would have opened this ticket.

I suppose it's all about what we understand as an "exceptional case".

Hello guys,

I apologize if I did something wrong, this was my first untag in Fedora Rawhide. I was in a hurry because the ruby package has reported an issue (https://github.com/ruby/spec/issues/932#event-6933754688) with the new zlib version (this rebase was quite a big update), so I got scared that it could break other packages as well and I wanted to prevent this by untagging the new release as soon as possible and find out the core of this issue.

After reading these comments I can see that I should have done things differently. However, there are multiple better solutions for this, could someone please wrap them up, so I know how to do it right the next time I have this type of problem?

Should I bump the Epoch and build an old package with bumped epoch without untagging the "new" build?

Thank you for clarifying and again, I'm very sorry for the inconvenience.

I don't think you did anything wrong here @ljavorsk :) It's just that we don't have a real process for these things (yet).

I'm fine with doing a post to devel list on it, adding a note to the bodhi update, and (if desired) a releng ticket...

You could bump the Epoch and do this yourself, but that has some drawbacks as well. (If there's things that BuildRequire/Require your thing by version, all of them have to update to include the Epoch too) and the Epoch stays forever.

Metadata Update from @phsmoura:
- Issue tagged with: low-gain, low-trouble, ops

2 years ago

The patched zlib release is now pushed to stable: https://bodhi.fedoraproject.org/updates/FEDORA-2022-1bd6fd68f5

This release fixes the issue with ruby.

I don't think you did anything wrong here @ljavorsk :) It's just that we don't have a real process for these things (yet).

Indeed. You didn't do anything wrong @ljavorsk. I opened this ticket originally because I thought there was some bug in our system that accidentally made the package version go down.

This lead to a discussion about our lack of process here and what the potential process should be. Overall it was productive.

There was some productive discussion here. Releng will always track untagged builds in pagure tickets and keep track of them.

Metadata Update from @humaton:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata
Boards 1
Ops Status: Backlog