I'm not sure it's easy to 'unfreeze' now. We would have to disable bodhi and change tags and figure out how to adjust existing bodhi updates to behave differently and other things I am sure...

How about we push all packages to stable that are requested stable at 23:59 on 2019-08-29 and then move to only pushing updates that match a FE or blocker?

That wouldn't solve everything, but would allow people who wanted to get something in ability to do so?

I have retired packages that should have been retired before the freeze.

I also have some updates in, but I have no idea how to request stable on them so fast (that seems impossible without karma cheating).

All the f31 updates are pending into testing ATM:

https://bodhi.fedoraproject.org/updates/?releases=F31&status=pending
https://bodhi.fedoraproject.org/updates/?releases=F31&status=testing

I'm fine if we decide that this is all OK. I just feel that this was not handled very well. I proposed to delay the freeze before it happened (on the mailing list), but it was too late. I'm trying to figure out if there is anything to do.

Should we just tell people to request Freeze Exceptions if they were caught in this?

I still remember this https://pagure.io/fesco/issue/2092#comment-558351 comment. If FESCo as a whole group still think the same then why not abandon publishing Fedora schedule on the wiki pages?

I still remember this https://pagure.io/fesco/issue/2092#comment-558351 comment. If FESCo as a whole group still think the same then why not abandon publishing Fedora schedule on the wiki pages?

That's already my plan for F32 and beyond. Some tooling improvements have made it easier to represent the high-level milestones in an easily understandable way.

Thanks for the update @bcotton

So, we are kinda running low on time to decide anything here, and most of fesco has not weighed in yet.

All the f31 updates are pending into testing ATM:
https://bodhi.fedoraproject.org/updates/?releases=F31&status=pending
https://bodhi.fedoraproject.org/updates/?releases=F31&status=testing

Thats due to the stuck testing push thats happening right now. As soon as that goes out (hopefully soon here) things should show up as being pushable to stable.

I guess we could just ask people affected to do FE's indeed.

@adamwill @tflink @kparal any qa thoughts here?

+1 stick with the freeze and just tell people to request FEs. We can say we'll accept any reasonable FE proposed for a package that was queued for stable before the 29th, I guess.

+1

@churchyard yo yo yo. I said we'd accept FEs FOR PACKAGES WHERE AN UPDATE WAS QUEUED FOR STABLE BEFORE THE 29TH. Not that we'd blanket accept 30+ FEs proposed on bugs where it doesn't look like anyone was doing any work at all.

I am going to blanket un-propose all the bugs you proposed where there is no indication that any fix is actually present. After all, you said the packages would be retired on the 29th and it's now the 30th. There's no justification for granting them blind FEs to rush in a fix now, we only wanted to take fixes which were ready between 26th and 29th.

OK, so, it seems @churchyard filed 30+ FE proposals to propose going ahead with the retirements of the packages, I think because the freeze kicking in when it did prevented him from doing the retirements.

I don't think we really need a separate FE proposal for every package that was supposed to be retired, I think we can handle it with a single bug...

Further: @kevin and I both checked, and the freeze did not prevent retirement of these packages at all, as far as we can tell. They were all properly retired and are not present in recent Branched composes - @churchyard , you can just look through the source RPMs in a recent compose to verify for yourself, e.g. https://kojipkgs.fedoraproject.org/compose/branched/Fedora-31-20190829.n.0/compose/Everything/source/tree/Packages/ .

So I unproposed all the bugs (and you should probably re-close them), and I don't think we need to propose any new FE. Please post here if you think something is still outstanding. Thanks!

Thanks for the investigation. The problem is that I can still see the retired packages in the "fedora" repo, so I've assumed it's the freeze that's holding theme there.

The 'fedora' repo should pull from https://dl.fedoraproject.org/pub/fedora/linux/development/31/ and I don't see them in https://dl.fedoraproject.org/pub/fedora/linux/development/31/Everything/source/tree/Packages/ either.

Maybe it is a mirror issue then. Will recheck again in couple days and close the bugs when the packages disappear.

The packages are listed in https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/T57QIWMUR5MCEP7KBTJVLXJK6FITOD3T/ :thumbsup:

Nothing to do here. Thanks for help everybody!