With D755 deployed, we started getting upgradepath crashes:
File "/usr/lib/python2.7/site-packages/libtaskotron/directives/python_directive.py", line 155, in execute output = task_method(**kwargs) File "upgradepath.py", line 639, in main return upgradepath.run_once(args.koji_tag) File "upgradepath.py", line 553, in run_once tag_builds, detail, check_pending=True) File "upgradepath.py", line 161, in compare proposed_build['name'], tag_tuple) File "upgradepath.py", line 267, in check_pending_request releases=[bodhi_tag], limit=1) File "/usr/lib/python2.7/site-packages/fedora/client/bodhi.py", line 96, in wrapper raise BodhiClientException(problems) BodhiClientException: Invalid releases specified: F24
https://taskotron.fedoraproject.org/taskmaster/builders/x86_64/builds/189470/steps/runtask/logs/stdio
The problem is that bodhi is not aware of F24. dgilmore claims:
<dgilmore> kparal: we enable bodhi at alpha change freeze <dgilmore> it has been that way for a lot of releases now <dgilmore> kparal: the sop is incorrect That is true, but I believe the release was always defined in Bodhi at branching point, and //enabled// at Alpha freeze point. I checked upgradepath sources and the code was not changed since 2014. It used to work just fine. However, some assumptions might have been changed with Bodhi2, because that was deployed after F23 Alpha, and therefore this is the first time we have bodhi2 in this situation (pre-Alpha).
We need to either make sure bodhi devs fix this quickly, or we need to work around this in upgradepath and yumrepoinfo.conf. One of that needs to happen soon, or we can't deploy latest yumrepoinfo.conf, and therefore can't respond to F24 events (koji builds, etc).
This ticket had assigned some Differential requests: D755
When you try to query bodhi for supported releases, be aware of this issue: https://github.com/fedora-infra/bodhi/issues/784
There's the branching SOP: https://docs.pagure.org/releng/sop_mass_branching.html#bodhi And Bodhi SOP: https://infrastructure.fedoraproject.org/infra/docs/bodhi.rst
To me, it seems to indicate that the bodhi release should be created as "pending" (locked) at the branch day, and unlocked during Alpha freeze when it starts receiving updates. However, dgilmore does not agree with this and says everything should be performed on alpha freeze (and always has been). We need to figure out what exactly the document means, what is a locked release, and whether there are any issues with creating a locked release in bodhi at branch day, and unlocking it at alpha freeze.
Otherwise we'll need to make yumrepoinfo.conf account for the fact that there's a several-week window in which all systems know about f24 except bodhi, make our tasks understand that, and push yumrepoinfo.conf updates to our systems twice (at branch point and at alpha freeze point). Which is quite a hell from automation standpoint.
So, I checked with threebean and lmacken. This is not expected to work in Bodhi OOTB. The code is mostly there, but it will need some UI changes to make sure the pending/disabled branches do not show up in the UI and updates can't be submitted to it. Also the SOP will need updates. I'll file a Bodhi RFE and link it here, and it should be prepared for F25.
For F24, we'll need to wait a few weeks before Alpha freeze is in effect and Bodhi learns about F24, and only then deploy new yumrepoinfo.conf. Or we can work around the issues and ignore the particular Bodhi exception, but I wonder whether it is worth the troubles. According to the current schedule, the Alpha freeze is planned for 2016-03-08, which is in a week. So for a week, upgradepath won't be able to detect issues related to F24 builds. That's not that serious. (But in a year, when we have package-specific checks supported, we'll definitely want to have this working without issues or delays.)
I have created this ticket to cover F24 transition, and I created https://github.com/fedora-infra/bodhi/issues/794 to ask for needed changed in Bodhi (so that we're prepared better for F25). I'm removing this from the planning workboard, this is not actionable right now, and not immediately needed.
So, our tasks are now crashing heavily due to this for F25-related jobs.
Do we want to try cornering Luke while we're all here at Flock to figure out how to fix this or assume it won't change and devise a scheme for fixing/masking the problem on our end?
My current recommendation is to revert the yumrepoinfo change (revert adding F25) so that we stop our tasks from crashing (mainly upgradepath, it seems). Alpha freeze and Bodhi activation point is set to Aug 9, so after a week we can repeat the yumrepoinfo change and everything should work fine.
Yes, we should definitely corner Luke at Flock. The Bodhi feature should be almost ready, we need to make sure this does not repeat again for F26. Masking the problem on our end (long term) does not make any sense to me. If we're to provide a decent continuous testing tool, the infrastructure must accommodate for it. Even if we had to write the Bodhi patch ourselves, I think that would be a better way forward than making complex hacks in Taskotron. But I hope we won't need to do any of that :-)
Lowering priority since F25 release was added to Bodhi earlier this week.
It sounds like the bodhi issue is not likely to be fixed before f26 branch. Do we want to just delay updating yumrepoinfo.conf until f26 alpha or do we want to look at other solutions so that at least our tasks don't crash between f26 branch and the time at which it's added to bodhi?
yumrepoinfo.conf
! In #729#11890, @tflink wrote: It sounds like the bodhi issue is not likely to be fixed before f26 branch.
{meme, src=victorybaby, above="Just you wait", below="Lmacken..."}
Do we want to just delay updating yumrepoinfo.conf until f26 alpha or do we want to look at other solutions so that at least our tasks don't crash between f26 branch and the time at which it's added to bodhi?
I'll ask bowlofeggs whether this could be implemented before F26. He's been doing a lot of work on Bodhi recently.
I pushed a change in upgradepath that could work around this issue: https://pagure.io/taskotron/task-upgradepath/c/5ad0c5586085ba32e810c4974eaee09209160206?branch=develop
That of course doesn't help if any other tasks had the same issue, but I think upgradepath is the only one.
The fix for upgradepath has been deployed. I'll keep this ticket open, because it's a generic problem that we need to resolve rather sooner than later, but at least upgradepath should now not crash.
Log in to comment on this ticket.