#241 Support "latest stable build" with koji directive
Closed: Fixed None Opened 8 years ago by tflink.

Before ABI checking can be supported, we need a method for downloading the "latest stable" build from koji instead of just the latest build.

Add this ability to the koji directive and update docs accordingly.


This ticket had assigned some Differential requests:
D417

My first thought on this would be to add an optional "type" (or something similar) arg to koji_directive that would accept "latest" and "latest stable", defaulting to "latest" if ommitted so we maintain backwards compatibility.

For "latest stable", we'd search koji for a package, finding the most recent one tagged with the tag associated with the most recent stable tag (updates for released, fXX for branched, not sure what rawhide would be but we'll have to deal with that later since there's not much of a delay between build and tag for rawhide)

Since I haven't tried to write the code, this might not work but figured I would at least write it down :)

type: latest seems confusing to me, because right now we're not downloading latest NEVR, we're downloading exact NEVR. So maybe type: exact as the default.

But after looking into [[ https://docs.qadevel.cloud.fedoraproject.org/libtaskotron/latest/directives/koji_directive_module.html | documentation ]], I'd rather extend the existing action argument than add a new type argument. action could be one of [ 'build', 'tag', 'latest_stable', 'previous'], where action=latest_stable behaves as described above, and action=previous downloads the very previous build ("the highest lower NEVR", I hope it's understandable) to the one specified. I believe that will be another quite common use case for such comparison tools (that's what [[ https://fedorahosted.org/autoqa/wiki/rpmguard | rpmguard ]] does, anyway -- you might even steal some code from it).

! In #491#6063, @kparal wrote:
type: latest seems confusing to me, because right now we're not downloading latest NEVR, we're downloading exact NEVR. So maybe type: exact as the default.

I'm not following you here but I'm not sure it matters much because ...

But after looking into [[ https://docs.qadevel.cloud.fedoraproject.org/libtaskotron/latest/directives/koji_directive_module.html | documentation ]], I'd rather extend the existing action argument than add a new type argument. action could be one of [ 'build', 'tag', 'latest_stable', 'previous'], where action=latest_stable behaves as described above, and action=previous downloads the very previous build ("the highest lower NEVR", I hope it's understandable) to the one specified. I believe that will be another quite common use case for such comparison tools (that's what [[ https://fedorahosted.org/autoqa/wiki/rpmguard | rpmguard ]] does, anyway -- you might even steal some code from it).

Using action makes a lot more sense than what I wrote above - serves me right for writing what was off the top of my head instead of doing a bit of research first :)

The only thing I'm not sure about is what to do about the koji_build arg - should that be changed to be a bit more general, collapsing koji_build and koji_tag into something more generic that can be used for everything? I suppose we could just use koji_build as the package name or build nvr but that feels a bit messy to me.

Another thing that I suspect will be needed is a flag to download debuginfo. It sounds like they're using [[https://sourceware.org/libabigail/index.html|libabigail]] and if I'm reading those docs correctly, libabigail uses binaries and debuginfo to figure out abi.

! In #491#6069, @tflink wrote:
The only thing I'm not sure about is what to do about the koji_build arg - should that be changed to be a bit more general, collapsing koji_build and koji_tag into something more generic that can be used for everything? I suppose we could just use koji_build as the package name or build nvr but that feels a bit messy to me.

Yes, that's something to consider. I think using koji_build with action=latest_stable is quite fine, and it makes perfect sense with action=previous. But if somebody puts more thinking into this and comes up with something clearer, I'm all ears :)

Another thing that I suspect will be needed is a flag to download debuginfo. It sounds like they're using [[https://sourceware.org/libabigail/index.html|libabigail]] and if I'm reading those docs correctly, libabigail uses binaries and debuginfo to figure out abi.

You're right, I've created #494.

Dodji came asking for help with the Koji client API on IRC, so I figured I'd just cook up a patch that does it. :)

Untested patch is at: https://bochecha.fedorapeople.org/tmp/0001-koji-Allow-downloading-the-last-tagged-build.patch

@bochecha Thanks for the patch! Just for next time, please make sure that the task you're sending a patch for is not already assigned to anyone and if so, please contact the person beforehand so that two people don't work on the same task independently. ;-)

From a first look I see one major problem with the patch and that it works with 'koji_build' and 'koji_tag'. Currently we don't support triggering a taskotron task on more that one item (nor taskotron is able to get more data from the trigger, we accept patches, though! :)). In this case, if I am not mistaken, we'll trigger the task on 'koji_build' and so the koji tag of the build needs to be somehow figured out from Koji. Or am I missing something?

Thanks!

Just for next time, please make sure that the task you're sending a patch for is not already assigned to anyone and if so, please contact the person beforehand so that two people don't work on the same task independently. ;-)

Apologies for that.

It took me all of 2 minutes to write this patch, so I figured if nobody had done it, then it meant nobody had actually started looking at it.

I'll pay more attention next time, though. ;)

From a first look I see one major problem with the patch and that it works with 'koji_build' and 'koji_tag'. Currently we don't support triggering a taskotron task on more that one item (nor taskotron is able to get more data from the trigger, we accept patches, though! :)). In this case, if I am not mistaken, we'll trigger the task on 'koji_build' and so the koji tag of the build needs to be somehow figured out from Koji.

Oh, I assumed those were parameters passed to the task, and so the ABI check task would just pass both. I didn't know they were in fact triggers and only one can be present.

That's annoying then.

Automatically figuring the tag from the build is hard, because... the Koji build will be tagged with e.g f22-updates-testing (or a -pending tag, depending on when the task is triggered).

You can't go from f22-updates-testing to f22-updates, without putting some knowledge of the Fedora releng workflow (i.e the naming convention) in the libtaskotron code.

Which makes this a bit more complex. Not sure what the best way to solve this is, though...

Hello,

So, after discussing this with bochecha and mkrizek, it turns out the code of the koji directive in libtaskotron, for the 'download_lastest' value of the 'action' parameter will have to harcode how to deduce the stable tag for a given NEVR from its current tag.

For instance, suppose a new libfoo-1.1.0-0.fc22 is built in Koji. The tag of that build is going to be f22-updates-testing. So the code will have to deduce that the stable tag is f22-updates. When we have that stable tag,we know how to ask koji to get the latest package with that tag; and that is the latest stable package we are looking for.

The problem is that this implies to hard-code how to get the stable tag name from the current tag name. Unfortunately, the relationship between a given tag name and its corresponding stable tag name is a fragile convention which depends on the version of Fedora we are looking at. So tomorrow when releng decides to change the convention, we'll need to update that code.

Ideally, we should have a kind of ReleaseDB service that stores, for every single NEVR, interesting pieces of data like, well, the name of its stable tag. Said otherwise, that ReleaseDB would provide some semantics to tag names, among other things. And that semantics could be queried by applications.

Until then I guess we are poised to hard-code this kind of logic here and there, over and over again. But the good news is, we can just remove that hard-coded coded when the ReleaseDB service is available.

So for now, from what I understand, mkrizek is not opposed to go the hard-coding route, as a stopgap measure. Now I guess the idea is to figure out the best way to implement that thing.

Ideally, we should have a kind of ReleaseDB service that stores, for every single NEVR, interesting pieces of data like, well, the name of its stable tag.

More like "for every Fedora release", rather than "for every single NEVR".

Bodhi 2 has something like that, that I had added to it a while ago. (it even has [[https://github.com/fedora-infra/bodhi/blob/develop/tools/managereleases.py|a client API]] to query/modify releases, but it might not support things like "give me the release which has "f22-updates-testing" as its testing tag)

So that could also be a possibility.

I personally think this kind of information should eventually be extracted from Bodhi into its own application, but maybe Bodhi would be sufficient for the short term... for a definition of "short" which accounts for the fact that Bodhi2 isn't even deployed yet. ;)

We already have some mapping of dist tags <-> koji tag + yum repository names in libtasktron:
https://bitbucket.org/fedoraqa/libtaskotron/src/d76218626e4a940442afb19be760a93f4fa2563f/conf/yumrepoinfo.conf.example?at=develop

So I believe the easiest way here is to deduce the fedora version from the package dist tag (i.e. extract fc22 from libfoo-1.1.0-0.fc22, convert it to f22), and then query the appropriate stable repos for older packages (i.e. f22 and f22-updates).

@mkrizek is, I believe, working on such solution right now.

Metadata Update from @tflink:
- Issue tagged with: easyfix

6 years ago

Login to comment on this ticket.

Metadata