This ticket also depends on https://pagure.io/taskotron/taskotron-trigger/issue/25
This ticket also depends on https://pagure.io/taskotron/taskotron-trigger/issue/24
This ticket also depends on https://pagure.io/taskotron/issue/191

One note here. I believe this ticket is a result of the following discussion:
https://lists.fedoraproject.org/pipermail/devel/2015-June/211160.html

However, the ticket seems to be talking about checking Bodhi updates:

To start, we'd like to have an automated way to check the ABI
compatibility of binaries embedded in packages that are submitted to the
updates-testing repository. When an incompatible change[1] is detected,
the package will be prevented from auto-increasing it's karma count. I
guess a manual intervention will then be required to increase the karma
or a new package version needs to be submitted again.

Now, #491 has been filed against koji directive. But unless I'm really mistaken, we're not able to easily take a Bodhi update, split it into pieces per included Koji builds and then run the koji directive on each of them, all in the task formula. (And we don't even have a bodhi_update trigger in taskotron-trigger yet, because nothing has needed it.)

So, it might be much easier for us to have this triggered on every Koji build. However, that has two disadvantages:
a) we have currently no way how to deliver the result to the maintainer
b) once the Bodhi update is created, there's no process to stop the autokarma push. This will be solved only once Bodhi2 is in place //and// they implement taskotron results collection (i.e. they define abidiff to be an essential check, they listen for every result reported by taskotron for this check, and automatically disable autokarma once we find a failed build that also is currently proposed in some update in Bodhi).

Alternatively, we need to implement the bodhi_update trigger in taskotron-trigger, and then implement the new "latest stable" downloading features in the bodhi directive. However, that seems to be quite complicated, and it also involves writing a wrapper to run the abidiff tool individually on every Koji build from that update and then collecting results. From the quick glance, I think the first approach mentioned would be much more suitable for this abi diffing use case.

I'm confused: taskotron already disables autokarma if the upgrade path test fails. How is this different?

I added my logs from the IRC conversation that got it started for me: P2

! In #490#6070, @sgallagh wrote:
I'm confused: taskotron already disables autokarma if the upgrade path test fails. How is this different?

Actually, that's bodhi - it looks for posted failures and disables autokarma when a failure is detected.

It's mostly a question of how we actually implement the check - because the formula syntax is yaml based, there are limits to what can be done sanely with it. @kparal is mostly exploring options that could be used to get all the input data that we think is needed for the ABI checking task.

Triggering on koji builds gives easier access to the components of that build but it's not possible to associate results from a koji build with its eventual bodhi update using our current toolset and code. It might be more possible with bodhi2 but if that code works how I think it works, I suspect not, though.

We'll get something figured out, though :)

! In #490#6070, @sgallagh wrote:
I'm confused: taskotron already disables autokarma if the upgrade path test fails. How is this different?

The inconvenient truth is that there's a lot of dark magic behind depcheck and upgradepath. Currently, we don't execute it with bodhi_update type (triggered after bodhi update is proposed or changed), but we execute it with koji_tag type (triggered every time a certain koji tag changes, in our case fxx-updates{,-testing}-pending). We execute it over and over again, and test all available updates every time (i.e. each update is tested tens or even hundreds of times before it is finally pushed). After each execution, we make a list of all tested koji builds, query Bodhi heavily to derive all affected Bodhi builds from it, and then report the results for every update.

It is quite ineffective way of testing things, but we're haven't arrived with a better idea yet. The reason is that both depcheck and upgradepath results are not static, but they might change with every repo change - they are affected by other packages that get pushed into the repo. So we cannot just test it once and declare pass/failed.

OTOH, the abidiff check seems to be a good example of run-once check - you compare the new and the old build, and say whether it is OK or not OK. No other packages or repo state can affect that result (am I correct in this assumption, @sgallagh)? Therefore we can run this immediately once a new koji build is complete, we don't need to wait until a new update is proposed. Which is a great advantage, I think. It's also much easier to implement this approach with our current task directives and task formula design. But it has the issues as I mentioned, namely we don't have good ways to announce the result yet (you can always look into ResultsDB manually, though).

It might be more possible with bodhi2 but if that code works how I think it works, I suspect not, though.

It should be possible with Bodhi2 if it's going to work how we would //like// it to work :o)

We'll get something figured out, though :)

It might even involve some black magic, if needed. What I'm trying is to figure out the easiest way to do this.

! In #490#6067, @kparal wrote:
One note here. I believe this ticket is a result of the following discussion:
https://lists.fedoraproject.org/pipermail/devel/2015-June/211160.html

However, the ticket seems to be talking about checking Bodhi updates:

To start, we'd like to have an automated way to check the ABI
compatibility of binaries embedded in packages that are submitted to the
updates-testing repository. When an incompatible change[1] is detected,
the package will be prevented from auto-increasing it's karma count. I
guess a manual intervention will then be required to increase the karma
or a new package version needs to be submitted again.

I suspect that the important part is preventing unintentional or unknown-to-others ABI breakage from making it into the stable repo before release or into the stable updates repo post-release. I don't think they care a whole lot about the lower level implementation details as long as the result is a relatively easy to use interface (preferrably not a new interface) that makes it difficult to get ABI breaking builds to stable without some level of approval.

On a related note, I wonder if bodhi2 going to support the level of manual intervention desired here? If they're looking for the same level of manual intervention that bodhi currently has (update has to be manually requested for stable push), that'll be easily done but if it's anything more, I'm not sure that the bodhi2 folks are planning for that.

Now, #491 has been filed against koji directive. But unless I'm really mistaken, we're not able to easily take a Bodhi update, split it into pieces per included Koji builds and then run the koji directive on each of them, all in the task formula. (And we don't even have a bodhi_update trigger in taskotron-trigger yet, because nothing has needed it.)

So, it might be much easier for us to have this triggered on every Koji build. However, that has two disadvantages:
a) we have currently no way how to deliver the result to the maintainer
b) once the Bodhi update is created, there's no process to stop the autokarma push. This will be solved only once Bodhi2 is in place //and// they implement taskotron results collection (i.e. they define abidiff to be an essential check, they listen for every result reported by taskotron for this check, and automatically disable autokarma once we find a failed build that also is currently proposed in some update in Bodhi).

Actually, that's not quite true - bodhi1 has code that disables autokarma push when it sees comments from taskotron saying that something failed.

Alternatively, we need to implement the bodhi_update trigger in taskotron-trigger, and then implement the new "latest stable" downloading features in the bodhi directive. However, that seems to be quite complicated, and it also involves writing a wrapper to run the abidiff tool individually on every Koji build from that update and then collecting results. From the quick glance, I think the first approach mentioned would be much more suitable for this abi diffing use case.

Yeah, I think it makes more sense for abi checking to be run on every build instead of waiting for the update to be created. ABI checking could be done on a per-update basis and reported as a run with multiple results in resultsdb but I agree with @kparal that it makes more sense to run it on each koji build instead of for every bodhi update
* simpler implementation (less hacking build extracting into the per-update task)
* provides earlier feedback to maintainers - letting them know about abi breakage as soon as it happens in koji instead of waiting for an update to be created
* has the potential to work in rawhide (check for unannounced ABI breakage, possibly. not sure how that'd work, though)

As far as reporting goes, another thought would be to collect per-build results at update creation time in a task that's triggered on update creation/change. This task would collect the latest results for each build in the update and come up with a global pass/fail, reporting as something like "build checks for update $UPDATEID". Another advantage here is that we'd be future-proofing a bit by associating future per-build tasks with updates before they're written :) Bodhi2 should generate update ids on update creation (instead of push to testing like in bodhi1) so we shouldn't have as much complexity in saving results for updates as we've had in the past.

As far as I know, any gating is going to be done at the bodhi level for the foreseeable future and we'll have to get important results associated with the update somehow - whether that's through hacking together a bunch of tasks that pretend to run on the update itself or finding a way to associate builds with their updates.

Pushing forward with the resultsdb-next idea (using graphs to store relationships between items) would be an "option" - not practical and not my first choice given the amount of effort it would require but it has the potential to solve this problem :)

After discussing this with @tflink on IRC, we reached the conclusion that having this as a Koji build task makes the most sense. As for reporting, initially just a manual inspection in ResultsDB would work, together with fedmsg notifications we plan to have implemented soon. The abidiff check needs to mature anyway, we don't want to alert all maintainers about the results immediately, until it is properly tested. In the long term view, we need to talk to Bodhi2 developers whether a solution from their side is possible in a reasonable timeframe, or we'll need to implement the koji->bodhi reporting bridge that was described in #490#6089.

We should be ready now for abipkgdiff :-)

I spoke with @dodji yesterday and abipkgdiff should be ready for use.

If we ended up writing the task files for abipkgdiff, here's a blog post showing how to use it:
http://developerblog.redhat.com/2016/02/04/abipkgdiff-ensuring-abi-compliance-for-shared-elf-library-packages/

The abidiff check was deployed to production, if there are any issues please open a new ticket.