Koschei already supports this feature for more than 2 years. For more details see Koschei issue 113.

The workflow is:

• user builds one or more packages (eg. in Koji side tag, or in Copr) and creates YUM repo out of built packages
• user asks Koschei to test the above YUM repo
• Koschei scratch-builds all or selected (upon user choice) Fedora packages that build-depend on any package from the YUM repo
• Koschei reports back to the user whether packages from the YUM repo affect (break or fix) any packages from distribution

Thanks @mizdebsk - that's very helpful - the main challenge is that currently the CI scratch builds in Koji and there is no repo.

It shouldn't be hard to create a repo out of scratch build. The main reason Koschei needs a repo and not a single (scratch) build is that I want to be able to test multiple changes together. Single-package changes are a special case of that.

See also https://github.com/msimacek/koschei/issues/293

See also https://pagure.io/stewardship-sig/blob/master/f/scripts/review_pr.py

This basically creates a Copr repo for a Pull Request and rebuilds everything dependent in it.

I am currently working on making Koschei support gating side tags, which I think will resolve this issue.

So far planned workflow is:

• User creates side tag in Koji using fedpkg request-side-tag command
• Koschei notices that side tag was created and starts tracking it
• User builds packages in side tag
• As builds in side tag are done, Koschei re-resolves dependencies for all packages in side tag (including packages inherited from base tag) and submits scratch builds to check whether packages are still buildable against packages in side tag
• Koschei provides API that makes it possible to compare side tag with base tag, which CI systems can use to check whether builds done in side tag do not break anything

This is indeed useful but won't resolve the issue, unless it can build in the side tag from a PR.

@mizdebsk Checking in to see if we have any updates on Koschei supporting gating side tags

The most important code changes have already been implemented upstream in PR 308 and PR 314. The code is deployed in staging.

Koschei can emit CI messages to message bus, examples from staging message bus: test running, test passed, test failed, test error.

@jimbair so what are the next steps here from the perspective of OSCI team?

In Python Maint team, we have identified that within the "updating a library package to a newer version" task, the most tedious and time-consuming subtask is to check the impact of the update. Currently, we do this:

1. Create a copr repository and build the new version of the package there.
2. repoquery everything that (transitively) BuildRequires the package and manually check if something does not require e.g. < NEW_VERSION.
3. Build everything from (2) in copr from (1).
4. Check all failures and compare them with Koshei status or a control copr.

I have opened this ticket almost 3 years ago but unfortunately, we still need to do this manually.

Should we write our own tooling to automate this, or is this something the CI is likely to have eventually?

@bookwar were Koschei plans pushed back or we are lacking just human resources to get it going?

@churchyard to me feels like if we have a service that can do this, feels a step back and waste of time reimplement it :( Even investing time in solution just for your team maybe it would be easier to contribute to Fedora CI or Zuul to get this done for more folks?

to me feels like if we have a service that can do this, feels a step back and waste of time reimplement it :(

Well, obviously. But doing it manually is worse.

Even investing time in solution just for your team maybe it would be easier to contribute to Fedora CI or Zuul to get this done for more folks?

Better? Yes. Easier? I doubt that.

@mizdebsk is it possible to run koschei as a CI test instead of using it as a service? We now have a fairly straight forward way how to run generic tests. WDYT?

@mizdebsk is it possible to run koschei as a CI test instead of using it as a service?

Sure, a test can do anything. Theoretically a test can install and configure Koschei with dependencies (Koji, PostgreSQL) and run it to get results. But that's not how Koschei was intended to be used.

The intended way to use Koschei in CI is as CI system - Koschei can send messages conformant with Fedora CI messages schemas (build group test queued, running, complete, error). It can test artifacts of types "koji-build-group" and "brew-build-group". This functionality is already implemented and deployed, but not enabled in production yet. However this functionality is enabled in staging - staging Koschei already acts as a CI system.

Testing of "koji-build" and "brew-build" artifacts is possible by creating build groups (side tags) consisting of a singe build (Koschei requires Koji tag with test subject builds tagged because it runs scratch builds of other packages in this side tag). Creating side tag and tagging builds needs to be done by requestor of a test run - Koschei does not create Koji side tags by itself. I'm open to adding support for more artifact types.

For now triggering testing is not implemented. One needs to manually tell Koschei that a side tag needs to be tested. But triggering could be implemented easily once it's decided how Koschei tests should be triggered. I would prefer Koschei not to rely on {brew,koji}-build-group.build.complete messages for triggering, but to poll Koji for side tags and test them, even before they are considered finished.

.

Building a build in side-tag assumes the change has been already committed to dist git. At that point, we would have already tested the impact.

Right, the description is about on PR CI :facepalm:

@mizdebsk so we are looking @ that first comment of yours:

https://pagure.io/fedora-ci/general/issue/45#comment-606344

We would need Koschei to behave like a service instead, we create the yum.repo from the scratch build built from PR (already done by Zuul) and submit it to Koschei and wait for results, report to the user ...

Seems to me for this use case would make more sense, the wrapper around calling Koschei could be a simple tmt test that the user enables via Zuul and Testing Farm.

"Testing arbitrary YUM repos" feature that I briefly described in my first comment was recently removed from Koschei for two reasons:

1. It depended on on Copr, which did not pass Request For Resources and therefore is considered as a third-party service not supported by Fedora Infrastructure - Koschei as a supported service can't rely on third-party service that has no SLE/SLA defined.
2. No one seemed interested in using it and unused code only increased maintenance cost of Koschei.

Testing packages that were not yet built in Koji is no longer in scope for Koschei project.

until we decided to use it, it got dropped :) Thanks for the update, no need to keep unused code, makes sense.

@churchyard I need to think about how to implement this, can you share with me maybe some example how to build a scratch build against another scratch build in mock?? To get an idea how it is done?

My current idea is:

• Have a Zuul task that generates a git repo which generates tmt plans per all reverse dependencies as a git repo
• Pass the repo to the Zuul rpm-tmt-test role and let Testing Farm handle the parallelization

@churchyard I need to think about how to implement this, can you share with me maybe some example how to build a scratch build against another scratch build in mock?? To get an idea how it is done?

If you wish to build in mock having the tested package available there, you could use something like: https://src.fedoraproject.org/rpms/pyproject-rpm-macros/blob/rawhide/f/tests/mocktest.sh (obviously, this is specific to having spec files in the tests folder, it should be even easier for actual Fedora packages).

If you wish to build scratch builds (with Koji) that could see another scratch builds, I don't think that's possible.

Considering Koschei is out of the picture, I believe that using Copr for this particular check makes perfect sense, and reimplementing it using mock is a waste of resources. I can even share some commands that help with this enormously. The hard part is analyzing the failures and comparing them with actual Fedora.

Have a Zuul task that generates a git repo which generates tmt plans per all reverse dependencies as a git repo

I don't understand this at all, sorry. Too many git repos in that sentence. Could you rephrase a bit?

Even investing time in solution just for your team maybe it would be easier to contribute to Fedora CI or Zuul to get this done for more folks?

If it helps, we might be able to allocate some resources to hack a proof of concept that works when used locally so you can then take it and hook it up to the CI machinery.

Considering Koschei is out of the picture, I believe that using Copr for this particular check makes perfect sense, and reimplementing it using mock is a waste of resources. I can even share some commands that help with this enormously. The hard part is analyzing the failures and comparing them with actual Fedora.

+1 for sharing something via copr then, thanks for the pointers, if we can use copr for this, it simplyfies stuff, it will take care of parallel rebuilding ...

I guess the output could be just the links to copr logs building each dep ...

Maybe we can also have a short call to specify the MVP

If it helps, we might be able to allocate some resources to hack a proof of concept that works when used locally so you can then take it and hook it up to the CI machinery.

That would be great, this would definitely help ... I am happy to have a call, just ping me somewhere, if needed

We've talked to @mvadkert over a video meeting today. I've demoed what we do in Copr and noted down some commands into https://hackmd.io/@python-maint/rJSm5WC9Y

Yep, thanks, we are tracking this in downstream TFT-912 and @guyinger will be working on it early next year ...

Just a note that https://gitlab.com/fedora/packager-tools/mass-prebuild exists now and the CI job "just" needs to hook that in. In the meantime, we will continue using it manually.

oh wow, I guess we can drop our silly too long coming patch then and work with this. Thanks for sharing!

oh wow, I guess we can drop our silly too long coming patch then and work with this. Thanks for sharing!

Don't hesitate to ping me if need be. I'd be happy to help getting the mass pre-builder integrated and improved to fit the needs.
For the record, as of today we use COPR as a back-end, but I'm planning to use Koji and likely a mock-based back-end in 2023.

@fberat will do, thanks for the offer.

Currently, I do not see we will get to it this year :(

I will submit a downstream issue so we revisit in Q1/2023.