This is getting worse, and it's making koschei status basically useless for me. If you look at the Rust SIG packages, basicall all failures reported for EPEL 9 Next are false positives for packages that don't even have epel9-next branches: https://koschei.fedoraproject.org/groups/rust-sig

I'm not sure I understand where the problem is here.

epel9-next is a complemetary thing on top of epel9.

we have to inherit the epel9-build tag in order to build anything.

I'll try and look more at it next week, in the mean time perhaps @tdawson or @carlgeorge can see whats going on ?

Should koschei only detect builds from epel9-next target and not from epel9 target then?

Because at the moment koschei attempts to track packages in epel9-next even if they have no epel9-next branch and have never been built for epel9-next.

Yeah, I would think so...

@mizdebsk perhaps you could chime in here... what needs adjusting from the koschei point of view?

First of all, dist-git branches are irrelevant to Koschei. Koschei supports arbitrary branching by not making any assumptions about dist-git branch names or their existence. Instead, Koschei respects information from Koji tag package list (package names allowed for the tag) as well as Koji tag listings (builds tagged with the tag).

There are several Koji tags for EPEL 9 Next, notably "epel9-next" and "epel9-next-build". When a package is allowed for "epel9-next" tag then it means that the package is allowed to be released as part of EPEL 9 Next. When a package is allowed for "epel9-next-build" tag then it means that the package is allowed to be part of EPEL 9 Next buildroot.

Like for any other release, Koschei for EPEL 9 Next is configured to follow the build tag -- "epel9-next-build", but this can be adjusted to "epel9-next" or any other tag.

Before getting into details about Koji tag inheritance and ways to configure it, I created two collections in staging Koschei - EPEL 9 NEXT I and EPEL 9 NEXT II, which are configured differently (EPEL 9 NEXT I config vs EPEL 9 NEXT II config) and therefore see different packages (as of writing this message, 5932 vs 623 packages). Staging Koschei can be used to inspect what effect would have changing tag from "epel9-next-build" to "epel9-next".

I can adjust configuration of production Koschei to match "EPEL 9 NEXT II" in staging, if that's expected or better. In this case no action from Release Engineering will be needed.

I would expect that a package that has never been built for epel9-next wouldn't be tracked in EPEL 9 Next in koschei ... so the "EPEL 9 Next II" configuration looks much more reasonable to me.

I think II is reasonable too.

Isn't it useful to know that a package is failing to build in epel9-next even if it has not ever been built there? Isn't that an indication that a fix will need to be made for it soon? Or am I missing something?

Theoretically that might be useful, but it doesn't work right.

If you look at the rust-sig group on koschei:
https://koschei.fedoraproject.org/groups/rust-sig

All failures reported for epel9-next (both failure to resolve dependencies and failing builds) are false positives. I have no idea how these failures are computed, but if I tried to build for centos-stream+epel-next-9-x86_64 locally, the builds should succeed without any issues.

Whats the status here? @mizdebsk you want to go ahead and adjust prod and we can close this?

Note that I've now also encountered strange issues with building epel9-next in koji, so this might not only affect koschei.

For example, in the following scenario

• package foo, version 1 does not build in epel9 built in epel9-next instead
• epel9 is updated and allows package foo version 1 to be built there
• package version 1 is built in epel9
• new version 2 of the package is released
• package version 2 is built in epel9
• packages depending on "foo" in epel9-next pull in version 1 from epel9-next instead of version 2 from epel9, possibly causing issues

I don't know if this was a tagging issue that only affected one package though.

Ok, this is very strange. I submitted some builds to epel9-next that were missing from epel9-next but present in epel9 (i.e. "build foo version 2 in epel9-next" in the scenario of the previous comment), and now dependencies in koschei are resolving again. So it appears that for some reason, the old versions from epel9-next indeed overrode the newer version from epel9 ... :(

Yeah, I am not sure we can really fix this.

What should ideally happen is that when foo1 is built in epel9, all foo versions in epel9-next should be untagged.

I'm not sure we can automate this, but it is possible to do manually when you hit this issue.

Hm, yes, this is not obvious how to fix it. For now, if a package has both an epel9 and an epel9-next branch, I've started building updates in both branches ... (even though that's not supposed to be necessary, if I understood the responses correctly when I asked about this in the Fedora EPEL channel). It's not a big deal.

I'm still not sure if it's a good idea to track packages in EPEL9-Next that don't have an epel9-next branch ... but I think that should be addressed by switching koschei to the proposed "EPEL9-Next II" configuration?

Yes, I think once koschei is adjusted that should fix that part... but not sure how to solve the other issue. I can bring it up at the epel meeting later today...