@puiterwijk @kevin

Currently the task simply uses the standard system dnf repos. If the package targets stable updates, it compares it against main+updates repos. If the package targets updates-testing, it compares it against main+updates+updates-testing repos. Therefore if there are some packages tagged but not yet pushed, they are not taken into account. If you think this should work differently, can you describe the exact workflow? Please note that downloading packages just to create a repodata out of them is pretty expensive, so we try to avoid that every time it's possible.

Ah. Then in that case I know what the problem was, yes: it's wrong when composes are failing. As F28 composes failed for a long time, many packages which were tagged f28 were not in the repos.

I think it'd be best if the tests ran against the set of packages tagged with the appropriate stable tag(s) for the release, but I don't know how hard that would be.

OK, I'll try to look at it when I have time.

How about, for the few repos that you would need, we create koji dist repos automatically from them whenever something's changed?
That means that Koji will automatically generate the repodata and offer it in side-repos, meaning you would just need to reconfigure the repos you'd look at.

Note that generating these might still take some time for huge tags (10 minutes or so), but having koji build and maintain the repodata would be a lot more efficient than downloading all packages and doing repo generation locally.

Alternatively, you could look at the already generated fXX-build repos, if that information is indicative enough of rpmdeplint failures?

Having Koji build the repodata would be extremely awesome for us. It could also fix the multilib issues we have (see #6). I'll have a look at fXX-build repos, whether we could use them.

The build repos aren't necessarily correct as they include packages for which a buildroot override has been submitted, but which are not 'stable' in any sense that maps to "something a user of the standard repos will definitely get".

Right, I did not compare what rpmdep needed for its tests.
Dist repos should work just fine though, I think.

Are we going around in circles here? :P The problem is dist repos aren't always just fine, when composes are failing.

This isn't really an issue for stable releases, but it is for Branched, because we need the overall nightly Branched compose to succeed before anything that uses the official repos will see packages that were 'pushed stable' so far as Bodhi and the Koji tags are concerned. If the nightly composes are failing, we get a backlog of packages that are 'stable' so far as Bodhi is concerned, and are tagged f28 or whatever, but don't get used in this test.

That's exactly what happened here: this update 'failed' the test because all the packages in it were built against OpenCV 3.4 (that's the whole point of the update), but because we hadn't had a successful Branched compose with the OpenCV 3.4 package in it - even though it was tagged f28 - rpmdeplint was using OpenCV 3.3.