So, the reason this happened was a newer bodhi build was tagged in and signed, then this builds written out rpms were garbage collected, then the newer build was moved to f27-infra, making this older build the last tagged one, so it tried to add it to the repo, but there were no written out signed copies.

So, yeah, we could just call write-signed-rpm here and it would handle this case, but I wonder if we have some better way to untag all the old builds when one is promoted. I guess we could just try and remember anytime we move a package from the staging tag to prod, we also untag all the old builds of that same package.

Thoughts?

On Tue, 2018-10-02 at 18:21 +0000, Kevin Fenzi wrote:

I guess we could just try and remember anytime we move a package from
the staging tag to prod, we also untag all the old builds of that
same package.=20
=20
Thoughts?

If we decide to go that route, we should document it at

https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/infr=
a-repo.html

That link was supposed to be https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/infra-repo.html but rendered funny due to https://pagure.io/pagure/issue/3811.

This was discussed on the last infrastructure meeting and there was consensus that it should be fine to try to write missing signed RPMs before attempting to generate dist repo.

Another interesting case of dist-repo failure: task 30171086 and task 30171087 - KeyError: 'split_debuginfo'

This might be due to koji 1.16.0? (which we never deployed, but we did move to 1.16.1...
from the 1.16.0 changes:

Split debuginfo for dist repos
PR: https://pagure.io/koji/pull-request/914

Dist repos can now be generated with debuginfo files split into a separate repo. The behavior is controlled by passing the --split-debuginfo option to the dist-repo subcommand.

When this option is in effect, the main repo will be in the normal location. The debuginfo repo will be in the debug subdirectory. So, you will see a directory structure like:

Packages/
repodata/
debug/
debug/repodata

Regardless of the split, all the rpms are located in the top level Packages directory.

But I am not sure why it would cause a keyerror...

The 'split_debuginfo' was fixed in 0fe9715

ok. so we are all done here? Please re-open if there's anything further to do...

This is still not fixed.

This is still not fixed.

Are you sure it's the same root cause? We had an issue until yesterday about tag2distrepo not running properly.
I believe this has been fixed now so it could be that this is fixed as well :)

Yes, I'm sure it's the same issue and that it's still not fixed. I will try to fix it this or next week.

PR submitted: https://pagure.io/releng/tag2distrepo/pull-request/2

There has been no upstream response in 7 days, so I'm going to apply the patch downstream.

This should be fixed for all tags except f25-infra.

python-tag2distrepo (fedmsg listener) was fixed in build python-tag2distrepo-1-0.9.fc29, dist-repo-regen (cron job) was fixed in ansible commit 9ead67d. Both fixes are deployed in production.

Dist repo for f25-infra failed to generate, see task 31191166. Since we shouldn't have any systems deployed on Fedora 25 (I hope) I am ignoring this problem. Instead dist-repos for f25-infra should be disabled.