#7273 Dist-repo failed to generate due to missing signed RPMs
Closed: Fixed a year ago Opened a year ago by mizdebsk.

dist-repo-regen.py script that is used to prevent infra dist-repos from expiring (implemented in #7246) failed to generate dist repo for tag f27-infra-stg with the following error: GenericError: Packages missing from the filesystem. See missing_files.log.. Failed task ID is 30000710. The log indicates that task failure is caused by missing signed RPMs.

What do we want to do about this? I can see some options:

  • the script would check whether all signed RPMs are available and if not, request writing them (with API call equivalent of koji write-signed-rpm) - the script runs as bodhi Koji user and therefore has admin permission which is sufficient to write signed RPMs
  • notify Koji admins so that they can write signed RPMs manually (and wait for next script run to re-generate it)
  • nothing; ignore the issue

So far dist-repo failed only for f27-infra-stg, all other tasks submitted by the script were successful:


So, the reason this happened was a newer bodhi build was tagged in and signed, then this builds written out rpms were garbage collected, then the newer build was moved to f27-infra, making this older build the last tagged one, so it tried to add it to the repo, but there were no written out signed copies.

So, yeah, we could just call write-signed-rpm here and it would handle this case, but I wonder if we have some better way to untag all the old builds when one is promoted. I guess we could just try and remember anytime we move a package from the staging tag to prod, we also untag all the old builds of that same package.

Thoughts?

Metadata Update from @kevin:
- Issue priority set to: Next Meeting (was: Needs Review)

a year ago

On Tue, 2018-10-02 at 18:21 +0000, Kevin Fenzi wrote:

I guess we could just try and remember anytime we move a package from
the staging tag to prod, we also untag all the old builds of that
same package.=20
=20
Thoughts?

If we decide to go that route, we should document it at

https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/infr=
a-repo.html

This was discussed on the last infrastructure meeting and there was consensus that it should be fine to try to write missing signed RPMs before attempting to generate dist repo.

Metadata Update from @mizdebsk:
- Issue priority set to: Waiting on Assignee (was: Next Meeting)

a year ago

Another interesting case of dist-repo failure: task 30171086 and task 30171087 - KeyError: 'split_debuginfo'

This might be due to koji 1.16.0? (which we never deployed, but we did move to 1.16.1...
from the 1.16.0 changes:

Split debuginfo for dist repos
PR: https://pagure.io/koji/pull-request/914

Dist repos can now be generated with debuginfo files split into a separate repo. The behavior is controlled by passing the --split-debuginfo option to the dist-repo subcommand.

When this option is in effect, the main repo will be in the normal location. The debuginfo repo will be in the debug subdirectory. So, you will see a directory structure like:

Packages/
repodata/
debug/
debug/repodata

Regardless of the split, all the rpms are located in the top level Packages directory.

But I am not sure why it would cause a keyerror...

The 'split_debuginfo' was fixed in 0fe9715

ok. so we are all done here? Please re-open if there's anything further to do...

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

a year ago

This is still not fixed.

Metadata Update from @mizdebsk:
- Issue status updated to: Open (was: Closed)

a year ago

This is still not fixed.

Are you sure it's the same root cause? We had an issue until yesterday about tag2distrepo not running properly.
I believe this has been fixed now so it could be that this is fixed as well :)

Yes, I'm sure it's the same issue and that it's still not fixed. I will try to fix it this or next week.

Metadata Update from @mizdebsk:
- Issue assigned to mizdebsk

a year ago

Metadata Update from @mizdebsk:
- Issue priority set to: Waiting on External (was: Waiting on Assignee)

a year ago

There has been no upstream response in 7 days, so I'm going to apply the patch downstream.

Metadata Update from @mizdebsk:
- Issue priority set to: Waiting on Assignee (was: Waiting on External)

a year ago

This should be fixed for all tags except f25-infra.

python-tag2distrepo (fedmsg listener) was fixed in build python-tag2distrepo-1-0.9.fc29, dist-repo-regen (cron job) was fixed in ansible commit 9ead67d. Both fixes are deployed in production.

Dist repo for f25-infra failed to generate, see task 31191166. Since we shouldn't have any systems deployed on Fedora 25 (I hope) I am ignoring this problem. Instead dist-repos for f25-infra should be disabled.

Metadata Update from @mizdebsk:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

a year ago

Metadata Update from @mizdebsk:
- Issue priority set to: None (was: Waiting on Assignee)

a year ago

Login to comment on this ticket.

Metadata