fedora-infrastructure

#6429 pagure-sync-bugzilla script has lots of errors
Closed: Fixed Opened by kevin.

Closed: Fixed
Reopen Issue

The pagure-sync-bugzilla script is not working right/spewing a lot of errors.

There's are 16,809 lines like: 

No results for {u'custom_keys': [], u'description': u'The bauble rpms', u'parent': None, u'date_modified': u'1501866253', u'access_users': {u'admin': [], u'commit': [], u'ticket': [], u'owner': [u'orphan']}, u'namespace': u'rpms', u'priorities': {}, u'id': 1180, u'access_groups': {u'admin': [], u'commit': [], u'ticket': []}, u'milestones': {}, u'user': {u'fullname': u'orphan', u'name': u'orphan'}, u'date_created': u'1501866253', u'fullname': u'rpms/bauble', u'close_status': [], u'tags': [], u'name': u'bauble'} in PDC.
No results for {u'custom_keys': [], u'description': u'The docker-machine rpms', u'parent': None, u'date_modified': u'1507272874', u'access_users': {u'admin': [u'fpokorny', u'jchaloup'], u'commit': [], u'ticket': [], u'owner': [u'lsm5']}, u'namespace': u'rpms', u'priorities': {}, u'id': 2640, u'access_groups': {u'admin': [], u'commit': [], u'ticket': []}, u'milestones': {}, u'user': {u'fullname': u'Lokesh Mandvekar', u'name': u'lsm5'}, u'date_created': u'1501867109', u'fullname': u'rpms/docker-machine', u'close_status': [], u'tags': [], u'name': u'docker-machine'} in PDC.
No results for {u'custom_keys': [], u'description': u'The gap-pkg-tomlib rpms', u'parent': None, u'date_modified': u'1501867812', u'access_users': {u'admin': [], u'commit': [], u'ticket': [], u'owner': [u'jjames']}, u'namespace': u'rpms', u'priorities': {}, u'id': 4149, u'access_groups': {u'admin': [], u'commit': [], u'ticket': []}, u'milestones': {}, u'user': {u'fullname': u'Jerry James', u'name': u'jjames'}, u'date_created': u'1501867812', u'fullname': u'rpms/gap-pkg-tomlib', u'close_status': [], u'tags': [], u'name': u'gap-pkg-tomlib'} in PDC.
No results for {u'custom_keys': [], u'description': u'The golang-github-oschwald-maxminddb-golang rpms', u'parent': None, u'date_modified': u'1501868752', u'access_users': {u'admin': [], u'commit': [], u'ticket': [], u'owner': [u'decathorpe']}, u'namespace': u'rpms', u'priorities': {}, u'id': 5561, u'access_groups': {u'admin': [], u'commit': [], u'ticket': []}, u'milestones': {}, u'user': {u'fullname': u'Fabio Valentini', u'name': u'decathorpe'}, u'date_created': u'1501868752', u'fullname': u'rpms/golang-github-oschwald-maxminddb-golang', u'close_status': [], u'tags': [], u'name': u'golang-github-oschwald-maxminddb-golang'} in PDC.

Wild guess: perhaps it shouldn't be passing rpms/name but only name? (since it has namespace in a different variable).

@ralph or @mprahl any ideas?

Per @mprahl suggestion I ran this with a DRY_RUN=True and it has a lot of output. Here's some from 1 package: 

Querying https://pdc.fedoraproject.org/rest_api/v1/component-branches/ {'type': 'rpm', 'global_component': u'docker-machine'}
No results for {u'custom_keys': [], u'description': u'The docker-machine rpms', u'parent': None, u'date_modified': u'1507635108', u'access_users': {u'admin': [u'fpokorny', u'jchaloup'], u'commit': [], u'ticket': [], u'owner': [u'lsm5']}, u'namespace': u'rpms', u'priorities': {}, u'id': 2640, u'access_groups': {u'admin': [], u'commit': [], u'ticket': []}, u'milestones': {}, u'user': {u'fullname': u'Lokesh Mandvekar', u'name': u'lsm5'}, u'date_created': u'1501867109', u'fullname': u'rpms/docker-machine', u'close_status': [], u'tags': [], u'name': u'docker-machine'} in PDC.
Querying https://src.fedoraproject.org/api/0/rpms/docker-machine/watchers
Querying https://apps.fedoraproject.org/mdapi/rawhide/srcpkg/docker-machine
Querying https://pdc.fedoraproject.org/rest_api/v1/component-branches/ {'type': 'rpm', 'name': 'epel7', 'global_component': u'docker-machine'}
Querying https://pagure.io/releng/fedora-scm-requests/raw/master/f/rpms/docker-machine
Querying https://pdc.fedoraproject.org/rest_api/v1/component-branches/ {'type': 'rpm', 'name': 'master', 'global_component': u'docker-machine'}

So looks like the 'global component' call to pdc is failing, but the branches are working?

@kevin, running something similar to what the script is doing works for on my laptop.

import requests
rv = requests.get('https://pdc.fedoraproject.org/rest_api/v1/component-branches/', params={'type': 'rpm', 'global_component': u'docker-machine'})
print(rv.json())

I'll think about it some more.

OK - this is "fine" in most cases. It happens when we ask PDC if something is retired in el6 that was never in el6 to begin with.

Here, I demoted the print statement to only show during --debug runs: https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=48db25c0e6c2cfc27d49ccac5b78db41fa710fdf

So, we need to see if it completes ok now after fixing a few things?

@kevin yeah, sounds good here.

Last time, we were watching ruby to see if the script fixed it. Do you know of another "problem" component we can watch for?

I'm afraid I do not know any more for sure. ;(

It seems like the errors are now gone and it does complete (as I see it sending the mismatched account emails again).

@kevin reported in channel that python-lxml is also borked. Will take a look.

Metadata Update from @ralph:
- Issue assigned to ralph
- Issue tagged with: factory2

I think @pingou and I are going to hack on this tomorrow.

ok. So, there's now a number of 404 errors on random packages. I think this happens when haproxy marks the service as down and puts up a unavailable page, it then hits the proxy and gets a 404.

RuntimeError: The connection to "https://src.fedoraproject.org/api/0/rpms/erlang-luke/watchers" failed with the status code 404 and output "<!DOCTYPE html>
RuntimeError: The connection to "https://src.fedoraproject.org/api/0/rpms/fest-swing-junit/watchers" failed with the status code 404 and output "<!DOCTYPE html>
RuntimeError: The connection to "https://src.fedoraproject.org/api/0/rpms/gpxe/watchers" failed with the status code 404 and output "<!DOCTYPE html>
RuntimeError: The connection to "https://src.fedoraproject.org/api/0/rpms/fest-swing-junit/watchers" failed with the status code 404 and output "<!DOCTYPE html>
RuntimeError: The connection to "https://src.fedoraproject.org/api/0/rpms/mediawiki119-HTTP302Found/watchers" failed with the status code 404 and output "<!DOCTYPE html>
RuntimeError: The connection to "https://src.fedoraproject.org/api/0/rpms/python26-jinja2/watchers" failed with the status code 404 and output "<!DOCTYPE html>
RuntimeError: The connection to "https://src.fedoraproject.org/api/0/rpms/perl-CSS-Squish/watchers" failed with the status code 404 and output "<!DOCTYPE html>
RuntimeError: The connection to "https://src.fedoraproject.org/api/0/rpms/sqljet/watchers" failed with the status code 404 and output "<!DOCTYPE html>
RuntimeError: The connection to "https://src.fedoraproject.org/api/0/rpms/ocaml-camlimages/watchers" failed with the status code 404 and output "<!DOCTYPE html>
RuntimeError: The connection to "https://src.fedoraproject.org/api/0/rpms/zimlib/watchers" failed with the status code 404 and output "<!DOCTYPE html>

These all load ok here now, so I suspect this is just load/random. Perhaps the script could retry 404's after a small backoff?

Also, the script now bombs out with:

pdc-dev-list@redhat.com has no bugzilla email, valid account?
Traceback (most recent call last):
File "/usr/local/bin/pagure-sync-bugzilla.py", line 838, in
pkgInfo['cclist']
File "/usr/local/bin/pagure-sync-bugzilla.py", line 325, in add_edit_component
owner = self._get_bugzilla_email(owner)
File "/usr/local/bin/pagure-sync-bugzilla.py", line 306, in _get_bugzilla_email
return self.userCache[username]['bugzilla_email'].lower()
KeyError: 'pdc-dev-list@redhat.com'

I removed the pdc-dev-list@redhat.com override in the fedora-scm-requests repo for now.
Can someone make that bugzilla account (hopefully someone with access to the list so they can moderate the email from it and set its password).

Then I hit another group that was causing problems (@gitabrt) but that was a odd group that had been manually added, so we just removed it.

The script now runs again, but there are still some 404's. Perhaps the script could catch that and retry those a few times (as it seems transitory).

Current status: rewriting the script to use cached json dumps from both src.fp.o and pdc.fp.o. This should speed up the script dramatically. It's so slow today that it's impossible to work on. :(
Edited by ralph

This is in place in staging now.

https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=b3be4754777d88a202cadd6179156e943d932b99

Will give it a few runs there, in staging. If it looks okay to you @kevin we could try it in prod. I'm going afk for the holidays at the end of this week though, so maybe we should wait until the new year.

If you're able, and it runs ok in stg, it might be nice to run it in prod tomorrow and if it runs ok there, enable it...

if it causes problems over the holidays we can always revert to the old one.

OK, I've got this prepped in commit b8a1e0e0e0c968da00968231fd3c72ed8a71542c which I haven't pushed yet. Will push it in the morning and give it a spin.

I gave it multiple --debug runs this morning and everything seems fine. Giving it a real run now.

It looks all good. I've run it ~4 times today with no errors.

The only last change to consider is to run it more frequently, like this:: 

diff --git a/roles/distgit/pagure/tasks/main.yml b/roles/distgit/pagure/tasks/main.yml
index bb5935c..b1ab18c 100644
--- a/roles/distgit/pagure/tasks/main.yml
+++ b/roles/distgit/pagure/tasks/main.yml
@@ -232,12 +232,12 @@
   - pagure
   - bugzilla
-- name: Configure cron job for a daily pagure-sync-bugzilla.py script run
+- name: Configure cron job for a pagure-sync-bugzilla.py script run
   cron:
     name: pagure-sync-bugzilla
     user: root
     minute: 0
-    hour: 18
+    hour: "*/4"
     job: /usr/local/bin/lock-wrapper pagure-sync-bugzilla "/usr/local/bin/pagure-sync-bugzilla.py"
     cron_file: pagure-sync-bugzilla
     state: present

But, maybe we wait a few days and then try that.

@kevin, would you like to close this or keep it open a little longer?

OK - going to close for now. Please revert and re-open if it gets back into trouble.

Metadata Update from @ralph:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata

factory2

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.