#7935 Nightlies (Rawhide and Branched) not imported to PDC
Opened 4 months ago by adamwill. Modified a month ago

I just noticed that neither Nightly nor Branched composes seem to have been imported to PDC since March. The last nightly to make it to PDC was Fedora-Rawhide-20190306.n.1 . No Rawhide or Branched compose since has appeared in PDC.

This is a big problem for fedfind and things that depend on it (e.g. it prevents check-compose from discovering the 'previous' compose for comparison).


There is a nightly cron job that is supposed to run and audit this, but it seems to have been broken since... 2016 or so.

I've fixed and and it should run and output to the releng-cron list the missing ones.

However, it seems to be failing with tracebacks:

May 24 11:31:56 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: BeanBagException: Bad response code: 400,...a bunch of stuff...

May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: Traceback (most recent call last):
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/moksha/hub/api/consumer.py", line 207, in _do_work
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: self.consume(message)
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/pdcupdater/consumer.py", line 75, in consume
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: pdcupdater.utils.handle_message(pdc, self.handlers, msg)
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/pdcupdater/utils.py", line 477, in handle_message
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: handler.handle(client, msg)
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/pdcupdater/handlers/compose.py", line 57, in handle
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: self._import_compose(pdc, compose_id, compose_url)
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/pdcupdater/utils.py", line 675, in wrapper
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: return function(*args, **kwargs)
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/pdcupdater/handlers/compose.py", line 152, in _import_compose
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: image_manifest=images,
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/pdc_client/__init__.py", line 347, in __call__
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: return self.client(*args, **kwargs)
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "<string>", line 1, in <lambda>
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/beanbag/namespace.py", line 131, in fn
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: *args, **kwargs)
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/beanbag/url_v1.py", line 102, in call
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: return self.make_request(path, verb, kwargs, body)
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/beanbag/url_v1.py", line 155, in make_request
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: "Bad response code: %d, %r %r %r" % (r.status_code, params, body, r.content))
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: BeanBagException: Bad response code: 400

@cverna can you take a look? or @ralph ?

perhaps we should just bite the bullet and move to fpdc ?

at a guess it may not be able to import composes that have now been garbage collected, because they've been garbage collected? or is it only trying to import more recent ones that are still there?

So, the audit ran:

0 extra entries in PDC unaccounted for 540 entries absent from PDC

We could try and import them all, but it needs the name of each one and the audit has: "- (plus 440 more... truncated.)"

Do we know if it's working now, or still not? That would be good to fix before we worry about importing the old ones.

Metadata Update from @kevin:
- Issue priority set to: Waiting on Assignee (was: Needs Review)

4 months ago

No Rawhide compose has shown up in PDC since the ticket was filed, so no, it's not working.

You can just look here: https://pdc.fedoraproject.org/compose/

all composes show up there.

Metadata Update from @kevin:
- Issue tagged with: backlog

a month ago

Yep.

Additionally the audit runs are failing with:

No handlers could be found for logger "moksha.hub"
[2019-09-11 18:00:02][pdcupdater.commands    INFO] Performing audit for ModuleStateChangeHandler
[2019-09-11 18:00:02][pdcupdater.commands    INFO] Performing audit for RetireComponentHandler
[2019-09-11 18:00:02][pdcupdater.handlers.retirement    INFO] Looking up all branches from PDC.
[2019-09-11 18:00:02][requests.packages.urllib3.connectionpool    INFO] Starting new HTTP connection (1): pdc-web01.stg.phx2.fedoraproject.org
[2019-09-11 18:00:03][requests.packages.urllib3.connectionpool    INFO] Starting new HTTPS connection (1): src.fedoraproject.org
Traceback (most recent call last):
  File "/usr/bin/pdc-updater-audit", line 9, in <module>
    load_entry_point('pdc-updater==0.9.3', 'console_scripts', 'pdc-updater-audit')()
  File "/usr/lib/python2.7/site-packages/pdcupdater/commands.py", line 72, in audit
    results[name] = handler.audit(pdc)
  File "/usr/lib/python2.7/site-packages/pdcupdater/handlers/retirement.py", line 123, in audit
    namespace=self._pdc_to_namespace(branch['type']),
  File "/usr/lib/python2.7/site-packages/pdcupdater/handlers/retirement.py", line 104, in _pdc_to_namespace
    .format(pdc_type))
ValueError: The PDC type "flatpak" is not supported

@cverna has been working on a pdc replacement, perhaps he has ideas here.

Yep.
Additionally the audit runs are failing with:
No handlers could be found for logger "moksha.hub"
[2019-09-11 18:00:02][pdcupdater.commands INFO] Performing audit for ModuleStateChangeHandler
[2019-09-11 18:00:02][pdcupdater.commands INFO] Performing audit for RetireComponentHandler
[2019-09-11 18:00:02][pdcupdater.handlers.retirement INFO] Looking up all branches from PDC.
[2019-09-11 18:00:02][requests.packages.urllib3.connectionpool INFO] Starting new HTTP connection (1): pdc-web01.stg.phx2.fedoraproject.org
[2019-09-11 18:00:03][requests.packages.urllib3.connectionpool INFO] Starting new HTTPS connection (1): src.fedoraproject.org
Traceback (most recent call last):
File "/usr/bin/pdc-updater-audit", line 9, in <module>
load_entry_point('pdc-updater==0.9.3', 'console_scripts', 'pdc-updater-audit')()
File "/usr/lib/python2.7/site-packages/pdcupdater/commands.py", line 72, in audit
results[name] = handler.audit(pdc)
File "/usr/lib/python2.7/site-packages/pdcupdater/handlers/retirement.py", line 123, in audit
namespace=self._pdc_to_namespace(branch['type']),
File "/usr/lib/python2.7/site-packages/pdcupdater/handlers/retirement.py", line 104, in _pdc_to_namespace
.format(pdc_type))
ValueError: The PDC type "flatpak" is not supported

@cverna has been working on a pdc replacement, perhaps he has ideas here.

I could try to give it a look, but I realistically can't spare time on it this week and next week I am afk.

Login to comment on this ticket.

Metadata