There is a nightly cron job that is supposed to run and audit this, but it seems to have been broken since... 2016 or so.

I've fixed and and it should run and output to the releng-cron list the missing ones.

However, it seems to be failing with tracebacks:

May 24 11:31:56 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: BeanBagException: Bad response code: 400,...a bunch of stuff...

May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: Traceback (most recent call last):
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/moksha/hub/api/consumer.py", line 207, in _do_work
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: self.consume(message)
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/pdcupdater/consumer.py", line 75, in consume
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: pdcupdater.utils.handle_message(pdc, self.handlers, msg)
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/pdcupdater/utils.py", line 477, in handle_message
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: handler.handle(client, msg)
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/pdcupdater/handlers/compose.py", line 57, in handle
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: self._import_compose(pdc, compose_id, compose_url)
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/pdcupdater/utils.py", line 675, in wrapper
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: return function(*args, **kwargs)
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/pdcupdater/handlers/compose.py", line 152, in _import_compose
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: image_manifest=images,
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/pdc_client/__init__.py", line 347, in __call__
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: return self.client(*args, **kwargs)
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "<string>", line 1, in <lambda>
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/beanbag/namespace.py", line 131, in fn
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: *args, **kwargs)
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/beanbag/url_v1.py", line 102, in call
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: return self.make_request(path, verb, kwargs, body)
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: File "/usr/lib/python2.7/site-packages/beanbag/url_v1.py", line 155, in make_request
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: "Bad response code: %d, %r %r %r" % (r.status_code, params, body, r.content))
May 24 11:35:37 pdc-backend01.phx2.fedoraproject.org fedmsg-hub[3290]: BeanBagException: Bad response code: 400

@cverna can you take a look? or @ralph ?

perhaps we should just bite the bullet and move to fpdc ?

at a guess it may not be able to import composes that have now been garbage collected, because they've been garbage collected? or is it only trying to import more recent ones that are still there?

So, the audit ran:

0 extra entries in PDC unaccounted for 540 entries absent from PDC

We could try and import them all, but it needs the name of each one and the audit has: "- (plus 440 more... truncated.)"

Do we know if it's working now, or still not? That would be good to fix before we worry about importing the old ones.

No Rawhide compose has shown up in PDC since the ticket was filed, so no, it's not working.

You can just look here: https://pdc.fedoraproject.org/compose/

all composes show up there.

This is still broken.

Yep.

Additionally the audit runs are failing with:

No handlers could be found for logger "moksha.hub"
[2019-09-11 18:00:02][pdcupdater.commands    INFO] Performing audit for ModuleStateChangeHandler
[2019-09-11 18:00:02][pdcupdater.commands    INFO] Performing audit for RetireComponentHandler
[2019-09-11 18:00:02][pdcupdater.handlers.retirement    INFO] Looking up all branches from PDC.
[2019-09-11 18:00:02][requests.packages.urllib3.connectionpool    INFO] Starting new HTTP connection (1): pdc-web01.stg.phx2.fedoraproject.org
[2019-09-11 18:00:03][requests.packages.urllib3.connectionpool    INFO] Starting new HTTPS connection (1): src.fedoraproject.org
Traceback (most recent call last):
  File "/usr/bin/pdc-updater-audit", line 9, in <module>
    load_entry_point('pdc-updater==0.9.3', 'console_scripts', 'pdc-updater-audit')()
  File "/usr/lib/python2.7/site-packages/pdcupdater/commands.py", line 72, in audit
    results[name] = handler.audit(pdc)
  File "/usr/lib/python2.7/site-packages/pdcupdater/handlers/retirement.py", line 123, in audit
    namespace=self._pdc_to_namespace(branch['type']),
  File "/usr/lib/python2.7/site-packages/pdcupdater/handlers/retirement.py", line 104, in _pdc_to_namespace
    .format(pdc_type))
ValueError: The PDC type "flatpak" is not supported

@cverna has been working on a pdc replacement, perhaps he has ideas here.

Yep.
Additionally the audit runs are failing with:
No handlers could be found for logger "moksha.hub"
[2019-09-11 18:00:02][pdcupdater.commands INFO] Performing audit for ModuleStateChangeHandler
[2019-09-11 18:00:02][pdcupdater.commands INFO] Performing audit for RetireComponentHandler
[2019-09-11 18:00:02][pdcupdater.handlers.retirement INFO] Looking up all branches from PDC.
[2019-09-11 18:00:02][requests.packages.urllib3.connectionpool INFO] Starting new HTTP connection (1): pdc-web01.stg.phx2.fedoraproject.org
[2019-09-11 18:00:03][requests.packages.urllib3.connectionpool INFO] Starting new HTTPS connection (1): src.fedoraproject.org
Traceback (most recent call last):
File "/usr/bin/pdc-updater-audit", line 9, in <module>
load_entry_point('pdc-updater==0.9.3', 'console_scripts', 'pdc-updater-audit')()
File "/usr/lib/python2.7/site-packages/pdcupdater/commands.py", line 72, in audit
results[name] = handler.audit(pdc)
File "/usr/lib/python2.7/site-packages/pdcupdater/handlers/retirement.py", line 123, in audit
namespace=self._pdc_to_namespace(branch['type']),
File "/usr/lib/python2.7/site-packages/pdcupdater/handlers/retirement.py", line 104, in _pdc_to_namespace
.format(pdc_type))
ValueError: The PDC type "flatpak" is not supported

@cverna has been working on a pdc replacement, perhaps he has ideas here.

I could try to give it a look, but I realistically can't spare time on it this week and next week I am afk.

FWIW, it seems this got fixed somewhere along the line, and recent nightlies are being imported again. But the ones that were missed were never picked up, I don't think.

In updating fedfind the other day I also found out that the last two official stable release composes never got imported to PDC. Fedora-30-20190425.0 and Fedora-31-20191023.0 are missing, those being the F30 and F31 final release composes respectively. (This is a bit of a bummer for fedfind as it means it can't provide the detailed information on these releases that it provides for all other stable releases which were built with Pungi 4).

I'm trying to get toddlers to import the old composes. I'll let you know when it's done.

So it's done importing the past composes, however, it ran into a fairly large number of 503, 502 and even a few 500 errors, so I'm pretty sure it did not import old that it could.

While it was processing, I could see that notifications from nagios about PDC running out of memory, so I wonder if it could be related, after all uploading composes uploads fairly large chunk of data.
We can restart whenever we want (I'll add a blob on the howto repo if someone is curious).

Alright, I've tried importing the old composes again and basically it failed for a few for which it checked if they were there, said no but when it goes and try to upload them, PDC says they are already there... I guess something isn't entirely straight with the logic, but I don't know enough about PDC to be able to address it.

For all the other composes it says they have been uploaded. I am a little doubtful that this is the case, because of the 50* errors I saw yesterday, but here I was, I don't know enough about PDC to be able to address this.

I am therefore inclined to consider this ticket fixed as much as possibly can fix it.

I can confirm those composes are in now... fedfind finds them. ;)

Feel free to re-open if you see any further issues here. ;)

Thanks @pingou!

thanks for the work guys!

This seems to be happening again.

The last F33 nightly to get imported was Fedora-33-20201005.n.0 . The release compose (Fedora-33-20201019.0) was not imported, which means fedfind doesn't have a lot of data on it.

Last Rawhide that got imported was Fedora-Rawhide-20201004.n.1 , so seems like something broke right around October 4-5.

@pingou https://pagure.io/pungi-fedora/pull-request/923 should fix the imports, but we need to import the f33 RC compose, what would be the best way to do it?

Last Rawhide that got imported was Fedora-Rawhide-20201004.n.1 , so seems like something broke right around October 4-5.

Yup the some ARM compose got fixed and that broke the toddler updating pdc... :]

I believe this was fixed a while ago.

Please re-open if you still see the issue