NOTE
If your issue is for security or deals with sensitive info please mark it as private using the checkbox below.
Toddlers recently starting to throw error:
2024-01-19 17:59:04,657 - [ERROR toddlers.plugins.pdc_import_compose] {"reason":"Image: Field 'format' has invalid value: ociarchive","detail":"Error parsing productmd metadata."}
It doesn't recognize the new format of compose.
Here is the whole log:
2024-01-22 09:24:48,855 - [INFO fedora_messaging.twisted.consumer] Consuming message from topic org.fedoraproject.prod.pungi.compose.status.change (message id 0a0dfc82-077d-44ce-9133-bca8855567fc) 2024-01-22 09:24:48,855 - [INFO toddlers.runner] Toddlers processing: 0a0dfc82-077d-44ce-9133-bca8855567fc -- org.fedoraproject.prod.pungi.compose.status.change 2024-01-22 09:24:48,856 - [INFO toddlers.runner] Toddler 'pdc_import_compose' accepted to process message id: 0a0dfc82-077d-44ce-9133-bca8855567fc 2024-01-22 09:24:48,914 - [INFO toddlers.plugins.pdc_import_compose] Importing compose 'Fedora-Rawhide-20240121.n.0' /usr/lib/python3.11/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'pdc.fedoraproject.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings warnings.warn( /usr/lib/python3.11/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'pdc.fedoraproject.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings warnings.warn( /usr/lib/python3.11/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'pdc.fedoraproject.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings warnings.warn( 2024-01-22 09:24:52,141 - [ERROR toddlers.plugins.pdc_import_compose] Failed to import compose-images (BeenBagException): https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20240121.n.0/compose/metadata/rpms.json Traceback (most recent call last): File "/code/toddlers/plugins/pdc_import_compose.py", line 184, in _import_compose pdc["compose-images"]._( File "/usr/lib/python3.11/site-packages/pdc_client/__init__.py", line 347, in __call__ return self.client(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/beanbag/namespace.py", line 135, in fn return basefn(getattr(self, ".base"), getattr(self, ".path"), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/beanbag/url_v1.py", line 102, in call return self.make_request(path, verb, kwargs, body) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/beanbag/url_v1.py", line 154, in make_request raise BeanBagException(r, beanbag.bbexcept.BeanBagException: Bad response code: 400 2024-01-22 09:24:52,142 - [ERROR toddlers.plugins.pdc_import_compose] Bad response code: 400 2024-01-22 09:24:52,142 - [ERROR toddlers.plugins.pdc_import_compose] https://pdc.fedoraproject.org/rest_api/v1/compose-images/ 2024-01-22 09:24:52,142 - [ERROR toddlers.plugins.pdc_import_compose] {"reason":"Image: Field 'format' has invalid value: ociarchive","detail":"Error parsing productmd metadata."} 2024-01-22 09:30:34,839 - [ERROR toddlers.runner] Toddler 'pdc_import_compose' failed to process message id: 0a0dfc82-077d-44ce-9133-bca8855567fc -- putting it back in the queue Traceback (most recent call last): File "/code/toddlers/runner.py", line 96, in __call__ toddler.process(self.toddler_config[toddler.name], message) File "/code/toddlers/plugins/pdc_import_compose.py", line 64, in process self._process_single_compose(config, pdc, message) File "/code/toddlers/plugins/pdc_import_compose.py", line 118, in _process_single_compose self._import_compose(pdc, compose_id, compose_url) File "/code/toddlers/plugins/pdc_import_compose.py", line 184, in _import_compose pdc["compose-images"]._( File "/usr/lib/python3.11/site-packages/pdc_client/__init__.py", line 347, in __call__ return self.client(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/beanbag/namespace.py", line 135, in fn return basefn(getattr(self, ".base"), getattr(self, ".path"), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/beanbag/url_v1.py", line 102, in call return self.make_request(path, verb, kwargs, body) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/beanbag/url_v1.py", line 154, in make_request raise BeanBagException(r, beanbag.bbexcept.BeanBagException: Bad response code: 400 2024-01-22 09:30:34,839 - [WARNING fedora_messaging.twisted.consumer] Returning message id 0a0dfc82-077d-44ce-9133-bca8855567fc to the queue
And here is the message that caused the error https://apps.fedoraproject.org/datagrepper/v2/id?id=0a0dfc82-077d-44ce-9133-bca8855567fc&is_raw=true&size=extra-large
Metadata Update from @zlopez: - Issue assigned to zlopez
Looking at this more closely and the main reason is that the compose images.json metadata file contains ociarchive format for various image architectures. Here is an example of file.
images.json
ociarchive
I tried to look at how to add a new format to PDC and it seems that this is done through database migration. Not sure if we want to do something like this when we are currently working on removing it from our infrastructure.
The second option is to just ignore ociarchive format in toddlers and just remove them from images.json before sending it to PDC.
Because the PDC is going away soon, I'm for the second option, which will at least stop the toddlers from erroring out.
PR for removing ociarchive format is now available in toddlers.
The PR is now merged and the change was deployed on staging. We will test it there first before deploying to production.
The only problem I could see that causing is that @adamwill s scripts my query pdc for the image? Dunno if they do tho.
fedfind does query PDC for image info in some less important codepaths, but it doesn't care about the ociarchive stuff, so for simplicity's sake +1 to throw it away.
I deployed the change to production today and saw one of the messages being processed without the error. I also changed the number of pods from 7 to 4.
I'm closing the issue as it is now fixed.
Metadata Update from @zlopez: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Just FYI, here is one of the imported composes.
Log in to comment on this ticket.