Since around 2017-07-01, PDC seems not to have sent pdc.compose fedmsgs for several composes. Here are the composes for which pdc.compose fedmsgs appear during this time:
pdc.compose
"compose_id": "Fedora-Docker-25-20170703.0", "compose_id": "Fedora-Rawhide-20170702.n.0", "compose_id": "Fedora-Cloud-25-20170702.0", "compose_id": "Fedora-Cloud-24-20170702.0", "compose_id": "Fedora-Docker-25-20170702.0", "compose_id": "Fedora-26-20170701.n.0", "compose_id": "Fedora-Rawhide-20170701.n.0",
but here are the composes Pungi has produced in that time (approximately, this is based on a dumb check of the pungi compose fedmsgs so some of these may have failed):
"compose_id": "Fedora-Rawhide-20170703.n.0", "compose_id": "Fedora-Cloud-25-20170703.0", "compose_id": "Fedora-26-20170703.n.0", "compose_id": "Fedora-Atomic-25-20170703.0", "compose_id": "Fedora-Cloud-24-20170703.0", "compose_id": "Fedora-Docker-25-20170703.0", "compose_id": "Fedora-26-20170702.n.0", "compose_id": "Fedora-Rawhide-20170702.n.0", "compose_id": "Fedora-26-20170702.0", "compose_id": "Fedora-Cloud-25-20170702.0", "compose_id": "Fedora-Atomic-25-20170702.0", "compose_id": "Fedora-Cloud-24-20170702.0", "compose_id": "Fedora-Docker-25-20170702.0", "compose_id": "Fedora-26-20170701.0", "compose_id": "Fedora-26-20170701.n.0", "compose_id": "Fedora-Rawhide-20170701.n.0",
At first I thought the missing composes weren't imported at all, but in fact I do see at least Fedora-26-20170702.0 in the PDC database, haven't checked the others. So it seems like the issue is in fact with the sending of fedmsgs. Notably, the Fedora-26-20170702.0 compose that was missed is the 26 Final RC-1.3 compose; PDC failing to send a fedmsg meant the validation event for it was not automatically created by the bot, which relies on the PDC fedmsgs.
Fedora-26-20170702.0
A Fedora-26-20170703.0 compose was also done recently; it has not appeared in PDC yet, and there has been no fedmsg obviously.
Confirmed. Looks like these two made it into PDC:
But I don't see corresponding fedmsg events in datagrepper for either of them:
It seems like rawhide is working... are only those 2 f26 composes missing?
I don't have a list of exactly what's missing on-hand, but it should be possible to come up with a script relatively easily - just grab all the Pungi compose state change messages for the last while from datagrepper, filter down to changes to FINISHED or FINISHED_INCOMPLETE , then query PDC for all the composes and see which aren't there...
@ralph is this something you could do? (check datagrepper vs PDC for composes it's missing)
I think things have been pretty stable recently, so perhaps there is only that one short gap, I don't know.
Yeah - I can come up with a list. (Sorry for not seeing the question sooner).
Here's a script:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46
#!/usr/bin/env python from __future__ import print_function import collections import datetime import sys # dnf install python-requests import requests url = 'https://apps.fedoraproject.org/datagrepper/raw' topic = 'org.fedoraproject.prod.pungi.compose.status.change' def get_page(page): response = requests.get(url, params=dict( page=page, rows_per_page=100, topic=topic, order='asc', delta=datetime.timedelta(days=120).total_seconds() )) return response.json() def get_composes(): data = get_page(page=1) total_pages = data['pages'] for page in range(1, total_pages + 1): print("page", page, "of", total_pages, "for", topic, file=sys.stderr) data = get_page(page=page) for msg in data['raw_messages']: if 'FINISHED' not in msg['msg']['status']: continue yield msg['msg']['compose_id'], msg['msg_id'] pdc_url = 'https://pdc.fedoraproject.org/rest_api/v1/composes/' for compose_id, msg_id in get_composes(): params = dict(compose_id=compose_id) response = requests.get(pdc_url, params=params) print(" queried %s" % response.request.url, file=sys.stderr) if not bool(response): print(compose_id, "gave a %r error" % response) continue data = response.json() if data['count'] != 1: print("%s %s had %i entries in PDC" % (compose_id, msg_id, data['count'])) continue
It should output message ids for the missing composes. We can then try to reimport them by running pdc-updater-retry on pdc-backend01 with the given msg_ids.
pdc-updater-retry
pdc-backend01
OK, here's the set of composes that had a pungi message on the bus, but no compose in PDC. The msg_id can be used to retry them with pdc-updater-retry on pdc-backend01.
Fedora-Cloud-24-20170711.0 2017-895ea72b-f5c1-4ee0-bd96-1ff0c3480368 had 0 entries in PDC Fedora-Rawhide-20170711.n.0 2017-3ff3106c-732d-4661-8b18-a7d3e8f8414b had 0 entries in PDC Fedora-Atomic-26-20170713.0 2017-917c5455-217e-403a-83cf-34370045d9c0 had 0 entries in PDC Fedora-Atomic-26-20170717.0 2017-6700ec76-47da-42bf-be0b-1af208627140 had 0 entries in PDC Fedora-Docker-25-20170718.0 2017-519ff1e3-fda6-475e-a931-d902e11bbd3f had 0 entries in PDC Fedora-Cloud-24-20170719.0 2017-44b6e7e5-ef45-4e08-a23e-9065a12db855 had 0 entries in PDC Fedora-Atomic-26-20170720.0 2017-31ce1f2a-e06c-4ceb-93ce-e7c23b1bb780 had 0 entries in PDC Fedora-Rawhide-20170722.n.0 2017-d3b7c9c7-0220-4fdc-a671-14cd63754b72 had 0 entries in PDC Fedora-Cloud-24-20170725.0 2017-316553c9-c703-4403-af34-ea2bc30002c8 had 0 entries in PDC Fedora-Cloud-25-20170726.0 2017-6e79686c-be9d-4ae4-a70f-40cb61cb31a7 had 0 entries in PDC Fedora-Docker-25-20170727.0 2017-b4c04625-a112-44da-8c58-6974d6bf4f63 had 0 entries in PDC Fedora-Cloud-24-20170728.0 2017-ef0ac2b5-9bc8-4bec-968b-140ea33a7fbd had 0 entries in PDC Fedora-Cloud-25-20170730.0 2017-edb10d9a-dbae-474b-9176-a39699d3c616 had 0 entries in PDC Fedora-Cloud-25-20170731.0 2017-8cd8e23d-50d0-4e74-87d0-16ac4a9e8761 had 0 entries in PDC Fedora-Atomic-25-20170803.0 2017-0c1a2159-7c2b-4684-8e9a-81bedf91b16f had 0 entries in PDC Fedora-Docker-25-20170804.0 2017-e5032293-b23a-4347-afc7-25051720f506 had 0 entries in PDC Fedora-Docker-25-20170805.0 2017-f151c179-02a7-4a6b-9508-e850486bb804 had 0 entries in PDC Fedora-Modular-Rawhide-20170816.n.0 2017-5e1272e1-c95d-49f3-811e-bb7bbc65126f had 0 entries in PDC Fedora-Atomic-26-20170818.0 2017-01854dc5-e6f7-497d-a717-9213a9f43a46 had 0 entries in PDC Fedora-Atomic-25-20170823.0 2017-1ab84fea-0a90-4370-93d2-0123289bfd1d had 0 entries in PDC Fedora-Modular-Rawhide-20170823.n.0 2017-385bb9bf-497a-464c-8448-85e5f6ef007c had 0 entries in PDC Fedora-Rawhide-20170824.n.0 2017-378cdc2e-0d66-442c-8ebb-312f5cb409ad had 0 entries in PDC Fedora-Rawhide-20170825.n.0 2017-185f8234-a185-48dc-96c3-38977581db65 had 0 entries in PDC Fedora-Rawhide-20170827.n.0 2017-bda75f01-a6a9-4b5b-b1f3-9bfb7561873d had 0 entries in PDC Fedora-Modular-Rawhide-20170828.n.0 2017-7f847d56-7b79-4f6e-b78a-49903e87532d had 0 entries in PDC Fedora-Docker-25-20170902.0 2017-4cfa342d-e97e-4536-9e44-9bf5c97752ed had 0 entries in PDC Fedora-Docker-26-20170902.0 2017-3b91ab04-da4e-45b2-b167-39f6b04d2115 had 0 entries in PDC Fedora-Atomic-26-20170903.0 2017-bfbf3c8a-4eb1-407e-9f09-da0aad25e2e1 had 0 entries in PDC Fedora-Atomic-25-20170905.0 2017-c36f77f0-6a46-49d7-b1e9-2084583cf4b2 had 0 entries in PDC Fedora-Atomic-26-20170905.0 2017-d6c4fcf8-5b77-451e-ba4e-e5f8803b2eb6 had 0 entries in PDC Fedora-Docker-25-20170907.0 2017-300e42b7-49c1-4445-9cd2-fc44f7678add had 0 entries in PDC Fedora-27-20170909.n.0 2017-cde29cc5-6967-4b2b-9001-72aa035d4ceb had 0 entries in PDC Fedora-Rawhide-20170911.n.0 2017-36352402-a62e-4d29-9e9e-99a9549f9d62 had 0 entries in PDC Fedora-27-20170911.n.0 2017-5e84be9b-64d5-4f49-875d-ff7d45d8d4e8 had 0 entries in PDC Fedora-Modular-27-20170913.n.0 2017-66478205-e5c5-4cec-a3d3-4bcdb4d61101 had 0 entries in PDC Fedora-Atomic-25-20170916.0 2017-aff38874-b128-4dde-a14b-8066cf1dd73d had 0 entries in PDC Fedora-Cloud-25-20170917.0 2017-a98f2fc7-4551-4d25-9e51-7370d3e5227a had 0 entries in PDC Fedora-Docker-25-20170918.0 2017-1646b5a7-b0fe-41b6-9eba-b02d37eae904 had 0 entries in PDC Fedora-Cloud-25-20170919.0 2017-8d5672b7-c5e6-4182-b66e-9336d7a81319 had 0 entries in PDC Fedora-Cloud-25-20170921.0 2017-c7dc0ff7-ccbf-4d06-bc5f-9b5fd2c1d523 had 0 entries in PDC Fedora-Docker-25-20170922.0 2017-169359a2-b101-41c6-ad05-a678ff70e388 had 0 entries in PDC Fedora-Atomic-25-20170923.0 2017-a0f7a9a9-f3dd-4521-b2aa-626aaa3813e3 had 0 entries in PDC Fedora-Atomic-25-20170924.0 2017-583eb1e0-dfbd-449c-9ce0-9517985eb96d had 0 entries in PDC Fedora-Cloud-25-20170924.0 2017-e21c4372-5d33-4975-a085-9e74f06f6284 had 0 entries in PDC Fedora-Docker-25-20170925.0 2017-bed8ea3d-deb6-409c-93da-1bdbb0457e83 had 0 entries in PDC Fedora-Docker-26-20170926.0 2017-e48d792f-7511-4b2d-8512-048380adc7c1 had 0 entries in PDC Fedora-Docker-25-20170927.0 2017-f444c420-5115-460a-b7f5-00b663265dbc had 0 entries in PDC Fedora-Atomic-25-20170927.0 2017-7ee3154f-df16-44fb-8538-57e88f4d6153 had 0 entries in PDC Fedora-Modular-Bikeshed-20170927.n.2 2017-d087cae2-ebf3-463d-ba87-7f07912e15e3 had 0 entries in PDC Fedora-Modular-Bikeshed-20170927.n.5 2017-c5189516-efa6-47db-b928-000a56a323e1 had 0 entries in PDC Fedora-Modular-Rawhide-20170929.n.0 2017-0ed65c63-d7b0-477e-bef9-d16d62b564a9 had 0 entries in PDC Fedora-Modular-27-20170929.n.0 2017-d09a285e-1a2f-4935-9c68-bccffc787595 had 0 entries in PDC Fedora-Modular-Rawhide-20170929.n.1 2017-90c69652-d457-45eb-bab6-a8e1f6d9a34a had 0 entries in PDC Fedora-Modular-Rawhide-20170930.n.0 2017-8e30eef6-daf5-48f6-9fc5-12ff5072f69e had 0 entries in PDC Fedora-Modular-Rawhide-20171001.n.0 2017-18eee247-6b59-4f07-9e4b-131b6f59a4ce had 0 entries in PDC Fedora-Modular-Rawhide-20171002.n.0 2017-44bbbade-8714-44c8-9e6a-be8dc7534cd7 had 0 entries in PDC Fedora-Docker-25-20171003.0 2017-3c7522ce-2796-4915-8403-f88dc2625e5d had 0 entries in PDC Fedora-Modular-27-20171003.n.0 2017-f2d81e55-078f-4d43-a817-2e19fdfc33fd had 0 entries in PDC Fedora-Modular-Rawhide-20171003.n.0 2017-d8371fdd-4fb7-4e3c-8ad9-6bbbbd16d254 had 0 entries in PDC Fedora-Modular-27-20171003.n.1 2017-29ed8bc8-46ec-423c-ac36-ef167ff21840 had 0 entries in PDC Fedora-Modular-27-20171004.n.0 2017-2ee19f02-1450-4a2a-87e8-783137fc51da had 0 entries in PDC Fedora-Modular-Bikeshed-20171004.n.0 2017-bdcc254e-0ff6-4490-bb66-cfa822695397 had 0 entries in PDC Fedora-Atomic-25-20171005.0 2017-29c63d3d-3e50-467c-b5a5-1a67a985ef56 had 0 entries in PDC Fedora-Modular-Bikeshed-20171005.n.0 2017-6e7a2e61-dc97-4c8f-a3b7-9689246fa46d had 0 entries in PDC Fedora-Atomic-26-20171006.0 2017-6fe50f8e-dfe9-4341-81ec-8ee4c61f9fde had 0 entries in PDC Fedora-Modular-Bikeshed-20171007.n.1 2017-1f54e5b2-174b-4941-a3e5-ab5c5f0c1b07 had 0 entries in PDC Fedora-Modular-Bikeshed-20171008.n.0 2017-bddc63cf-395e-4328-9033-b03af103bd9e had 0 entries in PDC Fedora-Rawhide-20171009.n.0 2017-f71929ce-86a1-4cc5-b9c8-e3bf24c95409 had 0 entries in PDC Fedora-27-20171009.n.0 2017-dbc86f16-7e45-4f8b-862f-ff8c1565d74c had 0 entries in PDC Fedora-Modular-27-20171009.n.0 2017-72762ff0-c4d7-4191-be97-767a148b561c had 0 entries in PDC Fedora-27-20171011.n.0 2017-83562473-7355-428e-9324-df1f7ae9e4c4 had 0 entries in PDC Fedora-Modular-f27-updates-Fedora-Modular-27-20171013.3 2017-cb0d784d-e5c7-4ed5-b358-d999faafabad had 0 entries in PDC Fedora-Modular-f27-updates-updates-Fedora-Modular-27-20171013.0 2017-af8cc7f4-03bd-4bfc-afad-0fb83a494f2b had 0 entries in PDC Fedora-Modular-f27-updates-updates-20171013.0 2017-1e4795e6-8b0b-4c0f-8211-3a939bb8df33 had 0 entries in PDC Fedora-Modular-Bikeshed-20171015.n.0 2017-875ffe0b-805e-4505-869b-57dd7a93a39e had 0 entries in PDC Fedora-Atomic-26-20171016.0 2017-241ad636-9002-41a0-8a27-d55a9852db75 had 0 entries in PDC Fedora-Atomic-25-20171018.0 2017-c211c420-8289-4316-aa67-23c26a202840 had 0 entries in PDC Fedora-27-20171022.n.0 2017-3c79f048-b4ac-46b9-9913-d4f303145385 had 0 entries in PDC Fedora-Docker-26-20171024.0 2017-876c8a9c-f33f-40c6-a380-bf297e3a4743 had 0 entries in PDC Fedora-25-updates-testing-20171026.0 2017-330528d8-1e2c-4166-884d-168bc1d10571 had 0 entries in PDC Fedora-26-updates-testing-20171026.0 2017-f63f2732-c067-4712-833e-570a02cded18 had 0 entries in PDC Fedora-Epel-6-updates-20171026.0 2017-f5e240ef-c5e5-404d-b724-c45338df5f1a had 0 entries in PDC Fedora-Epel-6-updates-testing-20171026.0 2017-76281f75-595c-4bf5-8abe-e81128fb15a4 had 0 entries in PDC Fedora-25-updates-testing-20171026.1 2017-43ed8c91-b0fd-4c10-8651-ae57d94d68c0 had 0 entries in PDC Fedora-26-updates-testing-20171026.1 2017-9076b7ff-c66a-42e7-9294-8947b72535e9 had 0 entries in PDC Fedora-Epel-7-updates-20171026.0 2017-d474ea7f-ec3e-4789-a312-f75874d92c5f had 0 entries in PDC Fedora-Epel-7-updates-testing-20171026.0 2017-b224c3ed-2fee-48ad-b6ca-82833b8a7655 had 0 entries in PDC Fedora-Atomic-26-20171027.0 2017-0b229323-adae-4399-b436-498ceb49cc7f had 0 entries in PDC Fedora-26-updates-testing-20171027.0 2017-00cfccf0-fcdb-406c-b682-3a2d65eee55a had 0 entries in PDC Fedora-Epel-7-updates-testing-20171027.0 2017-39fd142b-a6c2-491f-8044-e7e0ea22e2ea had 0 entries in PDC Fedora-25-updates-testing-20171027.0 2017-f9a10a27-18f2-48d8-a0c7-f1c76904cebf had 0 entries in PDC Fedora-26-updates-testing-20171027.1 2017-686f7ea5-347a-4bbf-ad0e-2c0fc24d0481 had 0 entries in PDC Fedora-Epel-6-updates-testing-20171027.0 2017-853b2fb6-2458-4e1a-97e5-4529ed122bc8 had 0 entries in PDC Fedora-27-updates-testing-20171027.0 2017-f4b4d642-120a-475c-9b9e-607473987424 had 0 entries in PDC Fedora-Cloud-26-20171028.0 2017-e4f7eb62-5d18-4e6a-b5b0-7f4cf9139af5 had 0 entries in PDC Fedora-25-updates-testing-20171028.0 2017-81fea7e6-5190-4c23-8370-02fe319d52a8 had 0 entries in PDC Fedora-26-updates-testing-20171028.0 2017-8d2f08e9-61e3-4b62-90dd-87a27cf590d2 had 0 entries in PDC Fedora-Docker-25-20171029.0 2017-65bc825c-c72c-4530-952b-b1ee2b8f3764 had 0 entries in PDC Fedora-Epel-7-updates-20171029.0 2017-1aa851fd-ad49-40bb-89e5-f11097bf8afd had 0 entries in PDC Fedora-Epel-6-updates-20171029.0 2017-bd4e701f-b8ed-4be4-9d21-36588cc0cdf6 had 0 entries in PDC Fedora-Epel-7-updates-20171029.1 2017-66b90568-49ce-45b9-86a5-6f14677755e8 had 0 entries in PDC Fedora-Epel-6-updates-testing-20171029.0 2017-082fd734-24de-4c0f-8643-bbf079b043af had 0 entries in PDC Fedora-25-updates-testing-20171029.0 2017-3d327d11-9656-4eaf-bce1-9c525c59bf98 had 0 entries in PDC Fedora-Epel-7-updates-testing-20171029.0 2017-135e2796-0d32-414d-86ee-78092570fc7c had 0 entries in PDC Fedora-26-updates-testing-20171029.0 2017-981ca1b1-8b6d-4698-929f-04cb9b92bf59 had 0 entries in PDC Fedora-27-updates-testing-20171030.0 2017-b15b167c-dae0-4bca-8400-917de5b85c57 had 0 entries in PDC Fedora-Epel-6-updates-testing-20171030.0 2017-f9cff244-0de6-4bf4-ad6e-862baa1f9731 had 0 entries in PDC Fedora-25-updates-testing-20171030.0 2017-87c7fa67-5995-411c-887b-d74a52b586a0 had 0 entries in PDC Fedora-Epel-7-updates-testing-20171030.0 2017-d66ad537-9ec3-4906-867b-1e9c809bf6c1 had 0 entries in PDC Fedora-26-updates-testing-20171030.0 2017-23d304c1-58bb-4ee0-bab0-6840ec3d1ae4 had 0 entries in PDC Fedora-Epel-6-updates-20171030.0 2017-6535bdef-3465-4290-9231-9d26225433d6 had 0 entries in PDC Fedora-Epel-7-updates-20171030.0 2017-9cbd1610-7f06-4d4d-b79b-9ea584bbd5c8 had 0 entries in PDC Fedora-Modular-27-20171031.n.1 2017-c1493d88-0ff8-4200-9222-33d1d9e89fe9 had 0 entries in PDC Fedora-Modular-27-updates-testing-20171031.4 2017-4b15bfc2-9704-4348-9cc4-32dbc23162ef had 0 entries in PDC Fedora-Modular-27-updates-testing-20171031.5 2017-479253fd-d000-4f42-b345-7a49ecdfc152 had 0 entries in PDC Fedora-Modular-27-updates-testing-20171031.7 2017-b059d0ea-54e7-4244-82e8-184ab8c7339c had 0 entries in PDC Fedora-Modular-27-updates-testing-20171031.9 2017-8a4505c4-22ee-448e-a395-914b5c29721b had 0 entries in PDC Fedora-Modular-27-20171031.n.2 2017-06cba8eb-c242-40d1-b5cc-63466f741c1e had 0 entries in PDC Fedora-27-updates-testing-20171031.0 2017-16cecd37-50ca-43a8-8c51-18fb981f9089 had 0 entries in PDC
FYI, I'm trying an import of all of these now on pdc-backend01.
Some of the composes, like this one ( http://kojipkgs.fedoraproject.org/compose/Fedora-Cloud-24-20170711.0/compose/metadata/composeinfo.json ) no longer exist, and so cannot be imported.
Blocked by #6482.
It happened again. Compose Fedora-27-20171105.0 (a release candidate) was created and imported to PDC, but no fedmsg for it was sent, though fedmsgs were sent for composes that completed shortly before and after it.
Fedora-27-20171105.0
edit: actually that's slightly different, isn't it? that time the compose was imported, but no fedmsg was sent. Not sure if related or different.
OK, the missing fedmsg should be fixed by #6482, going forwards.
I'm going to regenerate the list of missing composes here and try reimporting them now that that is fixed, and we're out of freeze.
Metadata Update from @ralph: - Issue assigned to ralph
OK - there were two issues here. The fedmsg bit (hopefully fixed in #6482) and the missing composes, which should be fixed here. I ran through the old ones and imported ~50 of them. A bunch of them 404'd those, since they've been reaped from kojipkgs... I guess they're lost forever.
Metadata Update from @ralph: - Issue status updated to: Closed (was: Open)
"A bunch of them 404'd those, since they've been reaped from kojipkgs... I guess they're lost forever."
Yep, nightlies are generally cleaned up after 14 days. If we missed the data it's just not available anywhere any more, I don't think :/
Login to comment on this ticket.