PR#7519: Update atomic.twoweek fedmsg with multi-arch content - releng

releng

#7519 Update atomic.twoweek fedmsg with multi-arch content

Merged 5 years ago by dustymabe. Opened 5 years ago by sinnykumari.

sinnykumari/releng master into master

Update atomic.twoweek fedmsg with multi-arch content

Sinny Kumari • 5 years ago

ff2f2a5

file modified

+115 -172

		`@@ -17,7 +17,7 @@`
		`# Exit codes:`
		`# 0 - Success`
		`# 1 - required arg missing`
		`- # 2 - no successful AutoCloud builds found`
		`+ # 2 - no/partial information for given compose ID found`
		`# 3 - subcommand failed, error message will be logged.`
		`# 4 - execution canceled by user`
		`# 5 - masher lock file found`
		`@@ -40,6 +40,7 @@`
		`import requests`
		`from email.mime.multipart import MIMEMultipart`
		`from email.mime.text import MIMEText`
		`+ import fedfind.release`

		`# Set log level to logging.INFO`
		`logging.basicConfig(level=logging.INFO)`
		`@@ -48,6 +49,20 @@`
		`# Define "constants"`
		`ATOMIC_REPO = "/mnt/koji/atomic/repo/"`
		`ARCHES = ['x86_64', 'aarch64', 'ppc64le']`
		`+`
		`+ # Possible image types available for Atomic Host`
		`+ IMAGE_TYPES = ['qcow2', 'raw-xz', 'vagrant-libvirt',`
		`+ 'vagrant-virtualbox', 'dvd-ostree']`
		`+`
		`+ # Mapping for image types to keep it consistent with twoweek fedmsg sent in past`
		`+ IMAGE_TYPES_MAPPING = {`
		`+ 'qcow2': 'atomic_qcow2',`
		`+ 'raw-xz': 'atomic_raw',`
		`+ 'vagrant-libvirt': 'atomic_vagrant_libvirt',`
		`+ 'vagrant-virtualbox': 'atomic_vagrant_virtualbox',`
		`+ 'dvd-ostree': 'atomic_dvd_ostree'`
		`+ }`
		`+`
		`PREVIOUS_MAJOR_RELEASE_FINAL_COMMITS = {`
		`'aarch64': None,`
		`'ppc64le': None,`
		`@@ -116,8 +131,7 @@`
		`# delta = 2 weeks in seconds`
		`DATAGREPPER_DELTA = 1209600`
		`# category to filter on from datagrepper`
		`- # url: https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.autocloud.compose.complete`
		`- DATAGREPPER_AUTOCLOUD_TOPIC = "org.fedoraproject.prod.autocloud.compose.complete"`
		`+`
		`# url: https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.pungi.compose.ostree`
		`DATAGREPPER_OSTREE_TOPIC = "org.fedoraproject.prod.pungi.compose.ostree"`

		`@@ -128,21 +142,6 @@`
		`ATOMIC_HOST_COMPOSE_PERSIST_LIMIT = 20`


		`- def construct_url(msg):`
		`- """ Construct the final URL from koji URL.`
		`-`
		`- Takes an autocloud fedmsg message and returns the image name and final url.`
		`- """`
		`- iul = msg[u'image_url'].split('/')`
		`-`
		`- # This isn't used in the path for the destination dir, it's in there twice`
		`- iul.remove('compose')`
		`- iul.remove('compose')`
		`-`
		`- image_name = iul[-1]`
		`- image_url = os.path.join(ATOMIC_HOST_STABLE_BASEDIR, '/'.join(iul[4:]))`
		`- return image_name, image_url`
		`-`
		`def get_ostree_compose_info(`
		`ostree_pungi_compose_id,`
		`datagrepper_url=DATAGREPPER_URL,`
		`@@ -169,15 +168,15 @@`

		`# Start with page 1 response from datagrepper, grab the raw messages`
		`# and then continue to populate the list with the rest of the pages of data`
		`- autocloud_data = r.json()[u'raw_messages']`
		`+ ostree_data = r.json()[u'raw_messages']`
		`for rpage in range(2, r.json()[u'pages']+1):`
		`- autocloud_data += requests.get(`
		`+ ostree_data += requests.get(`
		`datagrepper_url,`
		`params=dict(page=rpage, **request_params)`
		`).json()[u'raw_messages']`

		`ostree_composes = [`
		`- compose[u'msg'] for compose in autocloud_data`
		`+ compose[u'msg'] for compose in ostree_data`
		`if ostree_pungi_compose_id in compose[u'msg'][u'compose_id']`
		`and 'atomic-host' in compose[u'msg'][u'ref']`
		`]`
		`@@ -191,145 +190,99 @@`

		`return ostree_compose_info`

		`- def get_latest_successful_autocloud_test_info(`
		`- release,`
		`- pungi_compose_id,`
		`- datagrepper_url=DATAGREPPER_URL,`
		`- delta=DATAGREPPER_DELTA,`
		`- topic=DATAGREPPER_AUTOCLOUD_TOPIC):`
		`- """`
		`- get_latest_successful_autocloud_test_info`
		`-`
		`- Query datagrepper[0] to find the latest successful Atomic Host images via`
		`- the autocloud[1] tests.`
		`+ def get_release_artifacts_info_from_compose(pungi_compose_id):`

		`- return -> dict`
		`- Will return the build information of the latest successful build`
		`-`
		`- [0] - https://apps.fedoraproject.org/datagrepper/`
		`- [1] - https://github.com/kushaldas/autocloud/`
		`"""`
		`-`
		`- # rows_per_page is maximum 100 from Fedora's datagrepper`
		`- request_params = {`
		`- "delta": delta,`
		`- "topic": topic,`
		`- "rows_per_page": 100,`
		`+ :param pungi_compose_id: Pungi Compose ID considered for twoweek release`
		`+ :return: dict - Returns images detail for all supported arches for given`
		`+ compose ID.`
		`+`
		`+ Image detail includes information like image url, image name,`
		`+ image size, Fedora release version and pungi compose ID in which image was`
		`+ built. Image can be of type qcow2, raw, iso and virtual box`
		`+ for which Fedora Atomic Host gets built against architectures. Currently we`
		`+ build for aarch64, ppc64le and x86_64 architectures.`
		`+`
		`+ Example: dict structure is something like:`
		`+ {`
		`+ "aarch64": {`
		`+ "atomic_dvd_ostree": {`
		`+ "name": "Fedora-AtomicHost-ostree-aarch64-28-20180515.1.iso",`
		`+ "image_name": "Fedora-AtomicHost-ostree-aarch64-28-20180515.1.iso",`
		`+ "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/aarch64/iso/Fedora-AtomicHost-ostree-aarch64-28-20180515.1.iso",`
		`+ "release": "28",`
		`+ "compose_id": "Fedora-Atomic-28-20180515.1",`
		`+ "size": 988649472`
		`+ },`
		`+ "atomic_qcow2": {`
		`+ "name": "Fedora-AtomicHost-28-20180515.1",`
		`+ "image_name": "Fedora-AtomicHost-28-20180515.1.aarch64.qcow2",`
		`+ "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/aarch64/images/Fedora-AtomicHost-28-20180515.1.aarch64.qcow2",`
		`+ "release": "28",`
		`+ "compose_id": "Fedora-Atomic-28-20180515.1",`
		`+ "size": 621911040`
		`+ },`
		`+ "atomic_raw": {`
		`+ "name": "Fedora-AtomicHost-28-20180515.1",`
		`+ "image_name": "Fedora-AtomicHost-28-20180515.1.aarch64.raw.xz",`
		`+ "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/aarch64/images/Fedora-AtomicHost-28-20180515.1.aarch64.raw.xz",`
		`+ "release": "28",`
		`+ "compose_id": "Fedora-Atomic-28-20180515.1",`
		`+ "size": 396619232`
		`+ }`
		`+ },`
		`+ "x86_64": {`
		`+ ...`
		`+ },`
		`+ "ppc64le": {`
		`+ ...`
		`+ }`
		`}`
		`- r = requests.get(datagrepper_url, params=request_params)`

		`- # Start with page 1 response from datagrepper, grab the raw messages`
		`- # and then continue to populate the list with the rest of the pages of data`
		`- autocloud_data = r.json()[u'raw_messages']`
		`- for rpage in range(2, r.json()[u'pages']+1):`
		`- autocloud_data += requests.get(`
		`- datagrepper_url,`
		`- params=dict(page=rpage, **request_params)`
		`- ).json()[u'raw_messages']`
		`-`
		`-`
		`- # XXX ignore this way of doing things for now (see below)`
		`- ##### List comprehension that will return a list of compose information from`
		`- ##### AutoCloud (the [u'msg'] payload of autocloud.compose.complete fedmsg)`
		`- ##### such that the following criteria are true:`
		`- #####`
		`- ##### - Is an Atomic Host compose (i.e. 'Atomic' is in the compose id)`
		`- ##### - No compose artifacts failed the tests`
		`- ##### - This is the current Fedora release we want`
		`- #####`
		`- ##### OR:`
		`- ##### - This compose was manually marked good`
		`- ####candidate_composes = [`
		`- #### compose[u'msg'] for compose in autocloud_data`
		`- #### if u'Atomic' in compose[u'msg'][u'id']`
		`- #### and compose[u'msg'][u'results'][u'failed'] == 0`
		`- #### and compose[u'msg'][u'release'] == str(release)`
		`- #### or compose[u'msg'][u'id'] in MARK_ATOMIC_HOST_GOOD_COMPOSES`
		`- ####]`
		`-`
		`- ####filtered_composes = list(candidate_composes)`
		`- ####for compose in candidate_composes:`
		`- #### if compose_manually_marked_bad(compose[u'id']):`
		`- #### filtered_composes.remove(compose)`
		`-`
		`- ##### sc = successful compose`
		`- ####sc = filtered_composes[0]`
		`-`
		`- # XXX For now ignore the autocloud results and just use the`
		`- # requested pungi compose id.`
		`- #`
		`- # List comprehension that will return a list of compose information from`
		`- # AutoCloud (the [u'msg'] payload of autocloud.compose.complete fedmsg)`
		`- # such that the following criteria are true:`
		`- candidate_composes = [`
		`- compose[u'msg'] for compose in autocloud_data`
		`- if pungi_compose_id in compose[u'msg'][u'id']`
		`- ]`
		`- # sc = successful compose`
		`- sc = candidate_composes[0]`
		`-`
		`- autocloud_info = {}`
		`-`
		`- # qcow2 image`
		`- qcow_msg = [`
		`- sc[u'results'][u'artifacts'][img] for img in sc[u'results'][u'artifacts']`
		`- if sc[u'results'][u'artifacts'][img][u'family'] == u'Atomic'`
		`- and sc[u'results'][u'artifacts'][img][u'type'] == u'qcow2'`
		`- ][0]`
		`- image_name, image_url = construct_url(qcow_msg)`
		`- autocloud_info["atomic_qcow2"] = {`
		`- "compose_id": sc[u'id'],`
		`- "name": qcow_msg[u'name'],`
		`- "release": sc[u'release'],`
		`- "image_name": image_name,`
		`- "image_url": image_url,`
		`- }`
		`+ """`

		`- # raw image`
		`- #`
		`- # FIXME - This is a bit of a hack right now, but the raw image is what`
		`- # the qcow2 is made of so only qcow2 is tested and infers the`
		`- # success of both qcow2 and raw.xz`
		`- autocloud_info["atomic_raw"] = {`
		`- "compose_id": sc[u'id'],`
		`- "name": qcow_msg[u'name'],`
		`- "release": sc[u'release'],`
		`- "image_name": image_name.replace('qcow2', 'raw.xz'), # HACK`
		`- "image_url": image_url.replace('qcow2', 'raw.xz'), # HACK`
		`- }`
		`+ # Use fedind to get image metadata information for given Pungi Compose ID`
		`+ # https://pagure.io/fedora-qa/fedfind`
		`+ log.info("Begin fetching image metadata information using fedfind for Compose ID %s", pungi_compose_id)`
		`+ compose_metadata = fedfind.release.get_release(cid=pungi_compose_id).metadata`
		`+ log.info("Finished fetching image metadata information using fedfind for Compose ID %s", pungi_compose_id)`

		`- # vagrant libvirt image`
		`- vlibvirt_msg = [`
		`- sc[u'results'][u'artifacts'][img] for img in sc[u'results'][u'artifacts']`
		`- if sc[u'results'][u'artifacts'][img][u'family'] == u'Atomic'`
		`- and sc[u'results'][u'artifacts'][img][u'type'] == u'vagrant-libvirt'`
		`- ][0]`
		`- image_name, image_url = construct_url(vlibvirt_msg)`
		`- autocloud_info["atomic_vagrant_libvirt"] = {`
		`- "compose_id": sc[u'id'],`
		`- "name": vlibvirt_msg[u'name'],`
		`- "release": sc[u'release'],`
		`- "image_name": image_name,`
		`- "image_url": image_url,`
		`- }`
		`+ if not compose_metadata and not compose_metadata[u'composeinfo'] and not compose_metadata[u'images']:`
		`+ log.error("Image details not found for Compose ID : %s", pungi_compose_id)`
		`+ sys.exit(2)`

		`- # vagrant vbox image`
		`- vvbox_msg = [`
		`- sc[u'results'][u'artifacts'][img] for img in sc[u'results'][u'artifacts']`
		`- if sc[u'results'][u'artifacts'][img][u'family'] == u'Atomic'`
		`- and sc[u'results'][u'artifacts'][img][u'type'] == u'vagrant-virtualbox'`
		`- ][0]`
		`- image_name, image_url = construct_url(vvbox_msg)`
		`- autocloud_info["atomic_vagrant_virtualbox"] = {`
		`- "compose_id": sc[u'id'],`
		`- "name": vvbox_msg[u'name'],`
		`- "release": sc[u'release'],`
		`- "image_name": image_name,`
		`- "image_url": image_url,`
		`- }`
		`+ compose_id = compose_metadata[u'images'][u'payload'][u'compose'][u'id']`
		`+ release = compose_metadata[u'composeinfo'][u'payload'][u'release'][u'version']`
		`+ # Get Atomic Host related messages`
		`+ atomic_host_msg = compose_metadata[u'images'][u'payload'][u'images'][u'AtomicHost']`
		`+ release_artifacts_info = dict()`

		`- return autocloud_info`
		`+ for arch in ARCHES:`
		`+ if arch in atomic_host_msg:`
		`+ release_artifact_info = dict()`
		`+ images_detail = atomic_host_msg[arch]`
		`+ for image_detail in images_detail:`
		`+ if image_detail['type'] in IMAGE_TYPES:`
		`+ image_url = os.path.join(ATOMIC_HOST_STABLE_BASEDIR,`
		`+ compose_id,`
		`+ image_detail[u'path'] )`
		`+ image_name = image_detail[u'path'].split('/')[-1]`
		`+ name = image_name.split('.' + arch)[0]`
		`+ release_artifact_info[IMAGE_TYPES_MAPPING[image_detail[u'type']]] = {`
		`+ "compose_id": compose_id,`
		`+ "name": name,`
		`+ "release": release,`
		`+ "image_name": image_name,`
		`+ "image_url": image_url,`
		`+ "size": image_detail[u'size']`
		`+ }`
		`+ if release_artifact_info:`
		`+ release_artifacts_info[arch] = release_artifact_info`
		`+ else:`
		`+ log.error("Image details not available for arch : %s", arch)`
		`+ sys.exit(2)`

		`+ return release_artifacts_info`

		`def compose_manually_marked_bad(compose_id, bad_composes=MARK_ATOMIC_HOST_BAD_COMPOSES):`
		`"""`
		`@@ -764,28 +717,18 @@`
		`log.info("Release Blocked: Exiting.")`
		`sys.exit(0)`

		`- log.info("Querying datagrepper for latest AutoCloud successful tests")`
		`- # Acquire the latest successful builds from datagrepper`
		`- tested_autocloud_info = get_latest_successful_autocloud_test_info(`
		`- pargs.release, pargs.pungi_compose_id`
		`- )`
		`- log.info("{}\n{}".format("TESTED_AUTOCLOUD_INFO", json.dumps(tested_autocloud_info, indent=2)))`
		`+ log.info("Fetching images information for Compose ID %s", pargs.pungi_compose_id )`
		`+ # Get image artifacts information for given Pungi Compose ID`
		`+ release_artifacts_info = get_release_artifacts_info_from_compose(`
		`+ pargs.pungi_compose_id)`
		`+ log.info("{}\n{}".format("RELEASE_ARTIFACTS_INFO", json.dumps(release_artifacts_info, indent=2)))`

		`- log.info("Query to datagrepper complete")`
		`- # If the dict is empty, there were no successful builds in the last two`
		`- # weeks, error accordingly`
		`- if not tested_autocloud_info:`
		`- log.error("No successful builds found")`
		`+ log.info("Fetching images information from compose ID %s complete", pargs.pungi_compose_id)`
		`+ # If the dict is empty, artifacts information not found for given compose ID`
		`+ if not release_artifacts_info:`
		`+ log.error("Images information not found for Compose ID %s", pargs.pungi_compose_id)`
		`sys.exit(2)`

		`- ## XXX Not needed since we are specifying compose ID`
		`- #log.info("Extracting compose_id from tested autocloud data")`
		`- #compose_id = tested_autocloud_info['atomic_qcow2']['compose_id']`
		`- #`
		`- ## TODO: https://github.com/kushaldas/tunirtests/pull/59 will allow us to`
		`- ## extract this from the autocloud test results.`
		`- #print('Releasing compose %s' % compose_id)`
		`-`
		`# Initialize a empty dict that we will populate with information`
		`# about each commit. We'll use this information to do the release.`
		`# Example: => {`
		`@@ -855,7 +798,7 @@`
		`log.info("Sending fedmsg releng.atomic.twoweek.begin")`
		`fedmsg_publish(`
		`topic="atomic.twoweek.begin",`
		`- msg=dict(**tested_autocloud_info)`
		`+ msg=dict(**release_artifacts_info)`
		`)`

		`log.info("Signing image metadata - compose")`
		`@@ -900,7 +843,7 @@`
		`log.info("Sending fedmsg releng.atomic.twoweek.complete")`
		`fedmsg_publish(`
		`topic="atomic.twoweek.complete",`
		`- msg=dict(**tested_autocloud_info)`
		`+ msg=dict(**release_artifacts_info)`
		`)`

		`log.info("Sending Two Week Atomic Host announcement email")`

sinnykumari commented 5 years ago

No longer depend on autocloud fedmsg
More details in ticket - https://pagure.io/atomic-wg/issue/392

Signed-off-by: Sinny Kumari sinny@redhat.com

dustymabe commented on line 6 of scripts/push-two-week-atomic.py 5 years ago

typo: information

dustymabe commented on line 14 of scripts/push-two-week-atomic.py 5 years ago

we need to verify this library is installed on the machine we run this on

dustymabe commented on line 114 of scripts/push-two-week-atomic.py 5 years ago

For first time readers of the code it would be nice to get an "example" of what this dict will look like and the information it will contain

dustymabe commented on line 114 of scripts/push-two-week-atomic.py 5 years ago

i.e. since this dict is essentially what gets sent out in the fedmsg. Also do we need to update any fedmsg docs with an example of what the new fedmsg will look like?

dustymabe commented 5 years ago

a few comments.. mostly LGTM

sinnykumari commented on line 14 of scripts/push-two-week-atomic.py 5 years ago

Right! I will check into ansible repo. Just to confirm, this twoweek script runs on machine bodhi-backend01.phx2.fedoraproject.org , right?

Edited 5 years ago by sinnykumari

sinnykumari commented on line 114 of scripts/push-two-week-atomic.py 5 years ago

We may need to update fedmsg doc at http://fedora-fedmsg.readthedocs.io/en/latest/topics.html#releng.
Also, updating sop wiki https://docs.pagure.org/releng/sop_two_week_atomic.html will be needed.

dustymabe commented on line 14 of scripts/push-two-week-atomic.py 5 years ago

yeah according to the SOP it's bodhi-backend01.phx2.fedoraproject.org

rebased onto 64b306948245b16b8b2527d81195d229e5b9a4b6

5 years ago

sinnykumari commented 5 years ago

Thanks @dustymabe for the review

I have made required fixes as per review comments which are related to twoweek script.

I will be working separately on updating fedmsg doc. By looking into ansible repo, it looks to me that fedfind is not installed in bodhi-backend01.phx2.fedoraproject.org machine. Will also work on getting fedfind added on host.

dustymabe commented on line 125 of scripts/push-two-week-atomic.py 5 years ago

I actually wouldn't mind just copying that content here. We can trim some of it:

"msg": {
    "aarch64": {
      "atomic_dvd_ostree": {
         "name": "Fedora-AtomicHost-ostree-aarch64-28-20180515.1.iso", 
         "image_name": "Fedora-AtomicHost-ostree-aarch64-28-20180515.1.iso", 
         "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/aarch64/iso/Fedora-AtomicHost-ostree-aarch64-28-20180515.1.iso", 
         "release": "28", 
         "compose_id": "Fedora-Atomic-28-20180515.1", 
         "size": 988649472
      }, 
      "atomic_qcow2": {
         "name": "Fedora-AtomicHost-28-20180515.1", 
         "image_name": "Fedora-AtomicHost-28-20180515.1.aarch64.qcow2", 
         "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/aarch64/images/Fedora-AtomicHost-28-20180515.1.aarch64.qcow2", 
         "release": "28", 
         "compose_id": "Fedora-Atomic-28-20180515.1", 
         "size": 621911040
      }, 
      "atomic_raw": {
         "name": "Fedora-AtomicHost-28-20180515.1", 
         "image_name": "Fedora-AtomicHost-28-20180515.1.aarch64.raw.xz", 
         "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/aarch64/images/Fedora-AtomicHost-28-20180515.1.aarch64.raw.xz", 
         "release": "28", 
         "compose_id": "Fedora-Atomic-28-20180515.1", 
         "size": 396619232
      }
   }, 
   "x86_64": {
   ...
   }, 
   "ppc64le": {
   ...
   }
}

dustymabe commented 5 years ago

can we add a 2nd commit that deletes MARK_ATOMIC_HOST_BAD code? I really don't think we use that any longer?

sinnykumari commented 5 years ago

can we add a 2nd commit that deletes MARK_ATOMIC_HOST_BAD code? I really don't think we use that any longer?

I will do a separate PR for it. I felt that few more clean-ups can be done but I was afraid to touch them in case it breaks something I am not aware of :)

sinnykumari commented on line 125 of scripts/push-two-week-atomic.py 5 years ago

I tried to put something like this in comment with trimming of stuff and then It looked to me a bit of lengthy comment and hence I just added the link. If we are ok with this comment, will add it.

dustymabe commented on line 125 of scripts/push-two-week-atomic.py 5 years ago

yeah I like having it inline - that way it can be updated if things change.

dustymabe commented 5 years ago

I will do a separate PR for it.

rebased onto ff2f2a5

5 years ago

sinnykumari commented 5 years ago

Added dict structure in inline comment

dustymabe commented 5 years ago

LGTM.

mohanboddu commented 5 years ago

LGTM +1

mohanboddu commented 5 years ago

Also as @dustymabe mentioned earlier, fedfind is not installed on bodhi-backend01 from which we run the two week release.

So, we need to work with infra to get it installed.

dustymabe commented 5 years ago

Also as @dustymabe mentioned earlier, fedfind is not installed on bodhi-backend01 from which we run the two week release.
So, we need to work with infra to get it installed.

@sinnykumari has taken that as an action item. She will open a ticket to make it happen. I'm going to go ahead and merge this.