I have started working on adding support for multi-arch in releng Atomic Two Week release script. Before working on a formal PR, I want to bring out how changes is going to be look like and it's impact. This will help to avoid doing re-work.

Current design

Current Two Week release script gets images detail from autocloud.compose.complete fedmsg topic. Right now, autocloud provides information for x86_64 only. For example, autocloud.compose.complete fedmsg data from Fedora-Atomic-28-20180515.1 is here

New approach

In new approach, I will be using directly fedfind on Atomic Compose to get images information for all arches.

Example

compose - Fedora-Atomic-28-20180515.1
- Format of current Two Week msg taken from here

"msg": {
    "atomic_qcow2": {
      "release": "28", 
      "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/x86_64/images/Fedora-AtomicHost-28-20180515.1.x86_64.qcow2", 
      "image_name": "Fedora-AtomicHost-28-20180515.1.x86_64.qcow2", 
      "compose_id": "Fedora-Atomic-28-20180515.1", 
      "name": "Fedora-AtomicHost-28-20180515.1."
    }, 
    "atomic_vagrant_libvirt": {
      "release": "28", 
      "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/x86_64/images/Fedora-AtomicHost-Vagrant-28-20180515.1.x86_64.vagrant-libvirt.box", 
      "image_name": "Fedora-AtomicHost-Vagrant-28-20180515.1.x86_64.vagrant-libvirt.box", 
      "compose_id": "Fedora-Atomic-28-20180515.1", 
      "name": "Fedora-AtomicHost-Vagrant-28-20180515.1."
    }, 
    "atomic_raw": {
      "release": "28", 
      "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/x86_64/images/Fedora-AtomicHost-28-20180515.1.x86_64.raw.xz", 
      "image_name": "Fedora-AtomicHost-28-20180515.1.x86_64.raw.xz", 
      "compose_id": "Fedora-Atomic-28-20180515.1", 
      "name": "Fedora-AtomicHost-28-20180515.1."
    }, 
    "atomic_vagrant_virtualbox": {
      "release": "28", 
      "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/x86_64/images/Fedora-AtomicHost-Vagrant-28-20180515.1.x86_64.vagrant-virtualbox.box", 
      "image_name": "Fedora-AtomicHost-Vagrant-28-20180515.1.x86_64.vagrant-virtualbox.box", 
      "compose_id": "Fedora-Atomic-28-20180515.1", 
      "name": "Fedora-AtomicHost-Vagrant-28-20180515.1."
    }
  }

• With new approach, msg would look something like below:

"msg": {
    "aarch64": {
      "atomic_dvd_ostree": {
         "name": "Fedora-AtomicHost-ostree-aarch64-28-20180515.1.iso", 
         "image_name": "Fedora-AtomicHost-ostree-aarch64-28-20180515.1.iso", 
         "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/aarch64/iso/Fedora-AtomicHost-ostree-aarch64-28-20180515.1.iso", 
         "release": "28", 
         "compose_id": "Fedora-Atomic-28-20180515.1", 
         "size": 988649472
      }, 
      "atomic_qcow2": {
         "name": "Fedora-AtomicHost-28-20180515.1", 
         "image_name": "Fedora-AtomicHost-28-20180515.1.aarch64.qcow2", 
         "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/aarch64/images/Fedora-AtomicHost-28-20180515.1.aarch64.qcow2", 
         "release": "28", 
         "compose_id": "Fedora-Atomic-28-20180515.1", 
         "size": 621911040
      }, 
      "atomic_raw": {
         "name": "Fedora-AtomicHost-28-20180515.1", 
         "image_name": "Fedora-AtomicHost-28-20180515.1.aarch64.raw.xz", 
         "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/aarch64/images/Fedora-AtomicHost-28-20180515.1.aarch64.raw.xz", 
         "release": "28", 
         "compose_id": "Fedora-Atomic-28-20180515.1", 
         "size": 396619232
      }
   }, 
   "x86_64": {
      "atomic_dvd_ostree": {
         "name": "Fedora-AtomicHost-ostree-x86_64-28-20180515.1.iso", 
         "image_name": "Fedora-AtomicHost-ostree-x86_64-28-20180515.1.iso", 
         "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/x86_64/iso/Fedora-AtomicHost-ostree-x86_64-28-20180515.1.iso", 
         "release": "28", 
         "compose_id": "Fedora-Atomic-28-20180515.1", 
         "size": 1034944512
      }, 
      "atomic_qcow2": {
         "name": "Fedora-AtomicHost-28-20180515.1", 
         "image_name": "Fedora-AtomicHost-28-20180515.1.x86_64.qcow2", 
         "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/x86_64/images/Fedora-AtomicHost-28-20180515.1.x86_64.qcow2", 
         "release": "28", 
         "compose_id": "Fedora-Atomic-28-20180515.1", 
         "size": 635537920
      }, 
      "atomic_vagrant_libvirt": {
         "name": "Fedora-AtomicHost-Vagrant-28-20180515.1", 
         "image_name": "Fedora-AtomicHost-Vagrant-28-20180515.1.x86_64.vagrant-libvirt.box", 
         "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/x86_64/images/Fedora-AtomicHost-Vagrant-28-20180515.1.x86_64.vagrant-libvirt.box", 
         "release": "28", 
         "compose_id": "Fedora-Atomic-28-20180515.1", 
         "size": 603786982
      }, 
      "atomic_raw": {
         "name": "Fedora-AtomicHost-28-20180515.1", 
         "image_name": "Fedora-AtomicHost-28-20180515.1.x86_64.raw.xz", 
         "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/x86_64/images/Fedora-AtomicHost-28-20180515.1.x86_64.raw.xz", 
         "release": "28", 
         "compose_id": "Fedora-Atomic-28-20180515.1", 
         "size": 457994952
      }, 
      "atomic_vagrant_virtualbox": {
         "name": "Fedora-AtomicHost-Vagrant-28-20180515.1", 
         "image_name": "Fedora-AtomicHost-Vagrant-28-20180515.1.x86_64.vagrant-virtualbox.box", 
         "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/x86_64/images/Fedora-AtomicHost-Vagrant-28-20180515.1.x86_64.vagrant-virtualbox.box", 
         "release": "28", 
         "compose_id": "Fedora-Atomic-28-20180515.1", 
         "size": 617984000
      }
   }, 
   "ppc64le": {
      "atomic_dvd_ostree": {
         "name": "Fedora-AtomicHost-ostree-ppc64le-28-20180515.1.iso", 
         "image_name": "Fedora-AtomicHost-ostree-ppc64le-28-20180515.1.iso", 
         "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/ppc64le/iso/Fedora-AtomicHost-ostree-ppc64le-28-20180515.1.iso", 
         "release": "28", 
         "compose_id": "Fedora-Atomic-28-20180515.1", 
         "size": 1036103680
      }, 
      "atomic_qcow2": {
         "name": "Fedora-AtomicHost-28-20180515.1", 
         "image_name": "Fedora-AtomicHost-28-20180515.1.ppc64le.qcow2", 
         "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/ppc64le/images/Fedora-AtomicHost-28-20180515.1.ppc64le.qcow2", 
         "release": "28", 
         "compose_id": "Fedora-Atomic-28-20180515.1", 
         "size": 636539904
      }, 
      "atomic_raw": {
         "name": "Fedora-AtomicHost-28-20180515.1", 
         "image_name": "Fedora-AtomicHost-28-20180515.1.ppc64le.raw.xz", 
         "image_url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/AtomicHost/ppc64le/images/Fedora-AtomicHost-28-20180515.1.ppc64le.raw.xz", 
         "release": "28", 
         "compose_id": "Fedora-Atomic-28-20180515.1", 
         "size": 411513572
      }
   }
}

Notable changes

• Information available for multi-arches
• Information available for ISOs
• Images size information available (fedora-website can directly use size information from here instead of using requests python module on image urls)

Impact

• We will no longer be using autocloud in push two week atomic
• fedora-website atomic page will have to adapt to this new schema

Additional Note: I would like to remove "name" key information for images unless it is currently used somewhere. One of the reason of removing "name" key is that I am not sure what should its value look like for iso


Thoughts?

@mohanboddu @dustymabe

hey sinny - removing the dep on autocloud would be a great move! I think it all looks good to me!

What does fedfind use to discover information? Are we no longer using information from fedmsgs?

hey sinny - removing the dep on autocloud would be a great move! I think it all looks good to me!
What does fedfind use to discover information? Are we no longer using information from fedmsgs?

As far as I understand, fedfind knows where Fedora composes are available. It fetches information from metadata directory and gives us list of images available in dictionary format.
For example: For compose Fedora-Atomic-28-20180515.1, it fetches detail from https://kojipkgs.fedoraproject.org/compose/twoweek/Fedora-Atomic-28-20180515.1/compose/metadata/images.json

Yes, we will no longer be using autocloud femdsg information for getting images detail. Two week release script also make use of pungi.compose.ostree fedmsg, which is not going to change.

:thumbsup:

@sinnykumari

I would like to remove "name" key information for images unless it is currently used somewhere. One of the reason of removing "name" key is that I am not sure what should its value look like for iso

What do you mean by that? Because there is a name to the ostree iso's we generate, for ex:

"atomic_dvd_ostree": {
        "name": "Fedora-AtomicHost-ostree-ppc64le-28-20180515.1.iso",

What if the fedmsg just linked to the compose? Then each caller would parse the existing compose metadata JSON, and we wouldn't need to define a new schema.

@sinnykumari

I would like to remove "name" key information for images unless it is currently used somewhere. One of the reason of removing "name" key is that I am not sure what should its value look like for iso

What do you mean by that? Because there is a name to the ostree iso's we generate, for ex:
"atomic_dvd_ostree": {
"name": "Fedora-AtomicHost-ostree-ppc64le-28-20180515.1.iso",

If we look at "msg" content produced by two week release script while using autocloud to fetch data, it contains both "name" and "image_name" for each image type. For example:

atomic_vagrant_virtualbox": {
      "image_name": "Fedora-AtomicHost-Vagrant-28-20180515.1.x86_64.vagrant-virtualbox.box", 
      "compose_id": "Fedora-Atomic-28-20180515.1", 
      "name": "Fedora-AtomicHost-Vagrant-28-20180515.1."

Here, it make sense to have "image_name" but I am not sure what is the significance of having "name"? It looks to me that in "name", we just trim off extensions (.x86_64.vagrant-virtualbox.box)

What if the fedmsg just linked to the compose? Then each caller would parse the existing compose metadata JSON, and we wouldn't need to define a new schema.

Did you mean that fedmsg published by atomic two week release script should only contain link to compose and then callers like fedora-website fetches json from the compose url? Something like -

"msg" {
   "url": "/pub/alt/atomic/stable/Fedora-Atomic-28-20180515.1/metadata"
 }

Technically it is possible that callers can fetch json data from the url link. One drawback I see with this approach is callers will also have to be aware of any changes made in compose metadata json schema. On the other hand, if all relevant information is provided by twoweek fedmsg, then caller will have to only look for changes made in fedmsg content.

Also, I hope that this new schema change will be only one time to accommodate multi-arch information. Sadly, I don't see a possible way to extend existing two week fedmsg schema with multi-arch content without breaking compatibility with older schema.

One drawback I see with this approach is callers will also have to be aware of any changes made in compose metadata json schema.

But we should probably be avoiding breaking that anyways, right?

I guess my strawman proposal here is that we keep the existing fedmsg schema, just add a new key msg/url key like you suggested, and teach websites to parse the compose data.

One drawback I see with this approach is callers will also have to be aware of any changes made in compose metadata json schema.

But we should probably be avoiding breaking that anyways, right?
I guess my strawman proposal here is that we keep the existing fedmsg schema, just add a new key msg/url key like you suggested, and teach websites to parse the compose data.

Makes sense and I have no objection with this.
To decide which option to choose, I am interested in knowing if we have any existing fedmsg as example which just pass url and callers do processing?

Also @dustymabe and @mohanboddu what are your opinion here?

One drawback I see with this approach is callers will also have to be aware of any changes made in compose metadata json schema.
But we should probably be avoiding breaking that anyways, right?
I guess my strawman proposal here is that we keep the existing fedmsg schema, just add a new key msg/url key like you suggested, and teach websites to parse the compose data.

Makes sense and I have no objection with this.
To decide which option to choose, I am interested in knowing if we have any existing fedmsg as example which just pass url and callers do processing?
Also @dustymabe and @mohanboddu what are your opinion here?

whatever works for me. It would be interesting to know if there were any other consumers of this fedmsg other than the website. none that I know of.

To decide which option to choose, I am interested in knowing if we have any existing fedmsg as example which just pass url and callers do processing?
Also @dustymabe and @mohanboddu what are your opinion here?

whatever works for me. It would be interesting to know if there were any other consumers of this fedmsg other than the website. none that I know of.

Even I am also not aware of any consumers other than fedora-website who uses this fedmsg data.

I gave a bit of more thought on which way to go among following two options-
1. Keep existing twoweek fedmsg schema and add a new key/value "url": "link" where link can be used by callers to fetch compose metadata json. Callers can further parse the json data from url to get additional information like multi-arch images detail, isos info, image size, etc. In this case, existing website Atomic page will not break with the change.
2. Update existing twoweek fedmsg with required information (like: multi-arch images detail, isos info, image size info) and adjust schema accordingly. In that case caller will not have to parse compose metadata json file and they can fully rely on fedmsg data. But, they will need to make one time change to adapt to new schema to get existing website Atomic page working.

Considering both options and number of applications relying on this fedmsg, I am more leaned towards updating existing fedmsg with multi-arch content. Updating existing webpage with new schema should be an easy task. Also, this will help us to stop depending upon autocloud and makes easy to update twoweek fedmsg with isos and image size information.

Considering both options and number of applications relying on this fedmsg, I am more leaned towards updating existing fedmsg with multi-arch content. Updating existing webpage with new schema should be an easy task. Also, this will help us to stop depending upon autocloud and makes easy to update twoweek fedmsg with isos and image size information.

+1

PR link - https://pagure.io/releng/pull-request/7519

releng.atomic.twoweek fedmsg now contains information for multi-arch images since last twoweek release - https://apps.fedoraproject.org/datagrepper/id?id=2018-d8b306e6-f070-4431-9389-7ad11f59a493&is_raw=true&size=extra-large

yay! nice work sinny!

Thanks Dusty!