In the 04/14/2020 community meeting we discussed what it would take to ship cloud images more regularly.
One work item that is on the Cloud working group is to control the uploads to clouds. We currently do this with fedimg but we want to replace it with ore like what is done to upload images for Fedora CoreOS.
We'd essentially make a new listener app that watches the fedmsg bus for when a new compose is done and then grabs the cloud images and uploads them to each respective cloud.
We discussed this in the community meeting yesterday.
@michaelgugino volunteered to give a look at this with guidance from @dustymabe!
I think the way we achieve this is to write an openshift app that listens to fedmsgs, grabs the cloud image file, then kicks off upload/create to respective clouds.
ore aws upload
ore gcloud upload
It might be worth considering using SUSE's Mash tool to do this. It looks like it covers all the bases we need, is generic, supports emitting messages on AMQP already, which means it'd be fedora-messaging compatible, and it supports all the clouds we care about (AWS, GCP, and Azure).
The only thing is the unfortunate name (it collides with the existing mash tool previously used in Fedora infrastructure)... We might want to talk to them about renaming it.
Issue filed with the project on the name: https://github.com/SUSE-Enceladus/mash/issues/664
A few of us in the group met to discuss this on Monday.
Currently @michaelgugino is going to look at setting up an application to listen to fedmsg events and do some task. The current proposal is to use ore to upload cloud images, but @ngompa is looking to see if there are other healthily maintained and commonly used projects that we could possibly use instead.
WIP: listening to fedora-messaging bus: https://github.com/michaelgugino/fed-image-watcher
Potentially upload images with: https://github.com/SUSE-Enceladus/mash
Sorry to be late on this.. Finally got around to showing the example of how to upload to GCP using ore:
$ ore gcloud --project fedora-coreos-devel \
--json-key /srv/dustymabe-service-account-fedora-coreos-devel.json \
--log-level INFO upload --basename fedora-cloud \
--bucket gs://fedora-coreos-devel-image-uploads/image-import \
--name fedora-cloud-base-gcp-33-1-2-x86-64 \
--file /srv/Fedora-Cloud-Base-GCP-33-1.2.x86_64.tar.gz \
--description "Fedora, Fedora Cloud Base, 33, x86_64 published on 2020-10-29" \
2020-10-29T20:24:30Z cli: Started logging at level INFO
Writing /srv/Fedora-Cloud-Base-GCP-33-1.2.x86_64.tar.gz to gs://fedora-coreos-devel-image-uploads/image-import/fedora-cloud-base-gcp-33-1-2-x86-64.tar.gz ...
(Sometimes this takes a few minutes)
Creating image in GCE: fedora-cloud-base-gcp-33-1-2-x86-64...
Now I can launch instances with fedora-cloud-base-gcp-33-1-2-x86-64.
to comment on this ticket.