Issue #301: regular releases: uploading to clouds - cloud-sig

cloud-sig

#301 regular releases: uploading to clouds

Opened 4 years ago by dustymabe. Modified 8 months ago

In the 04/14/2020 community meeting we discussed what it would take to ship cloud images more regularly.

One work item that is on the Cloud working group is to control the uploads to clouds. We currently do this with fedimg but we want to replace it with ore like what is done to upload images for Fedora CoreOS.

We'd essentially make a new listener app that watches the fedmsg bus for when a new compose is done and then grabs the cloud images and uploads them to each respective cloud.

dustymabe commented 4 years ago

We discussed this in the community meeting yesterday.

@michaelgugino volunteered to give a look at this with guidance from @dustymabe!

Thanks @michaelgugino!

I think the way we achieve this is to write an openshift app that listens to fedmsgs, grabs the cloud image file, then kicks off upload/create to respective clouds.

example openshift app that listens to fedmsgs: https://pagure.io/releng/compose-tracker
we would run an upload using ore aws upload or ore gcloud upload commands.

ngompa commented 4 years ago

It might be worth considering using SUSE's Mash tool to do this. It looks like it covers all the bases we need, is generic, supports emitting messages on AMQP already, which means it'd be fedora-messaging compatible, and it supports all the clouds we care about (AWS, GCP, and Azure).

The only thing is the unfortunate name (it collides with the existing mash tool previously used in Fedora infrastructure)... We might want to talk to them about renaming it.

ngompa commented 4 years ago

Issue filed with the project on the name: https://github.com/SUSE-Enceladus/mash/issues/664

dustymabe commented 4 years ago

A few of us in the group met to discuss this on Monday.

Currently @michaelgugino is going to look at setting up an application to listen to fedmsg events and do some task. The current proposal is to use ore to upload cloud images, but @ngompa is looking to see if there are other healthily maintained and commonly used projects that we could possibly use instead.

michaelgugino commented 4 years ago

WIP: listening to fedora-messaging bus: https://github.com/michaelgugino/fed-image-watcher

michaelgugino commented 4 years ago

Potentially upload images with: https://github.com/SUSE-Enceladus/mash

dustymabe commented 4 years ago

Sorry to be late on this.. Finally got around to showing the example of how to upload to GCP using ore:

$ ore gcloud --project fedora-coreos-devel \
             --json-key /srv/dustymabe-service-account-fedora-coreos-devel.json \
             --log-level INFO upload --basename fedora-cloud                    \
             --bucket gs://fedora-coreos-devel-image-uploads/image-import       \
             --name fedora-cloud-base-gcp-33-1-2-x86-64                         \
             --file /srv/Fedora-Cloud-Base-GCP-33-1.2.x86_64.tar.gz             \
             --description "Fedora, Fedora Cloud Base, 33, x86_64 published on 2020-10-29" \
             --license https://compute.googleapis.com/compute/v1/projects/vm-options/global/licenses/enable-vmx

2020-10-29T20:24:30Z cli: Started logging at level INFO
Writing /srv/Fedora-Cloud-Base-GCP-33-1.2.x86_64.tar.gz to gs://fedora-coreos-devel-image-uploads/image-import/fedora-cloud-base-gcp-33-1-2-x86-64.tar.gz ...
(Sometimes this takes a few minutes)
Upload successful!
Creating image in GCE: fedora-cloud-base-gcp-33-1-2-x86-64...

Now I can launch instances with fedora-cloud-base-gcp-33-1-2-x86-64.

ngompa commented 11 months ago

Currently, @davdunc is working on an Ansible-based way to upload to clouds.

ngompa commented 8 months ago

This is being done now by @jcline with cloud-image-uploader.

Metadata Update from @ngompa:
- Issue assigned to jcline

8 months ago