#11860 Deploy new cloud image upload service
Closed: Fixed 12 days ago by jcline. Opened 2 months ago by jcline.

Describe what you would like us to do:

The Cloud SIG would like to automatically upload images as they are built. I believe this has historically been done with fedimg, which hasn't gotten an upstream commit in 6 years. There was a desire to have the images be uploaded by Ansible, so I wrote a playbook to upload Azure images, and a small AMQP consumer that runs the playbook whenever a new Azure image is published: https://pagure.io/cloud-image-uploader.

I would like to deploy this to Fedora's Infrastructure as Azure image uploads are currently a manual process, and then work towards uploading images for all the other cloud providers using this approach.

When do you need this to be done by? (YYYY/MM/DD)

Sooner is better, of course, but the workaround is to continue manually uploading images. I am happy to write the configuration, SOP, and ansible stuff to get it deployed, as well.

So, I assume this could run in openshift nicely?

If so, I think the next step would be to make a PR for it for staging.
Sadly it will likely not be easily testable in staging, as we don't do any composes there. :( But at least that should sort out the openshift side of things.

Then we can roll to prod. I assume it should be extendable to do aws/gcp/etc ?

Yes, we definitely would love to kill fedimg....

Metadata Update from @zlopez:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: high-gain, medium-trouble, ops

2 months ago

Yes, it should be fine in OpenShift. The only "oddity" is that it's writing out the images to /tmp and those can be a couple GBs. It doesn't need any persistent storage, though. I've tested it locally (in a container) so the OpenShift stuff will be the majority of the uncertainty, fortunately. I'll work on a PR for staging.

It should be pretty straightforward to add the other clouds, the AMQP consumer really just downloads the images, then maps the image name to an Ansible playbook to run, so we just need to drop in a couple more playbooks and add to an if image_name == ...block

So, this is all deployed, but currently failing with:

ERROR: Unable to create local directories(/.ansible/tmp): [Errno 13] Permission denied: b'/.ansible'
[2024-04-22 17:48:59,685 fedora_cloud_image_uploader.handler ERROR] Playbook failed with return code 5

I think it's somehow due to the venv setup there? not sure...

Yeah, I believe it just needs an adjustment to the Ansible variables it’s setting. I’m on vacation this week and didn’t bring a laptop, so I’ll fix it ASAP next Monday. Sorry :(

Yeah, no worries. I glanced at it, but wasn't sure where to adjust since it's using venvs?

But yeah, no worry... enjoy your time away!

So, I think this is all running now? Should we keep this open to track adding aws/amazon? Or close it out and just track that elsewhere? (possibly the 'retire fedimg' ticket)

I think tracking it in #11803 is good, so I'll go ahead and close this.

Metadata Update from @jcline:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

12 days ago

Login to comment on this ticket.

Boards 1
ops Status: Backlog