In discussion with @kevin , @pbrobinson and @siosm we agreed it would make sense as a medium-term effort (say, a week to a month) to replace these with a more substantially engineered Python tool. I've sketched this out mentally a bit already; my initial idea is to conceive of it as a kind of generic engine for recognizing things in composes that look like container images (per the productmd metadata for the compose), and publishing them to registries in ways influenced by each image's particular properties plus per-deployment configuration. It will be somewhat like the compose scheduler part of https://pagure.io/fedora-qa/fedora_openqa , which takes compose metadata and schedules openQA jobs for particular images in it. The deployment config should probably be able to specify the properties of the image(s) it wants to publish, and some per-image configuration for what registry project(s) and tag(s) to publish each image to. There might be defaults, I guess, and maybe the option to specify the registry/ies at a higher config level so you don't have to specify it per image unless you want to use a different registry for a specific image. Something like that.
Again like fedora_openqa I'd intend to build a core library which operates on a compose location, a CLI front end to that, and also a fedmsg consumer interface which listens for "compose complete" messages and calls into the library. That could be a standalone consumer (as in fedora_openqa) or an infra toddler.
For The Future, as a potential way to implement gating for container publishing, the consumer could be configured to initially push to some kind of staging tag (or just a tag based on the compose's date or whatever), then also listen to resultsdb and waiverdb messages, determine if they relate to a container image, and if so, request a decision from greenwave and then publish the container to the relevant 'production' tag if greenwave says go (this is more or less how update push works in Bodhi at present), but that's just one possible idea.
Compared to https://pagure.io/releng/issue/12081 , that is a simpler short-term improvement, this is the medium-term redesign.
When do you need this? (YYYY/MM/DD) No specific date, it's an RFE.
When is this no longer needed or useful? (YYYY/MM/DD) When/if we replace our whole compose pipeline with something shinier, or maybe come up with an alternative medium-term design using Bodhi or something.
If we cannot complete your request, what is the impact? We'll continue to rely on one or two somewhat janky and tricky-to-work-on bash scripts for publishing container images, which don't have any tests, don't use compose metadata but instead have to have magic knowledge about Koji task properties, and occasionally publish the wrong thing due to timing issues when we're running candidate and nightly composes at the same time.
in the medium term we hope to replace both scripts with a more configurable and extensible Python tool that implements the generic job of "publishing container images from a compose to one or more registries". There will be a separate ticket for that.
That would be an improvement, but IMO a bigger improvement would be making it not a "script" but a "controller" running as e.g. a kubernetes pod, attempting to reconcile state continuously (edge triggered where possible), hooked up to observability. Or even just starting as a Kubernetes Job task, not a cron job.
This is how e.g. things work in e.g. OpenShift CI.
I mean sure, that sounds lovely, but I can write a better python thingy in a week or two, I can't write a kubernetes controller in a week or two. ;) It also sounds like integrating that into the existing compose process would be rather more complicated. This is scoped to the problem of "get the images we already produce as part of the existing compose process published", not "completely re-engineer how we produce container images in the first place".
I don't want to go too far down a 'let's modernize the compose process' road independently, because, you know, we have a whole...thing...going on to do that already.
Metadata Update from @phsmoura: - Issue tagged with: medium-gain, medium-trouble, ops
So, while working around this whole area (I hadn't got to this ticket yet because I've instead been trying to improve the actual compose story a bit), I was reminded that we have https://pagure.io/cloud-image-uploader/ for publishing cloud images. Which, really, is a very similar job.
So, as an alternative to this, perhaps we should just extend that tool to do this job too? I've filed https://pagure.io/cloud-image-uploader/issue/4 to see what folks think of that idea. If folks are amenable to that direction, I'll close this ticket in favor of that one.
ok, I think we're going in that direction, so let's close this.
Metadata Update from @adamwill: - Issue close_status updated to: upstream - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.