#1039 RFE : modify sign+push process to take into account images/spins/media images
Closed: Fixed 11 months ago by arrfab. Opened a year ago by arrfab.

Our currently sign+push process was designed to automate signing rpm packages, and generating repositories (signed repodata too).
As some SIGs would like to start build images (.iso images, live media, etc), we should start designing a function to plumb in that process that wouldn't even try to sign rpm packages, but rather do something else, like downloading produced media image, and push it correct location.
That can also include other things like checksum files and detached signature on it


Metadata Update from @arrfab:
- Issue marked as blocking: #958
- Issue marked as blocking: #971
- Issue marked as blocking: #988

a year ago

Metadata Update from @arrfab:
- Issue assigned to arrfab

a year ago

Metadata Update from @arrfab:
- Issue tagged with: cbs, centos-build-pipeline, centos-common-infra, feature-request, high-gain, high-trouble, mini-initiative

a year ago

@tkopecek : simple question about kiwi
For image task, koji considers that it's like a pkg, so easy to then tag-build to various tasks
Example (on other koji) : https://koji.mbox.centos.org/koji/buildinfo?buildID=22804

Would that work the same for a kiwi build ? Ideally SIGs would just then have to tag-build to promote to other tags (which is triggering the sign+push process)

yes, it is exactly the same

yes, it is exactly the same

Perfect ! so that means we can use the same thing but just look at some tags to see if they have rpms (and so signing these/importing/call distrepo task) or just media images and so just download build and put it in place

@ngompa , @dcavalca , @tdawson : can you try a real (non --scratch one) build but at this stage let it be in -candidate tag (it's ignored at this stage by signing+push process and we can validate that it works ? I can then write/test a simple function to start moving things around in case of a media/image build for SIGs

Sure, I'll give it a shot momentarily.

Non-scratch build fails due to unregistered package name? https://cbs.centos.org/koji/taskinfo?taskID=3265887

which sounds logic if that's considered a build ? so https://sigs.centos.org/guide/cbs/#submit-a-build-on-cbs would apply (and like for a rpm pkg, one would just have to add it to the tags)

Here's a new run after registering the "package": https://cbs.centos.org/koji/taskinfo?taskID=3278044

ngompa@fedora ~/S/p/c/kiwi-descriptions (c9s)> cbs add-pkg --owner=ngompa hyperscale9s-spin_media-experimental-candidate CentOS-Stream-Hyperscale-Spin-OpenStack
Adding 1 packages to tag hyperscale9s-spin_media-experimental-candidate
ngompa@fedora ~/S/p/c/kiwi-descriptions (c9s)> cbs add-pkg --owner=ngompa hyperscale9s-spin_media-experimental-testing CentOS-Stream-Hyperscale-Spin-OpenStack
Adding 1 packages to tag hyperscale9s-spin_media-experimental-testing
ngompa@fedora ~/S/p/c/kiwi-descriptions (c9s)> cbs add-pkg --owner=ngompa hyperscale9s-spin_media-experimental-release CentOS-Stream-Hyperscale-Spin-OpenStack
Adding 1 packages to tag hyperscale9s-spin_media-experimental-release
ngompa@fedora ~/S/p/c/kiwi-descriptions (c9s)> bash ~/Scripts/centos-cbs-build-c9s-hsx-spin-kiwi.sh kiwi-descriptions CentOS-Stream-Hyperscale-Spin.kiwi oem OpenStack 0.n.20230314
Created task: 3278044
Task info: https://cbs.centos.org/koji/taskinfo?taskID=3278044
Watching tasks (this may be safely interrupted)...
3278044 kiwiBuild (noarch): free
3278044 kiwiBuild (noarch): free -> open (x86-5.cbs.centos.org)
  3278046 createKiwiImage (aarch64): free
  3278045 createKiwiImage (x86_64): free
  3278046 createKiwiImage (aarch64): free -> open (aarch64-01.rdu2.centos.org)
  3278045 createKiwiImage (x86_64): free -> open (x86-5.cbs.centos.org)

It looks like Koji doesn't like the *.sha256 checksum files:

BuildError: Unsupported file type: CentOS-Stream-Hyperscale-Spin-OpenStack.x86_64-9.0.0-0.n.20230314.qcow2.sha256

@tkopecek ^ ? any idea about that ?

Metadata Update from @arrfab:
- Assignee reset

a year ago

@tkopecek , @dcavalca : any news on the kiwi plugin issue for koji ? trying to revisit some open tickets and what's blocking and this one is still on the list

I'll look at it today (slipped through my queue)

Ah, I've forgotten to add some hints to docs. Kiwi produces a few more archivetypes than they enabled by default. It could be updated via running:

koji -p cbs call addArchiveType kiwi-checksum "Kiwi package list checksum" sha256
koji -p cbs call addArchiveType kiwi-changes "Kiwi changes file" "changes.xz changes"
koji -p cbs call addArchiveType kiwi-packages "Kiwi packages listing" packages
koji -p cbs call addArchiveTYpe kiwi-verified "Kiwi verified package list" verified

These for additional types should be sufficient for now. Anyway, if some kiwi target produce something new it could be added same way.

Metadata Update from @arrfab:
- Issue tagged with: centos-7-8s-eol

a year ago

Thanks a lot ... Added manually through koji calls !
@ngompa: can you eventually give it another try ?
If that works, I'll be able to play in .stg. env with a function to retrieve tagged builds and then we can decide on next steps for artifacts release automation

Metadata Update from @arrfab:
- Issue assigned to arrfab

a year ago

@ngompa kicked another kiwi build and this time it worked : https://cbs.centos.org/koji/buildinfo?buildID=43706

So I'll verify how to easily parse info and how to move artifacts somewhere

Metadata Update from @zlopez:
- Issue priority set to: High Priority (was: Needs Review)

a year ago

@ngompa : I had time to work on a function to process images builds and It's working in .stg. env.
For example, this is the kind of layout we can expect to land on mirrors :

└── images-latest
    └── x86_64
        ├── CentOS-Stream-Hyperscale-Appliance-OpenStack.x86_64-9.0.0-1.qcow2
        ├── CentOS-Stream-Hyperscale-Appliance-OpenStack.x86_64-9.0.0-1.qcow2.sha256
        ├── CentOS-Stream-Hyperscale-Appliance-OpenStack.x86_64-9.0.0-202305161312.qcow2
        ├── CentOS-Stream-Hyperscale-Appliance-OpenStack.x86_64-9.0.0-202305161312.qcow2.sha256
        ├── SHA256SUM
        └── SHA256SUM.asc

Dont' look at the names, it's just some tests I did and that's the name in the kiwi xml files
So it would create a directory with <arch> and then like for rpm repositories (except that would automatically adding/removing tagged/untagged builds)

If that works for you, I'd like to push that change on the new (migrated) signing process.
Worth reminding that we agreed to use (see altimage discussion).
So that would mean that we'd just need to recreate new tags for hyperscale (but we have automated the process, based on work done for altimages tags)

This is now pushed to prod (after some testing in .stg. env).
Closing this ticket and working in new ones for the new tags to be created for images builds

Metadata Update from @arrfab:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

11 months ago

Login to comment on this ticket.