For upcoming Hyperscale work for building images, I need CBS to run Koji 1.28 or newer. Specifically because Koji 1.28 introduces full(ish) support for KIWI:
Kiwi plugin for building images based on XML description files was extended and refactored a bit, so it is now almost production-ready. We expect that in one or two releases we can flip it to first-class plugin.
We'd like to use this in Hyperscale and so we'd like CBS upgraded and the kiwiBuild task enabled for Hyperscale so we can use it.
kiwiBuild
cc: @dcavalca @davdunc @salimma @dbrandonjohnson
Metadata Update from @arrfab: - Issue tagged with: cbs, feature-request
[backlog refinement] There is another ticket that needs koji to be updated to newer version.
Koji 1.29.0 is out now, so we should update to that directly.
After chatting with Davide and Neal at Devconf, they are ok to maintain and troubleshoot the kiwi plugin in CBS. Since this is included with koji, the Infra team can install the packages on the builders and the hub. The Hyperscale SIG will contribute any necessary config items to the staging branches on https://github.com/centos/ansible-role-kojihub and https://github.com/centos/ansible-role-kojid
We expect to have a CBS staging environment soon so that we can play public contributions for review.
@arrfab, @dcavalca and @ngompa does this match what you expect?
Sounds good to us.
Metadata Update from @arrfab: - Issue assigned to arrfab
Metadata Update from @arrfab: - Issue tagged with: high-gain, high-trouble
the plan is to have a cbs.stg where we can test this, and on VMs (no bare-metal so no hypervisor, and I guess it's fine if kiwi doesn't need this). Rebuilding needed pkgs that will be needed on kojihub and kojid builders is also needed. Can we come with a list ? I've never played with kiwi so if you can come with an overview and help , we'd be happy to do that with you :)
The packages needed for the kiwiBuild command are:
kiwi-cli
kiwi-systemdeps
Both packages come from the kiwi source package in EPEL. I recommend sourcing from EPEL rather than rebuilding the dependency chain again.
kiwi
For some Hyperscale deliverables, we also need btrfs-progs and btrfs usable on the builder (either through the btrfs kmod from the kmods SIG or using the hyperscale kernel)
btrfs-progs
wrt btrfs-progs, let me paste what was said on irc in #centos-meeting :
(17:59:51) Eighth_Doctor: oh, and "btrfs-progs" for hyperscale kiwi builds (18:00:00) Eighth_Doctor: and the builder host needs to have btrfs available (18:00:23) arrfab: so we just need to rebuild these in in the infra tags (18:00:31) arrfab: hmm (18:00:38) davide: btrfs-progs is not in EPEL, but we do have it in hyperscale if that helps (18:00:47) arrfab: that was not mentioned (the need for host to be running hyperscale) (18:01:00) arrfab: as clearly bstinson told me that the goal is to migrate to imagebuilder and kiwi is temporary (18:01:16) arrfab: having one plugin is easy, modifying more for a tmp solution would need to be reconsidered then :/ 18:01:26) davide: I don't think it needs to necessarily run our variant -- it just needs to be able to handle btrfs filesystems │ (18:01:38) davide: so e.g. the kmod-btrfs from the kmods sig + btrfs-progrs would do the trick (18:01:40) bstinson: we discussed starting with existing builders (18:01:50) bstinson: let's do enablement there first │ (18:02:05) davide: yeah I think we can take this in two stages (18:02:08) arrfab: bstinson: but if their image need btrfs directly, that will not work ? (18:02:16) davide: first let's get kiwi working, then we can figure out the btrfs side of things (18:02:25) bstinson: there are non-btrfs variants that can be used (18:02:32) arrfab: *ack*
So let's try first with default setup as it will also run in AWS, on CentOS Stream 8 images, so with vanilla kernel
WRT kiwi, I kicked a rebuild in the infra8-buildtools-common tags (inherited through our roles for all koji environments)
Can you also validate that https://docs.pagure.org/koji/plugins/?highlight=kiwi#image-builds-using-kiwi is enough ? not a lot of info but seems that it just needs to be installed without any configuration for the plugin, and then create a kiwi-build group for the build tag and then adding needed rpm pkgs in that group
kiwi-build
Yes, that should be sufficient.
Realizing that because it needs to happen within the buildroot itself, we probably need to have it for 8s and 9s ? and should be tagged for all tags (through inheritance) We don't support (yet) 9s in our infra so if we need that we should start first looking at supporting 9s/el8 in our infra and tags ...
But otoh, if you enable epel{8,9} in your tags, that's possible to then call kiwi-build task assuming that we also ensure that pkgs are added with add-group-pkg in thekiwi-build group in your tag. Something to test ?
add-group-pkg
@ngompa : it seems it's not possible to have deps in epel8 anyway :
dnf install kiwi-cli kiwi-systemdeps Last metadata expiration check: 0:00:14 ago on Thu 01 Sep 2022 08:36:08 AM UTC. Error: Problem: package kiwi-cli-9.24.44-1.el8.noarch requires python3-kiwi = 9.24.44-1.el8, but none of the providers can be installed - conflicting requests - nothing provides python3.6dist(pyxattr) needed by python3-kiwi-9.24.44-1.el8.noarch (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
Would you mind working on missing deps there and then we can revisit ?
Metadata Update from @arrfab: - Issue priority set to: Waiting on Reporter (was: Needs Review)
Metadata Update from @arrfab: - Issue tagged with: blocked
@arrfab That's in PowerTools for c8s.
oh ! completely forgot about that (disabled by default) ... thanks ! .. something to verify for our build tags but iirc it's enabled everywhere.
hey, I just closed #907 so we have now publicly available https://cbs.stg.centos.org on which we can play with new features and have community members interact with it. I'll so start working on some kojid/kojihub role[s] change[s] to test that it's doable to have kiwi plugin working. Once validated we'll just be able to replay on cbs.centos.org
Just for my own tests, can you point me to some profiles that we can use for kiwi ? Let's start with something basic/straight forward
There should be some simple stuff here: https://pagure.io/centos-kiwi-examples
I meant for koji and kiwi-build task :-) It seems koji kiwi-build is expecting a profile or scm (so is that expecting path to .xml ?)
koji kiwi-build
koji kiwi-build --help Usage: koji kiwi-build [options] <target> <description_scm> <description_path> (Specify the --help global option for a list of other help options) Options: -h, --help show this help message and exit --scratch Perform a scratch build --release=RELEASE Release of the output image --repo=REPO Specify a repo that will override the repo used to install RPMs in the image. May be used multiple times. The build tag repo associated with the target is the default. --noprogress Do not display progress of the upload --kiwi-profile=KIWI_PROFILE Select profile from description file --make-prep Run 'make prep' in checkout before starting the build --can-fail=ARCH1,ARCH2,... List of archs which are not blocking for build (separated by commas. --arch=ARCHES Limit arches to this subset --nowait --wait Wait on the image creation, even if running in the background
path is to the directory containing config.xml, I believe. Try the complex-build one with OpenStack?
config.xml
OpenStack
@ngompa trying to look at this but can you come with some trees/options that we can test with koji ? (not your standalone wrapper) also, worth knowing that we can't use metalink= for repositories as kojid builders don't have access to internet and also we have to point to valid but static repositories (like mirror.stream.centos.org which is then available)
Koji is supposed to rewrite the repos automatically. Is it not doing that?
there is no koji doc about how to use it btw :-)
Just a little bit of progress :
I created a git repo to have some modified/simplified .xml for kiwi (based on yours) and not using metalink (as it will not work in our infra) and then kicked :
cbs-stg kiwi-build --scratch hyperscale9s-packages-spin-el9s git+https://git.centos.org/centos/infra-playground#95c4d5d2e15b35255ccf213fec4d06f92d8e8ffc kiwi/simple-build/ Created task: 123 Task info: https://cbs.stg.centos.org/koji/taskinfo?taskID=123 Watching tasks (this may be safely interrupted)... 123 kiwiBuild (noarch): free 123 kiwiBuild (noarch): free -> open (kojid1.stg.centos.org) 124 createKiwiImage (x86_64): free 124 createKiwiImage (x86_64): free -> open (kojid1.stg.centos.org) 124 createKiwiImage (x86_64): open (kojid1.stg.centos.org) -> closed 0 free 1 open 1 done 0 failed 123 kiwiBuild (noarch): open (kojid1.stg.centos.org) -> closed 0 free 0 open 2 done 0 failed 123 kiwiBuild (noarch) completed successfully
=> https://cbs.stg.centos.org/koji/taskinfo?taskID=124
I tried to just build a cloud image but had to merge all .xml files into one otherwise koji kiwi plugin fails with :
Sep 13 07:14:49 kojid1.stg.euw1.centos.org kojid[267560]: with open(file, \'rb\') as fp: Sep 13 07:14:49 kojid1.stg.euw1.centos.org kojid[267560]: FileNotFoundError: [Errno 2] No such file or directory: \'this://./repositories/core.xml\' Sep 13 07:14:49 kojid1.stg.euw1.centos.org kojid[267560]: '>
I tried once again but it will be probably my last test (you're now welcome to test it yourself on cbs.stg with a TLS cert from accounts.stg.centos.org and pointing to cbs.stg.centos.org for koji profile)
it fails again but different issue (from root.log ):
DEBUG util.py:446: [ INFO ]: 07:26:54 | Processing SELinux file security contexts DEBUG util.py:446: [ WARNING ]: 07:26:54 | Could not parse setfiles output DEBUG util.py:444: [ ERROR ]: 07:27:05 | KiwiCommandError: chroot: stderr: setfiles: Could not set context for /run/motd: Invalid argument DEBUG util.py:444: setfiles: Could not set context for /run/motd.d: Invalid argument DEBUG util.py:444: setfiles: Could not set context for /var/lib/systemd/coredump: Invalid argument DEBUG util.py:444: setfiles: Could not set context for /usr/lib/systemd/user/dbus-broker.service: Invalid argument DEBUG util.py:444: setfiles: Could not set context for /usr/lib/systemd/systemd-network-generator: Invalid argument DEBUG util.py:444: setfiles: Could not set context for /usr/bin/rpmdb: Invalid argument DEBUG util.py:444: , stdout: 1k 2k 3k 4k 5k 6k 7k 8k 9k 10k 11k 12k 13k 14k 15k 16k 17k 18k 19k 20k 21k 22k 23k 24k 25k 26k 27k 28k 29k / 100.0% DEBUG util.py:446: [ INFO ]: 07:27:05 | Cleaning up SystemPrepare instance DEBUG util.py:598: Child return code was: 1
Worth knowing that host is a centos stream 8 VM (we don't support yet stream 9 in centos infra) and it seems it doens't allow kojid kiwi plugin to set correct selinux context either :
From audit.log :
type=AVC msg=audit(1663054015.307:54713): avc: denied { mac_admin } for pid=279225 comm="setfiles" capability=33 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0 type=AVC msg=audit(1663054015.308:54714): avc: denied { mac_admin } for pid=279225 comm="setfiles" capability=33 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0 type=AVC msg=audit(1663054015.813:54715): avc: denied { mac_admin } for pid=279225 comm="setfiles" capability=33 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0 type=AVC msg=audit(1663054016.057:54717): avc: denied { mac_admin } for pid=279225 comm="setfiles" capability=33 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0 type=AVC msg=audit(1663054016.169:54718): avc: denied { mac_admin } for pid=279225 comm="setfiles" capability=33 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0 type=AVC msg=audit(1663054019.551:54719): avc: denied { mac_admin } for pid=279225 comm="setfiles" capability=33 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0
We run with selinux in enforcing mode everywhere and it should stay like that, so can you verify with kiwi and koji about how to support that ? we can then revisit later if we want to enable it on centos cbs builders ?
I just tried with setenforce 0 on that isolated builder (and temporary for that test) and it then works :
setenforce 0
https://cbs.stg.centos.org/koji/taskinfo?taskID=138
So, apart from the selinux issue (already trying to see how to rebuild a custom .pp eventually), I gave the same build another try (no difference) but it now fails : https://cbs.stg.centos.org/kojifiles/work/tasks/142/142/root.log
DEBUG util.py:446: [ INFO ]: 08:28:55 | Creating volume group systemVG DEBUG util.py:444: [ ERROR ]: 08:28:55 | KiwiCommandError: vgcreate: stderr: /dev/systemVG: already exists in filesystem DEBUG util.py:444: Run `vgcreate --help' for more information. DEBUG util.py:444: , stdout: (no output on stdout)
From what I can see it initially created /dev/loop devices but it seems it doesn't clean these after builds, so new job would complain about some existing VG/LV on it. Something to report to upstream koji ?
cc: @tkopecek
Interesting, I'll try it locally. I'm running it definitely with enforcing selinux. dev leak seems to me like a kiwi issue, but will check it.
Finally found why : it's just that kojid by default doesn't clean-up directly the buildroots, but if you wait 2 minutes (default for buildroot_basic_cleanup_delay=120) it then just works fine.
buildroot_basic_cleanup_delay=120
For the selinux policy, I also built a custom policy and it seems to work :
cat /root/selinux/centos-kojid-kiwi.te ; echo ====== ; echo "selinux status is : $(getenforce)" module centos-kojid-kiwi 1.0; require { type unconfined_service_t; class capability2 mac_admin; } #============= unconfined_service_t ============== allow unconfined_service_t self:capability2 mac_admin; ====== selinux status is : Enforcing
I'll modify our ansible kojid role to distribute (and load) that .pp file (I don't like it but better than whole setenforce 0 afaics)
I submitted some scratch builds and they all worked : https://cbs.stg.centos.org/koji/index
@tkopecek I think the Koji kiwi config flattener doesn't recognize this:// as a protocol yet. It's basically a relative version of file:// so you can translate this:// to file://${PWD} and resolve the URLs.
this://
file://
file://${PWD}
Metadata Update from @arrfab: - Issue untagged with: blocked
https://pagure.io/koji/issue/3495
@tkopecek It also seems that based on the output from this build log that it's not using the kiwi description koji generates: https://cbs.stg.centos.org/koji/taskinfo?taskID=160
We should probably be renaming any existing config.xml or *.kiwi files out of the way so that the generated one is used.
*.kiwi
Maybe better to use directly --kiwi-file instead of --description.
--kiwi-file
--description
https://pagure.io/koji/issue/3497
When using --kiwi-file, you still need --description. But yeah, it may make sense to set that.
@ngompa happy to just deploy kiwi plugin on cbs.centos.org (we'll have one host in the image channel) as it seems to work at this stage. Thoughts ?
image
I pushed that change and it's now enabled on cbs (kojihub and kojid) but nothing is configured for tags (as we need to add the kiwi-build group on the build tags .. Waiting for feedback on which one you'd like to see it enabled
Let's configure the kiwi-build group to the hyperscale8s-spin_media-experimental-el8s-build and hyperscale8s-spin_media-main-el8s-build tags.
hyperscale8s-spin_media-experimental-el8s-build
hyperscale8s-spin_media-main-el8s-build
I also noticed that the hyperscale8s-spin_media-experimental-el8s-build tag doesn't have hyperscale8s-packages-experimental-release in its inheritance and it should be added.
hyperscale8s-packages-experimental-release
The kiwi-build group needs the following: kiwi-cli, kiwi-systemdeps, distribution-gpg-keys, btrfs-progs.
distribution-gpg-keys
(Having the btrfs-progs userspace in the group will prepare us for later when we have Btrfs-enabled CBS builders)
We also need versions of the hyperscale8s-spin_media-* tags for c9s.
hyperscale8s-spin_media-*
The formula we want here is the same as for c8s: hyperscale9s-spin_media-{main,experimental}-{el9s-build,candidate,testing,release}.
hyperscale9s-spin_media-{main,experimental}-{el9s-build,candidate,testing,release}
The hyperscale9s-spin_media-*-el9s-build tags should be configured the same way the hyperscale8s-spin_media-*-el8s-build tags were.
hyperscale9s-spin_media-*-el9s-build
hyperscale8s-spin_media-*-el8s-build
Let's try first with the 8s ones :
koji add-group hyperscale8s-spin_media-experimental-el8s-build kiwi-build koji add-group-pkg hyperscale8s-spin_media-experimental-el8s-build kiwi-build kiwi-cli kiwi-systemdeps distribution-gpg-keys btrfs-progs koji add-group hyperscale8s-spin_media-main-el8s-build kiwi-build koji add-group-pkg hyperscale8s-spin_media-main-el8s-build kiwi-build kiwi-cli kiwi-systemdeps distribution-gpg-keys btrfs-progs
Also added the inheritance on hyperscale8s-spin_media-experimental-el8s-build :
cbs taginfo hyperscale8s-spin_media-experimental-el8s-build Tag: hyperscale8s-spin_media-experimental-el8s-build [2341] Arches: x86_64 aarch64 Groups: build, livecd-build, livemedia-build, srpm-build Tag options: mock.package_manager : 'dnf' mock.yum.module_hotfixes : 1 rpm.macro.vendor : 'CentOS Hyperscale SIG' This tag is a buildroot for one or more targets Current repo: repo#964125: 2022-09-22 03:47:51.050014+00:00 Targets that build from this tag: hyperscale8s-spin_media-experimental-el8s External repos: 5 centos8s-cr (http://mirror.centos.org/centos/8-stream//cr/$arch/os/, merge mode: bare), arches: inherited from tag 10 centos8s-extras (http://mirror.centos.org/centos/8-stream//extras/$arch/os/, merge mode: bare), arches: inherited from tag 15 centos8s-powertools (http://mirror.centos.org/centos/8-stream//PowerTools/$arch/os/, merge mode: bare), arches: inherited from tag 20 centos8s-appstream (http://mirror.centos.org/centos/8-stream//AppStream/$arch/os/, merge mode: bare), arches: inherited from tag 25 centos8s-baseos (http://mirror.centos.org/centos/8-stream//BaseOS/$arch/os/, merge mode: bare), arches: inherited from tag 30 epel8 (https://cbs.centos.org/kojifiles/repos/epel/8/Everything/$arch/, merge mode: bare), arches: inherited from tag 35 epel8-next (https://cbs.centos.org/kojifiles/repos/epel/next/8/Everything/$arch/, merge mode: bare), arches: inherited from tag Inheritance: 0 .... hyperscale8s-packages-main-release [2249] 5 .... buildsys8s-release [1866] 10 .... hyperscale8s-spin_media-experimental-candidate [2338] 20 .... hyperscale8s-packages-hotfixes-release [2305] 25 .... hyperscale8s-packages-spin-release [2301] 30 .... hyperscale8s-packages-experimental-release [2245]
Can you give it now a try ? worth knowing that it should stay in -candidate for now as current signing+push releng process doesn't know anything about non RPM repositories (we recently added a function for DuD .iso artifacts for kmods SIG as one example) but that can be then another ticket
-candidate
Per discussion with @ngompa on irc, I built https://src.fedoraproject.org/rpms/koji/c/c15b855e4e075e2d5ffa586c9ed7fc870796393f?branch=rawhide , which is koji 1.30.0-2 , which has the backported patches/fixes for kiwi (merged in upstream koji) It's now deployed on CBS : https://cbs.centos.org/koji/api
Per discussion with @ngompa on irc (follow-up) I already also created the CentOS Stream 9 tags :
cbs taginfo hyperscale9s-spin_media-main-el9s-build Tag: hyperscale9s-spin_media-main-el9s-build [2661] Arches: x86_64 aarch64 Groups: build, kiwi-build, srpm-build Tag options: mock.new_chroot : 0 mock.package_manager : 'dnf' mock.yum.module_hotfixes : 1 rpm.macro.vendor : 'CentOS Hyperscale SIG' This tag is a buildroot for one or more targets Current repo: repo#965929: 2022-09-27 07:19:42.905150+00:00 Targets that build from this tag: hyperscale9s-spin_media-main-el9s External repos: 5 centos9s-baseos (http://mirror.stream.centos.org/9-stream/BaseOS/$arch/os/, merge mode: bare), arches: inherited from tag 10 centos9s-appstream (http://mirror.stream.centos.org/9-stream/AppStream/$arch/os/, merge mode: bare), arches: inherited from tag 15 centos9s-crb (http://mirror.stream.centos.org/9-stream/CRB/$arch/os/, merge mode: bare), arches: inherited from tag 20 epel9 (https://cbs.centos.org/kojifiles/repos/epel/9/Everything/$arch/, merge mode: bare), arches: inherited from tag 25 epel9-next (https://cbs.centos.org/kojifiles/repos/epel/next/9/Everything/$arch/, merge mode: bare), arches: inherited from tag Inheritance: 0 .... hyperscale9s-packages-main-release [2378] 5 .... buildsys9s-release [2363] 10 .... hyperscale9s-spin_media-main-candidate [2658] 15 .... hyperscale9s-packages-hotfixes-release [2414] 20 .... hyperscale9s-packages-spin-release [2382] cbs taginfo hyperscale9s-spin_media-experimental-el9s-build Tag: hyperscale9s-spin_media-experimental-el9s-build [2665] Arches: x86_64 aarch64 Groups: build, kiwi-build, srpm-build Tag options: mock.new_chroot : 0 mock.package_manager : 'dnf' mock.yum.module_hotfixes : 1 rpm.macro.vendor : 'CentOS Hyperscale SIG' This tag is a buildroot for one or more targets Current repo: repo#965930: 2022-09-27 07:21:13.216410+00:00 Targets that build from this tag: hyperscale9s-spin_media-experimental-el9s External repos: 5 centos9s-baseos (http://mirror.stream.centos.org/9-stream/BaseOS/$arch/os/, merge mode: bare), arches: inherited from tag 10 centos9s-appstream (http://mirror.stream.centos.org/9-stream/AppStream/$arch/os/, merge mode: bare), arches: inherited from tag 15 centos9s-crb (http://mirror.stream.centos.org/9-stream/CRB/$arch/os/, merge mode: bare), arches: inherited from tag 20 epel9 (https://cbs.centos.org/kojifiles/repos/epel/9/Everything/$arch/, merge mode: bare), arches: inherited from tag 25 epel9-next (https://cbs.centos.org/kojifiles/repos/epel/next/9/Everything/$arch/, merge mode: bare), arches: inherited from tag Inheritance: 0 .... hyperscale9s-packages-main-release [2378] 5 .... buildsys9s-release [2363] 10 .... hyperscale9s-spin_media-experimental-candidate [2662] 15 .... hyperscale9s-packages-hotfixes-release [2414] 20 .... hyperscale9s-packages-spin-release [2382] 25 .... hyperscale9s-packages-experimental-release [2410]
Can you verify and validate please ?
@tkopecek So I tried to run a build and Koji seems to choke on it? https://cbs.centos.org/koji/taskinfo?taskID=3032622
The command I ran was:
$ cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#f7475459b17b75adaf3b048bf4ec166830da3218 ./ --kiwi-profile=OpenStack --arch=x86_64
Hmm, that patch was improved a lot during testing. Isn't it the original version? + Now you have to specify path with the xml file ./ -> ./xyz.kiwi to not allow kiwi to choose something else.
$ cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#f7475459b17b75adaf3b048bf4ec166830da3218 ./CentOS-Stream-Hyperscale-Spin.kiwi --kiwi-profile=OpenStack --arch=x86_64 Created task: 3032636 Task info: https://cbs.centos.org/koji/taskinfo?taskID=3032636 Watching tasks (this may be safely interrupted)... 3032636 kiwiBuild (noarch): free 3032636 kiwiBuild (noarch): free -> open (x86-5.cbs.centos.org) 3032636 kiwiBuild (noarch): open (x86-5.cbs.centos.org) -> FAILED: GenericError: Repo must contain only one .kiwi file. Relevant logs: http://cbs.centos.org/kojifiles/work/tasks/2636/3032636/checkout.log 0 free 0 open 0 done 1 failed 3032636 kiwiBuild (noarch) failed
Yes - it is definitely the old code - This error message is not there anymore. https://pagure.io/koji/pull-request/3498#request_diff
@arrfab, did you deploy koji 1.30.0-2 for CBS?
yes, and built from your src.rpm : https://cbs.centos.org/koji/buildinfo?buildID=41314 (so with your backported patches, not verified)
@ngompa check the 3498 patch - testing found a bunch of problems there.
Submitted PR to update patches in the package: https://src.fedoraproject.org/rpms/koji/pull-request/14
Revisiting ticket without feedback and realizing that your PR was merged. Submitting the build at the cbs side and we can deploy on stg and update cbs.centos.org.
@ngompa : built and tested yesterday update on https://cbs.stg.centos.org and working so promoted the build to -release and bumped version to 1.30.0-3 in ansible inventory. https://cbs.centos.org is now running that version, both on hub and kojid level.
-release
1.30.0-3
Can you test and let me know if that works for you ?
@tkopecek New interesting error: https://cbs.centos.org/koji/taskinfo?taskID=3048455
Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/koji/daemon.py", line 1468, in runTask response = (handler.run(),) File "/usr/lib/python3.6/site-packages/koji/tasks.py", line 335, in run return koji.util.call_with_argcheck(self.handler, self.params, self.opts) File "/usr/lib/python3.6/site-packages/koji/util.py", line 271, in call_with_argcheck return func(*args, **kwargs) File "/usr/lib/koji-builder-plugins/kiwi.py", line 362, in handler desc, types = self.prepareDescription(path, name, version, repos, arch) File "/usr/lib/koji-builder-plugins/kiwi.py", line 205, in prepareDescription inc = xml.dom.minidom.parse(path) # nosec File "/usr/lib64/python3.6/xml/dom/minidom.py", line 1958, in parse return expatbuilder.parse(file) File "/usr/lib64/python3.6/xml/dom/expatbuilder.py", line 910, in parse with open(file, 'rb') as fp: NotADirectoryError: [Errno 20] Not a directory: '/var/lib/mock/hyperscale9s-spin_media-experimental-el9s-build-159345-971833/root/chroot_tmpdir/kiwi-descriptions/./CentOS-Stream-Hyperscale-Spin.kiwi/repositories/core.xml'
Command used:
$> cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#f7475459b17b75adaf3b048bf4ec166830da3218 ./CentOS-Stream-Hyperscale-Spin.kiwi --kiwi-profile=OpenStack --arch=x86_64
Let us know when it will be all working and only then we'll consider updating again (cbs.stg.centos.org should be used first to validate this works before we update cbs.centos.org again)
I'm waiting on this PR getting merged: https://src.fedoraproject.org/rpms/koji/pull-request/15
Looking at this and it was merged . Rebuilt on cbs for infra tags : https://cbs.centos.org/koji/buildinfo?buildID=41715 Modified ansible to deploy/upgrade on STG : https://cbs.stg.centos.org/koji/api
As it seems trivial, let me schedule the cbs.centos.org upgrade but ideally we'd validate that it works on STG through some scratch builds
Promoted to release tag and also applied by ansible on cbs : https://cbs.centos.org/koji/api @ngompa : let me know if that works for you and if it does we'll (finally :-) ) be able to close this ticket. We'll just have to create another one for the releng part and decide on tags and how to automatically promote content to mirror network (as it can't be actual process for rpms)
release
releng
Uhh...
$> cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#f7475459b17b75adaf3b048bf4ec166830da3218 ./CentOS-Stream-Hyperscale-Spin.kiwi --kiwi-profile=OpenStack --arch=x86_64 2022-10-27 08:12:59,584 [ERROR] koji: ParameterError: Invalid type for value '['OpenStack']': <class 'list'>, expected type <class 'str'>
@tkopecek, this seems goofy?
@ngompa so, my suggestion : I'll let you work with @tkopecek until you have something that works. Then we can give it a try on .stg. and only then consider deploying on cbs.centos.org.
So just update this ticket when you have something that can be tested and that is also tested upstream (koji side)
@arrfab koji 1.31 should have everything we need to do this properly now: https://koji.fedoraproject.org/koji/buildinfo?buildID=2091321
Can you get this deployed so we can try it?
@ngompa rebuilt : https://cbs.centos.org/koji/buildinfo?buildID=42022 and tagged to -testing so I'll give it a quick deploy on cbs.stg. Ideally I would have asked to test an image build on cbs.stg env but I guess all your complex examples would require building from hyperscale tags, which don't exist on cbs.stg. So let me just do a quick deploy/check and then I'll update cbs.centos.org too (by looking at koji Release Notes it shouldn't be introducing big changes)
-testing
@ngompa , @tkopecek : tested an update and Release Notes are mentioning the traditional schema update for postgres, but it's not in that rpm package , but it exists in upstream repo
Would you mind having that fixed and have a new rpm ?
@ngompa, @tkopecek : just gave it a quick try and the error Neal reported is still the same :
cbs-stg kiwi-build --scratch hyperscale9s-packages-spin-el9s git+https://git.centos.org/centos/infra-playground#f1a4a7c0b9c8cec74e836f0050b4208205565078 kiwi/complex-build/centos-test.kiwi --kiwi-profile=Openstack 2022-11-22 08:34:52,905 [ERROR] koji: ParameterError: Invalid type for value '['Openstack']': <class 'list'>, expected type <class 'str'>
@arrfab It is there: https://cbs.centos.org/koji/fileinfo?rpmID=411028&filename=/usr/share/doc/koji/docs/schema-update-1.30-1.31.sql
And build fails me on something else:
INFO: Running in chroot: ['kiwi-ng', '--profile', 'Openstack', '--kiwi-file', 'centos-test.x86_64.kiwi', 'system', 'build', '--description', 'infra-playground/kiwi/complex-build', '--target-dir', '/builddir/result/image'] Start: chroot ['kiwi-ng', '--profile', 'Openstack', '--kiwi-file', 'centos-test.x86_64.kiwi', 'system', 'build', '--description', 'infra-playground/kiwi/complex-build', '--target-dir', '/builddir/result/image'] [ INFO ]: 11:54:51 | Reading runtime config file: '/etc/kiwi.yml' [ INFO ]: 11:54:51 | Loading XML description [ INFO ]: 11:54:51 | Support for XML markup available [ ERROR ]: 11:54:51 | KiwiProfileNotFound: profile Openstack not found for host arch x86_64 Finish: chroot ['kiwi-ng', '--profile', 'Openstack', '--kiwi-file', 'centos-test.x86_64.kiwi', 'system', 'build', '--description', 'infra-playground/kiwi/complex-build', '--target-dir', '/builddir/result/image']
Profile names are case sensitive, I believe. It's OpenStack, not Openstack.
Openstack
I know and I saw it and forgot to update my comment here as I obviously tried then with correct name :
cbs-stg kiwi-build --scratch hyperscale9s-packages-spin-el9s git+https://git.centos.org/centos/infra-playground#f1a4a7c0b9c8cec74e836f0050b4208205565078 kiwi/complex-build/centos-test.kiwi --kiwi-profile=OpenStack 2022-11-22 12:04:32,062 [ERROR] koji: ParameterError: Invalid type for value '['OpenStack']': <class 'list'>, expected type <class 'str'>
Did you update the CLI locally? Once I did, it seems to have started:
ngompa@Belldandy-Slimbook ~> cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#4f30f9025a02b18585717d6d037e1c78a43da626 ./CentOS-Stream-Hyperscale-Spin.kiwi --kiwi-profile=OpenStack --arch=x86_64 Created task: 3100877 Task info: https://cbs.centos.org/koji/taskinfo?taskID=3100877 Watching tasks (this may be safely interrupted)... 3100877 kiwiBuild (noarch): free 3100877 kiwiBuild (noarch): free -> open (x86-5.cbs.centos.org) 3100878 createKiwiImage (x86_64): free 3100878 createKiwiImage (x86_64): free -> open (x86-5.cbs.centos.org)
@arrfab It is there: https://cbs.centos.org/koji/fileinfo?rpmID=411028&filename=/usr/share/doc/koji/docs/schema-update-1.30-1.31.sql And build fails me on something else: INFO: Running in chroot: ['kiwi-ng', '--profile', 'Openstack', '--kiwi-file', 'centos-test.x86_64.kiwi', 'system', 'build', '--description', 'infra-playground/kiwi/complex-build', '--target-dir', '/builddir/result/image'] Start: chroot ['kiwi-ng', '--profile', 'Openstack', '--kiwi-file', 'centos-test.x86_64.kiwi', 'system', 'build', '--description', 'infra-playground/kiwi/complex-build', '--target-dir', '/builddir/result/image'] [ INFO ]: 11:54:51 | Reading runtime config file: '/etc/kiwi.yml' [ INFO ]: 11:54:51 | Loading XML description [ INFO ]: 11:54:51 | Support for XML markup available [ ERROR ]: 11:54:51 | KiwiProfileNotFound: profile Openstack not found for host arch x86_64 Finish: chroot ['kiwi-ng', '--profile', 'Openstack', '--kiwi-file', 'centos-test.x86_64.kiwi', 'system', 'build', '--description', 'infra-playground/kiwi/complex-build', '--target-dir', '/builddir/result/image']
And build fails me on something else: INFO: Running in chroot: ['kiwi-ng', '--profile', 'Openstack', '--kiwi-file', 'centos-test.x86_64.kiwi', 'system', 'build', '--description', 'infra-playground/kiwi/complex-build', '--target-dir', '/builddir/result/image'] Start: chroot ['kiwi-ng', '--profile', 'Openstack', '--kiwi-file', 'centos-test.x86_64.kiwi', 'system', 'build', '--description', 'infra-playground/kiwi/complex-build', '--target-dir', '/builddir/result/image'] [ INFO ]: 11:54:51 | Reading runtime config file: '/etc/kiwi.yml' [ INFO ]: 11:54:51 | Loading XML description [ INFO ]: 11:54:51 | Support for XML markup available [ ERROR ]: 11:54:51 | KiwiProfileNotFound: profile Openstack not found for host arch x86_64 Finish: chroot ['kiwi-ng', '--profile', 'Openstack', '--kiwi-file', 'centos-test.x86_64.kiwi', 'system', 'build', '--description', 'infra-playground/kiwi/complex-build', '--target-dir', '/builddir/result/image']
When trying to update the schema :
psql koji < /usr/share/doc/koji/docs/schema-update-1.30-1.31.sql BEGIN ERROR: syntax error at or near "INCLUDE" LINE 1: ...sion || '-' || release || '.' || arch || '.rpm')) INCLUDE (i... ^ ROLLBACK
@tkopecek ^
Did you update the CLI locally? Once I did, it seems to have started: ngompa@Belldandy-Slimbook ~> cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#4f30f9025a02b18585717d6d037e1c78a43da626 ./CentOS-Stream-Hyperscale-Spin.kiwi --kiwi-profile=OpenStack --arch=x86_64 Created task: 3100877 Task info: https://cbs.centos.org/koji/taskinfo?taskID=3100877 Watching tasks (this may be safely interrupted)... 3100877 kiwiBuild (noarch): free 3100877 kiwiBuild (noarch): free -> open (x86-5.cbs.centos.org) 3100878 createKiwiImage (x86_64): free 3100878 createKiwiImage (x86_64): free -> open (x86-5.cbs.centos.org)
It's true that I haven't locally, as it's not available (yet) in epel9 .. let me try on the cbs.stg setup
Did you update the CLI locally? Once I did, it seems to have started: ngompa@Belldandy-Slimbook ~> cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#4f30f9025a02b18585717d6d037e1c78a43da626 ./CentOS-Stream-Hyperscale-Spin.kiwi --kiwi-profile=OpenStack --arch=x86_64 Created task: 3100877 Task info: https://cbs.centos.org/koji/taskinfo?taskID=3100877 Watching tasks (this may be safely interrupted)... 3100877 kiwiBuild (noarch): free 3100877 kiwiBuild (noarch): free -> open (x86-5.cbs.centos.org) 3100878 createKiwiImage (x86_64): free 3100878 createKiwiImage (x86_64): free -> open (x86-5.cbs.centos.org) It's true that I haven't locally, as it's not available (yet) in epel9 .. let me try on the cbs.stg setup
I enabled updates-testing temporarily to upgrade the koji client.
We're now failing on something that isn't from KIWI, but from the Koji repos?
DEBUG util.py:444: [ ERROR ]: 11:08:34 | KiwiInstallPhaseFailed: System package installation failed: Module or Group 'core' is not available.
Are the comps group metadata not available through the external repos merged into the build repo?
@ngompa : per koji documentation, and @tkopecek can confirm, it seems that koji will just replace all references to repo with the ones from koji so internal. But I don't know how koji is importing comps.xml and groups into koji
Also, koji is still not updated on cbs.centos.org as long as we can't have it properly working on cbs.stg.centos.org
upstream koji issue for the schema update : https://pagure.io/koji/issue/3594
INCLUDE
comps are always overriden by koji's groups, so external comps are lost during the process. Anyway, easy way to import needed ones is via 'koji import-comps' command (to buildtag).
Importing the CentOS and EPEL comps groups into the build tag should fix the problem I'm having now, since DNF is failing on resolving comps groups.
It's another step that we'll have to take into accounts but we can automate that part when a SIG will ask for some tags and kiwi support. FWIW, I kicked a scratch build on cbs.stg env and it worked then :
koji kiwi-build --scratch hyperscale9s-packages-spin-el9s git+https://git.centos.org/centos/infra-playground#f1a4a7c0b9c8cec74e836f0050b4208205565078 kiwi/complex-build/centos-test.kiwi --kiwi-profile=OpenStack Created task: 339 Task info: https://cbs.stg.centos.org/koji/taskinfo?taskID=339 Watching tasks (this may be safely interrupted)... 339 kiwiBuild (noarch): free 339 kiwiBuild (noarch): free -> open (kojid1.stg.centos.org) 340 createKiwiImage (x86_64): free 340 createKiwiImage (x86_64): free -> open (kojid1.stg.centos.org) 340 createKiwiImage (x86_64): open (kojid1.stg.centos.org) -> closed 0 free 1 open 1 done 0 failed 339 kiwiBuild (noarch): open (kojid1.stg.centos.org) -> closed 0 free 0 open 2 done 0 failed 339 kiwiBuild (noarch) completed successfully
So eventually I can consider upgrading now the cbs.centos.org env with that version too, and document the .sql issue for koji upgrade (mentioned above)
@tkopecek Where's the $target_dir/build/image-root.log file? I thought we were tailing/archiving that log file per architecture? I don't see it in https://cbs.stg.centos.org/koji/taskinfo?taskID=339
$target_dir/build/image-root.log
We're pretty much set aside from missing that log file, so we should be good for production use, though I'd like that log file captured asap.
Issue filed: https://pagure.io/koji/issue/3597
@ngompa : cbs.centos.org updated to 1.31 (including builders) today : https://cbs.centos.org/koji/api
For the comps to import, I realized that our internal mirror for epel doens't provide/import comps but that's something I can have a look at later In the meantime, I just imported manually the epel9/epel9-next/baseos/appstream/crb comps in the hyperscale9s-packages-spin-el9s-build build tag and resubmited a scratch build :
hyperscale9s-packages-spin-el9s-build
cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#4f30f9025a02b18585717d6d037e1c78a43da626 ./CentOS-Stream-Hyperscale-Spin.kiwi --kiwi-profile=OpenStack --arch=x86_64 Created task: 3105214 Task info: https://cbs.centos.org/koji/taskinfo?taskID=3105214 Watching tasks (this may be safely interrupted)... 3105214 kiwiBuild (noarch): free 3105214 kiwiBuild (noarch): free -> open (x86-5.cbs.centos.org) 3105223 createKiwiImage (x86_64): free 3105223 createKiwiImage (x86_64): free -> open (x86-5.cbs.centos.org) 3105223 createKiwiImage (x86_64): open (x86-5.cbs.centos.org) -> FAILED: GenericError: Kiwi failed Relevant logs: https://cbs.centos.org/kojifiles/work/tasks/5223/3105223/checkout-x86_64.log https://cbs.centos.org/kojifiles/work/tasks/5223/3105223/mock_output.log https://cbs.centos.org/kojifiles/work/tasks/5223/3105223/hw_info.log https://cbs.centos.org/kojifiles/work/tasks/5223/3105223/build.log https://cbs.centos.org/kojifiles/work/tasks/5223/3105223/state.log https://cbs.centos.org/kojifiles/work/tasks/5223/3105223/root.log 0 free 1 open 0 done 1 failed 3105214 kiwiBuild (noarch): open (x86-5.cbs.centos.org) -> FAILED: GenericError: Kiwi failed Relevant logs: https://cbs.centos.org/kojifiles/work/tasks/5214/3105214/checkout.log 0 free 0 open 0 done 2 failed
By looking at the log it seems that it's because it's referencing pkgs not present in repos/tags :
[ INFO ]: Processing: [########################################] 100% DEBUG util.py:444: [ ERROR ]: 11:17:27 | KiwiInstallPhaseFailed: System package installation failed: Error: Unable to find a match: centos-release-hyperscale centos-release-hyperscale-experimental centos-release-hyperscale-spin DEBUG util.py:446: [ INFO ]: 11:17:27 | Cleaning up SystemPrepare instance DEBUG util.py:598: Child return code was: 1
Oh, because those packages are in the centos extras repo, which we also need in our spin media creation tags. Can that repo also be added?
Added and resubmitted same scratch build (after the regen-repo task) and it worked : https://cbs.centos.org/koji/taskinfo?taskID=3105510
regen-repo
This is awesome, thanks!
Metadata Update from @ngompa: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Looks like it works great!
ngompa@Belldandy-Slimbook ~> cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#4f30f9025a02b18585717d6d037e1c78a43da626 ./CentOS-Stream-Hyperscale-Spin.kiwi --kiwi-profile=OpenStack Created task: 3105781 Task info: https://cbs.centos.org/koji/taskinfo?taskID=3105781 Watching tasks (this may be safely interrupted)... 3105781 kiwiBuild (noarch): free 3105781 kiwiBuild (noarch): free -> open (x86-5.cbs.centos.org) 3105783 createKiwiImage (aarch64): free 3105782 createKiwiImage (x86_64): free 3105782 createKiwiImage (x86_64): free -> open (x86-5.cbs.centos.org) 3105782 createKiwiImage (x86_64): open (x86-5.cbs.centos.org) -> closed 1 free 1 open 1 done 0 failed 3105783 createKiwiImage (aarch64): free -> open (aarch64-01.rdu2.centos.org)
It seems that none of the environment groups from CentOS Stream nor EPEL are present:
The GNOME image failure in the logs is:
DEBUG util.py:444: [ ERROR ]: 14:10:03 | KiwiInstallPhaseFailed: System package installation failed: Module or Group 'workstation-product-environment' is not available.
The KDE image failure in the logs is:
DEBUG util.py:444: [ ERROR ]: 14:17:47 | KiwiInstallPhaseFailed: System package installation failed: Module or Group 'kde-desktop-environment' is not available.
According to my local environment, these both exist:
ngompa@fedora ~> podman run --rm -it quay.io/centoshyperscale/centos:stream9 bash-5.1# dnf grouplist -v Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync DNF version: 4.14.0 cachedir: /var/cache/dnf User-Agent: constructed: 'libdnf (CentOS Stream 9; hyperscale; Linux.x86_64)' repo: downloading from remote: centos-hyperscale countme: event triggered for centos-hyperscale: bucket 2 CentOS Stream 9 - Hyperscale 463 kB/s | 248 kB 00:00 centos-hyperscale: using metadata from Thu Dec 8 00:04:34 2022. repo: downloading from remote: centos-hyperscale-experimental countme: event triggered for centos-hyperscale-experimental: bucket 1 CentOS Stream 9 - Hyperscale Experimental 5.5 MB/s | 2.4 MB 00:00 centos-hyperscale-experimental: using metadata from Wed Dec 7 18:00:01 2022. repo: downloading from remote: centos-hyperscale-spin countme: event triggered for centos-hyperscale-spin: bucket 1 CentOS Stream 9 - Hyperscale Spin 42 kB/s | 36 kB 00:00 centos-hyperscale-spin: using metadata from Wed Apr 6 23:34:08 2022. repo: downloading from remote: baseos countme: no event for baseos: budget to spend: 3 CentOS Stream 9 - BaseOS 13 MB/s | 6.0 MB 00:00 baseos: using metadata from Mon Dec 5 14:09:20 2022. repo: downloading from remote: appstream countme: event triggered for appstream: bucket 2 CentOS Stream 9 - AppStream 10 MB/s | 16 MB 00:01 appstream: using metadata from Mon Dec 5 14:11:11 2022. repo: downloading from remote: crb countme: event triggered for crb: bucket 1 CentOS Stream 9 - CRB 1.4 MB/s | 5.1 MB 00:03 crb: using metadata from Mon Dec 5 14:12:03 2022. repo: downloading from remote: extras-common countme: no event for extras-common: budget to spend: 1 CentOS Stream 9 - Extras packages 20 kB/s | 9.2 kB 00:00 extras-common: using metadata from Fri Nov 25 07:08:56 2022. repo: downloading from remote: epel countme: no event for epel: budget to spend: 1 Extra Packages for Enterprise Linux 9 - x86_64 6.5 MB/s | 12 MB 00:01 epel: using metadata from Sat Dec 10 00:29:42 2022. repo: downloading from remote: epel-next countme: no event for epel-next: budget to spend: 2 Extra Packages for Enterprise Linux 9 - Next - x86_64 1.2 MB/s | 1.4 MB 00:01 epel-next: using metadata from Thu Nov 24 03:28:46 2022. Completion plugin: Generating completion cache... Available Environment Groups: Server with GUI (graphical-server-environment) Server (server-product-environment) Minimal Install (minimal-environment) Workstation (workstation-product-environment) KDE Plasma Workspaces (kde-desktop-environment) Custom Operating System (custom-environment) Virtualization Host (virtualization-host-environment) Available Groups: Legacy UNIX Compatibility (legacy-unix) Console Internet Tools (console-internet) Container Management (container-management) Development Tools (development) .NET Development (dotnet) Graphical Administration Tools (graphical-admin-tools) Headless Management (headless-management) Network Servers (network-server) RPM Development Tools (rpm-development-tools) Scientific Support (scientific) Security Tools (security-tools) Smart Card Support (smart-card) System Tools (system-tools) Fedora Packager (fedora-packager) Xfce (xfce-desktop) bash-5.1# exit
Metadata Update from @ngompa: - Issue status updated to: Open (was: Closed)
The comps.xml were imported with koji import-comps <comps-file.xml> hyperscale9s-spin_media-experimental-el9s-build and if you use cbs taginfo hyperscale9s-spin_media-experimental-el9s-build|grep Groups you'll see groups koji imported.
koji import-comps <comps-file.xml> hyperscale9s-spin_media-experimental-el9s-build
cbs taginfo hyperscale9s-spin_media-experimental-el9s-build|grep Groups
As you can see, there is gnome-desktop or kde-desktop listed as groups there, but what you're referring to (workstation-product-environment and kde-desktop-environment) are in fact environment categories in comps.xml, which seems to be ignored by "koji import-comps" : it seems to import groups but not environments (you can browse the appstream comps.xml to see the difference)
gnome-desktop
kde-desktop
workstation-product-environment
kde-desktop-environment
So maybe you should just reference in your kickstart the groups declared in these environments ? Example for workstation-product-environment from c9s appstream comps :
<id>workstation-product-environment</id> <name>Workstation</name> <description>Workstation is a user-friendly desktop system for laptops and PCs.</description> <display_order>4</display_order> <grouplist> <groupid>base-x</groupid> <groupid>core</groupid> <groupid>fonts</groupid> <groupid>gnome-desktop</groupid> <groupid>guest-desktop-agents</groupid> <groupid>hardware-support</groupid> <groupid>internet-browser</groupid> <groupid>multimedia</groupid> <groupid>networkmanager-submodules</groupid> <groupid>print-client</groupid> <groupid>standard</groupid> <groupid>workstation-product</groupid> </grouplist>
Ugh, alright. I'm going to file a bug about this, because Koji shouldn't ignore them.
ack so nothing to be done at the infra level for now, so let me close this one again
Metadata Update from @arrfab: - Issue close_status updated to: Fixed with Explanation - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.