centos-infra

Clone
Source Code
GIT

#696 Enable kiwi plugin for Hyperscale
Closed: Fixed with Explanation a year ago by arrfab. Opened 2 years ago by ngompa.

Closed: Fixed with Explanation
Reopen Issue

For upcoming Hyperscale work for building images, I need CBS to run Koji 1.28 or newer. Specifically because Koji 1.28 introduces full(ish) support for KIWI:

Kiwi plugin for building images based on XML description files was extended and refactored a bit, so it is now almost production-ready. We expect that in one or two releases we can flip it to first-class plugin.

We'd like to use this in Hyperscale and so we'd like CBS upgraded and the kiwiBuild task enabled for Hyperscale so we can use it.

Metadata Update from @arrfab:
- Issue tagged with: cbs, feature-request
2 years ago

[backlog refinement]
There is another ticket that needs koji to be updated to newer version.

Koji 1.29.0 is out now, so we should update to that directly.

After chatting with Davide and Neal at Devconf, they are ok to maintain and troubleshoot the kiwi plugin in CBS. Since this is included with koji, the Infra team can install the packages on the builders and the hub. The Hyperscale SIG will contribute any necessary config items to the staging branches on https://github.com/centos/ansible-role-kojihub and https://github.com/centos/ansible-role-kojid

We expect to have a CBS staging environment soon so that we can play public contributions for review.

@arrfab, @dcavalca and @ngompa does this match what you expect?

Sounds good to us.

Metadata Update from @arrfab:
- Issue assigned to arrfab
2 years ago

Metadata Update from @arrfab:
- Issue tagged with: high-gain, high-trouble
2 years ago

the plan is to have a cbs.stg where we can test this, and on VMs (no bare-metal so no hypervisor, and I guess it's fine if kiwi doesn't need this).
Rebuilding needed pkgs that will be needed on kojihub and kojid builders is also needed.
Can we come with a list ? I've never played with kiwi so if you can come with an overview and help , we'd be happy to do that with you :)

The packages needed for the kiwiBuild command are:

  • kiwi-cli
  • kiwi-systemdeps

Both packages come from the kiwi source package in EPEL. I recommend sourcing from EPEL rather than rebuilding the dependency chain again.

For some Hyperscale deliverables, we also need btrfs-progs and btrfs usable on the builder (either through the btrfs kmod from the kmods SIG or using the hyperscale kernel)
Edited 2 years ago by ngompa

wrt btrfs-progs, let me paste what was said on irc in #centos-meeting : 

(17:59:51) Eighth_Doctor: oh, and "btrfs-progs" for hyperscale kiwi builds                                                      
(18:00:00) Eighth_Doctor: and the builder host needs to have btrfs available                                                    
(18:00:23) arrfab: so we just need to rebuild these in in the infra tags                                                       
(18:00:31) arrfab: hmm                                                                                                          
(18:00:38) davide: btrfs-progs is not in EPEL, but we do have it in hyperscale if that helps                                    
(18:00:47) arrfab: that was not mentioned (the need for host to be running hyperscale)                                          
(18:01:00) arrfab: as clearly bstinson told me that the goal is to migrate to imagebuilder and kiwi is temporary                
(18:01:16) arrfab: having one plugin is easy, modifying more for a tmp solution would need to be reconsidered then :/           
18:01:26) davide: I don't think it needs to necessarily run our variant -- it just needs to be able to handle btrfs     filesystems                                                                                                                     │
(18:01:38) davide: so e.g. the kmod-btrfs from the kmods sig + btrfs-progrs would do the trick                                  (18:01:40) bstinson: we discussed starting with existing builders                                                               (18:01:50) bstinson: let's do enablement there first                                                                            │
(18:02:05) davide: yeah I think we can take this in two stages                                                                  
(18:02:08) arrfab: bstinson: but if their image need btrfs directly, that will not work ?                                       
(18:02:16) davide: first let's get kiwi working, then we can figure out the btrfs side of things                                
(18:02:25) bstinson: there are non-btrfs variants that can be used                                                              
(18:02:32) arrfab: *ack*

So let's try first with default setup as it will also run in AWS, on CentOS Stream 8 images, so with vanilla kernel

WRT kiwi, I kicked a rebuild in the infra8-buildtools-common tags (inherited through our roles for all koji environments)

Can you also validate that https://docs.pagure.org/koji/plugins/?highlight=kiwi#image-builds-using-kiwi is enough ? not a lot of info but seems that it just needs to be installed without any configuration for the plugin, and then create a kiwi-build group for the build tag and then adding needed rpm pkgs in that group

Yes, that should be sufficient.

Realizing that because it needs to happen within the buildroot itself, we probably need to have it for 8s and 9s ? and should be tagged for all tags (through inheritance)
We don't support (yet) 9s in our infra so if we need that we should start first looking at supporting 9s/el8 in our infra and tags ...

But otoh, if you enable epel{8,9} in your tags, that's possible to then call kiwi-build task assuming that we also ensure that pkgs are added with add-group-pkg in thekiwi-build group in your tag. Something to test ?

@ngompa : it seems it's not possible to have deps in epel8 anyway : 

dnf install kiwi-cli kiwi-systemdeps
Last metadata expiration check: 0:00:14 ago on Thu 01 Sep 2022 08:36:08 AM UTC.
Error: 
 Problem: package kiwi-cli-9.24.44-1.el8.noarch requires python3-kiwi = 9.24.44-1.el8, but none of the providers can be installed
  - conflicting requests
  - nothing provides python3.6dist(pyxattr) needed by python3-kiwi-9.24.44-1.el8.noarch
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Would you mind working on missing deps there and then we can revisit ?

Metadata Update from @arrfab:
- Issue priority set to: Waiting on Reporter (was: Needs Review)
2 years ago

Metadata Update from @arrfab:
- Issue tagged with: blocked
2 years ago

@arrfab That's in PowerTools for c8s.

oh ! completely forgot about that (disabled by default) ... thanks ! .. something to verify for our build tags but iirc it's enabled everywhere.

hey, I just closed #907 so we have now publicly available https://cbs.stg.centos.org on which we can play with new features and have community members interact with it.
I'll so start working on some kojid/kojihub role[s] change[s] to test that it's doable to have kiwi plugin working.
Once validated we'll just be able to replay on cbs.centos.org

Just for my own tests, can you point me to some profiles that we can use for kiwi ? Let's start with something basic/straight forward

There should be some simple stuff here: https://pagure.io/centos-kiwi-examples

There should be some simple stuff here: https://pagure.io/centos-kiwi-examples

I meant for koji and kiwi-build task :-)
It seems koji kiwi-build is expecting a profile or scm (so is that expecting path to .xml ?)

koji kiwi-build --help
Usage: koji kiwi-build [options] <target> <description_scm> <description_path>
(Specify the --help global option for a list of other help options)

Options:
  -h, --help            show this help message and exit
  --scratch             Perform a scratch build
  --release=RELEASE     Release of the output image
  --repo=REPO           Specify a repo that will override the repo used to
                        install RPMs in the image. May be used multiple times.
                        The build tag repo associated with the target is the
                        default.
  --noprogress          Do not display progress of the upload
  --kiwi-profile=KIWI_PROFILE
                        Select profile from description file
  --make-prep           Run 'make prep' in checkout before starting the build
  --can-fail=ARCH1,ARCH2,...
                        List of archs which are not blocking for build
                        (separated by commas.
  --arch=ARCHES         Limit arches to this subset
  --nowait              
  --wait                Wait on the image creation, even if running in the
                        background

path is to the directory containing config.xml, I believe. Try the complex-build one with OpenStack?
Edited 2 years ago by ngompa

@ngompa trying to look at this but can you come with some trees/options that we can test with koji ? (not your standalone wrapper)
also, worth knowing that we can't use metalink= for repositories as kojid builders don't have access to internet and also we have to point to valid but static repositories (like mirror.stream.centos.org which is then available)

@ngompa trying to look at this but can you come with some trees/options that we can test with koji ? (not your standalone wrapper)
also, worth knowing that we can't use metalink= for repositories as kojid builders don't have access to internet and also we have to point to valid but static repositories (like mirror.stream.centos.org which is then available)

Koji is supposed to rewrite the repos automatically. Is it not doing that?

there is no koji doc about how to use it btw :-)

Just a little bit of progress :

I created a git repo to have some modified/simplified .xml for kiwi (based on yours) and not using metalink (as it will not work in our infra) and then kicked : 

cbs-stg kiwi-build --scratch hyperscale9s-packages-spin-el9s git+https://git.centos.org/centos/infra-playground#95c4d5d2e15b35255ccf213fec4d06f92d8e8ffc kiwi/simple-build/
Created task: 123
Task info: https://cbs.stg.centos.org/koji/taskinfo?taskID=123
Watching tasks (this may be safely interrupted)...
123 kiwiBuild (noarch): free
123 kiwiBuild (noarch): free -> open (kojid1.stg.centos.org)
  124 createKiwiImage (x86_64): free
  124 createKiwiImage (x86_64): free -> open (kojid1.stg.centos.org)
  124 createKiwiImage (x86_64): open (kojid1.stg.centos.org) -> closed
  0 free  1 open  1 done  0 failed
123 kiwiBuild (noarch): open (kojid1.stg.centos.org) -> closed
  0 free  0 open  2 done  0 failed

123 kiwiBuild (noarch) completed successfully

=> https://cbs.stg.centos.org/koji/taskinfo?taskID=124

I tried to just build a cloud image but had to merge all .xml files into one otherwise koji kiwi plugin fails with : 

Sep 13 07:14:49 kojid1.stg.euw1.centos.org kojid[267560]:     with open(file, \'rb\') as fp:
Sep 13 07:14:49 kojid1.stg.euw1.centos.org kojid[267560]: FileNotFoundError: [Errno 2] No such file or directory: \'this://./repositories/core.xml\'
Sep 13 07:14:49 kojid1.stg.euw1.centos.org kojid[267560]: '>

I tried once again but it will be probably my last test (you're now welcome to test it yourself on cbs.stg with a TLS cert from accounts.stg.centos.org and pointing to cbs.stg.centos.org for koji profile)

it fails again but different issue (from root.log ): 

DEBUG util.py:446:  [ INFO    ]: 07:26:54 | Processing SELinux file security contexts
DEBUG util.py:446:  [ WARNING ]: 07:26:54 | Could not parse setfiles output
DEBUG util.py:444:  [ ERROR   ]: 07:27:05 | KiwiCommandError: chroot: stderr: setfiles: Could not set context for /run/motd:  Invalid argument
DEBUG util.py:444:  setfiles: Could not set context for /run/motd.d:  Invalid argument
DEBUG util.py:444:  setfiles: Could not set context for /var/lib/systemd/coredump:  Invalid argument
DEBUG util.py:444:  setfiles: Could not set context for /usr/lib/systemd/user/dbus-broker.service:  Invalid argument
DEBUG util.py:444:  setfiles: Could not set context for /usr/lib/systemd/systemd-network-generator:  Invalid argument
DEBUG util.py:444:  setfiles: Could not set context for /usr/bin/rpmdb:  Invalid argument
DEBUG util.py:444:  , stdout: 
1k
2k
3k
4k
5k
6k
7k
8k
9k
10k
11k
12k
13k
14k
15k
16k
17k
18k
19k
20k
21k
22k
23k
24k
25k
26k
27k
28k
29k
/ 100.0%
DEBUG util.py:446:  [ INFO    ]: 07:27:05 | Cleaning up SystemPrepare instance
DEBUG util.py:598:  Child return code was: 1

Worth knowing that host is a centos stream 8 VM (we don't support yet stream 9 in centos infra) and it seems it doens't allow kojid kiwi plugin to set correct selinux context either :

From audit.log : 

type=AVC msg=audit(1663054015.307:54713): avc:  denied  { mac_admin } for  pid=279225 comm="setfiles" capability=33  scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0
type=AVC msg=audit(1663054015.308:54714): avc:  denied  { mac_admin } for  pid=279225 comm="setfiles" capability=33  scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0
type=AVC msg=audit(1663054015.813:54715): avc:  denied  { mac_admin } for  pid=279225 comm="setfiles" capability=33  scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0
type=AVC msg=audit(1663054016.057:54717): avc:  denied  { mac_admin } for  pid=279225 comm="setfiles" capability=33  scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0
type=AVC msg=audit(1663054016.169:54718): avc:  denied  { mac_admin } for  pid=279225 comm="setfiles" capability=33  scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0
type=AVC msg=audit(1663054019.551:54719): avc:  denied  { mac_admin } for  pid=279225 comm="setfiles" capability=33  scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0

We run with selinux in enforcing mode everywhere and it should stay like that, so can you verify with kiwi and koji about how to support that ? we can then revisit later if we want to enable it on centos cbs builders ?

Edited 2 years ago by arrfab

I just tried with setenforce 0 on that isolated builder (and temporary for that test) and it then works :

https://cbs.stg.centos.org/koji/taskinfo?taskID=138

So, apart from the selinux issue (already trying to see how to rebuild a custom .pp eventually), I gave the same build another try (no difference) but it now fails : https://cbs.stg.centos.org/kojifiles/work/tasks/142/142/root.log

DEBUG util.py:446:  [ INFO    ]: 08:28:55 | Creating volume group systemVG
DEBUG util.py:444:  [ ERROR   ]: 08:28:55 | KiwiCommandError: vgcreate: stderr:   /dev/systemVG: already exists in filesystem
DEBUG util.py:444:    Run `vgcreate --help' for more information.
DEBUG util.py:444:  , stdout: (no output on stdout)

From what I can see it initially created /dev/loop devices but it seems it doesn't clean these after builds, so new job would complain about some existing VG/LV on it.
Something to report to upstream koji ?

cc: @tkopecek

Interesting, I'll try it locally. I'm running it definitely with enforcing selinux. dev leak seems to me like a kiwi issue, but will check it.

Finally found why : it's just that kojid by default doesn't clean-up directly the buildroots, but if you wait 2 minutes (default for buildroot_basic_cleanup_delay=120) it then just works fine.

For the selinux policy, I also built a custom policy and it seems to work : 

cat /root/selinux/centos-kojid-kiwi.te ; echo ====== ; echo "selinux status is : $(getenforce)" 

module centos-kojid-kiwi 1.0;

require {
    type unconfined_service_t;
    class capability2 mac_admin;
}

#============= unconfined_service_t ==============
allow unconfined_service_t self:capability2 mac_admin;
======
selinux status is : Enforcing

I'll modify our ansible kojid role to distribute (and load) that .pp file (I don't like it but better than whole setenforce 0 afaics)

I submitted some scratch builds and they all worked : https://cbs.stg.centos.org/koji/index

@tkopecek I think the Koji kiwi config flattener doesn't recognize this:// as a protocol yet. It's basically a relative version of file:// so you can translate this:// to file://${PWD} and resolve the URLs.

Metadata Update from @arrfab:
- Issue untagged with: blocked
2 years ago

@tkopecek It also seems that based on the output from this build log that it's not using the kiwi description koji generates: https://cbs.stg.centos.org/koji/taskinfo?taskID=160

We should probably be renaming any existing config.xml or *.kiwi files out of the way so that the generated one is used.

Maybe better to use directly --kiwi-file instead of --description.

When using --kiwi-file, you still need --description. But yeah, it may make sense to set that.

@ngompa happy to just deploy kiwi plugin on cbs.centos.org (we'll have one host in the image channel) as it seems to work at this stage. Thoughts ?

I pushed that change and it's now enabled on cbs (kojihub and kojid) but nothing is configured for tags (as we need to add the kiwi-build group on the build tags ..
Waiting for feedback on which one you'd like to see it enabled

Let's configure the kiwi-build group to the hyperscale8s-spin_media-experimental-el8s-build and hyperscale8s-spin_media-main-el8s-build tags.

I also noticed that the hyperscale8s-spin_media-experimental-el8s-build tag doesn't have hyperscale8s-packages-experimental-release in its inheritance and it should be added.

The kiwi-build group needs the following: kiwi-cli, kiwi-systemdeps, distribution-gpg-keys, btrfs-progs.

(Having the btrfs-progs userspace in the group will prepare us for later when we have Btrfs-enabled CBS builders)

We also need versions of the hyperscale8s-spin_media-* tags for c9s.

The formula we want here is the same as for c8s: hyperscale9s-spin_media-{main,experimental}-{el9s-build,candidate,testing,release}.

The hyperscale9s-spin_media-*-el9s-build tags should be configured the same way the hyperscale8s-spin_media-*-el8s-build tags were.

Let's try first with the 8s ones : 

koji add-group hyperscale8s-spin_media-experimental-el8s-build kiwi-build
koji add-group-pkg hyperscale8s-spin_media-experimental-el8s-build kiwi-build kiwi-cli kiwi-systemdeps distribution-gpg-keys btrfs-progs

koji add-group hyperscale8s-spin_media-main-el8s-build kiwi-build
koji add-group-pkg hyperscale8s-spin_media-main-el8s-build kiwi-build kiwi-cli kiwi-systemdeps distribution-gpg-keys btrfs-progs

Also added the inheritance on hyperscale8s-spin_media-experimental-el8s-build : 

cbs taginfo hyperscale8s-spin_media-experimental-el8s-build
Tag: hyperscale8s-spin_media-experimental-el8s-build [2341]
Arches: x86_64 aarch64
Groups: build, livecd-build, livemedia-build, srpm-build
Tag options:
  mock.package_manager : 'dnf'
  mock.yum.module_hotfixes : 1
  rpm.macro.vendor : 'CentOS Hyperscale SIG'
This tag is a buildroot for one or more targets
Current repo: repo#964125: 2022-09-22 03:47:51.050014+00:00
Targets that build from this tag:
  hyperscale8s-spin_media-experimental-el8s
External repos:
    5 centos8s-cr (http://mirror.centos.org/centos/8-stream//cr/$arch/os/, merge mode: bare), arches: inherited from tag
   10 centos8s-extras (http://mirror.centos.org/centos/8-stream//extras/$arch/os/, merge mode: bare), arches: inherited from tag
   15 centos8s-powertools (http://mirror.centos.org/centos/8-stream//PowerTools/$arch/os/, merge mode: bare), arches: inherited from tag
   20 centos8s-appstream (http://mirror.centos.org/centos/8-stream//AppStream/$arch/os/, merge mode: bare), arches: inherited from tag
   25 centos8s-baseos (http://mirror.centos.org/centos/8-stream//BaseOS/$arch/os/, merge mode: bare), arches: inherited from tag
   30 epel8 (https://cbs.centos.org/kojifiles/repos/epel/8/Everything/$arch/, merge mode: bare), arches: inherited from tag
   35 epel8-next (https://cbs.centos.org/kojifiles/repos/epel/next/8/Everything/$arch/, merge mode: bare), arches: inherited from tag
Inheritance:
  0    .... hyperscale8s-packages-main-release [2249]
  5    .... buildsys8s-release [1866]
  10   .... hyperscale8s-spin_media-experimental-candidate [2338]
  20   .... hyperscale8s-packages-hotfixes-release [2305]
  25   .... hyperscale8s-packages-spin-release [2301]
  30   .... hyperscale8s-packages-experimental-release [2245]

Can you give it now a try ? worth knowing that it should stay in -candidate for now as current signing+push releng process doesn't know anything about non RPM repositories (we recently added a function for DuD .iso artifacts for kmods SIG as one example) but that can be then another ticket

Per discussion with @ngompa on irc, I built https://src.fedoraproject.org/rpms/koji/c/c15b855e4e075e2d5ffa586c9ed7fc870796393f?branch=rawhide , which is koji 1.30.0-2 , which has the backported patches/fixes for kiwi (merged in upstream koji)
It's now deployed on CBS : https://cbs.centos.org/koji/api

Per discussion with @ngompa on irc (follow-up) I already also created the CentOS Stream 9 tags : 

cbs taginfo hyperscale9s-spin_media-main-el9s-build
Tag: hyperscale9s-spin_media-main-el9s-build [2661]
Arches: x86_64 aarch64
Groups: build, kiwi-build, srpm-build
Tag options:
  mock.new_chroot : 0
  mock.package_manager : 'dnf'
  mock.yum.module_hotfixes : 1
  rpm.macro.vendor : 'CentOS Hyperscale SIG'
This tag is a buildroot for one or more targets
Current repo: repo#965929: 2022-09-27 07:19:42.905150+00:00
Targets that build from this tag:
  hyperscale9s-spin_media-main-el9s
External repos:
    5 centos9s-baseos (http://mirror.stream.centos.org/9-stream/BaseOS/$arch/os/, merge mode: bare), arches: inherited from tag
   10 centos9s-appstream (http://mirror.stream.centos.org/9-stream/AppStream/$arch/os/, merge mode: bare), arches: inherited from tag
   15 centos9s-crb (http://mirror.stream.centos.org/9-stream/CRB/$arch/os/, merge mode: bare), arches: inherited from tag
   20 epel9 (https://cbs.centos.org/kojifiles/repos/epel/9/Everything/$arch/, merge mode: bare), arches: inherited from tag
   25 epel9-next (https://cbs.centos.org/kojifiles/repos/epel/next/9/Everything/$arch/, merge mode: bare), arches: inherited from tag
Inheritance:
  0    .... hyperscale9s-packages-main-release [2378]
  5    .... buildsys9s-release [2363]
  10   .... hyperscale9s-spin_media-main-candidate [2658]
  15   .... hyperscale9s-packages-hotfixes-release [2414]
  20   .... hyperscale9s-packages-spin-release [2382]


cbs taginfo hyperscale9s-spin_media-experimental-el9s-build
Tag: hyperscale9s-spin_media-experimental-el9s-build [2665]
Arches: x86_64 aarch64
Groups: build, kiwi-build, srpm-build
Tag options:
  mock.new_chroot : 0
  mock.package_manager : 'dnf'
  mock.yum.module_hotfixes : 1
  rpm.macro.vendor : 'CentOS Hyperscale SIG'
This tag is a buildroot for one or more targets
Current repo: repo#965930: 2022-09-27 07:21:13.216410+00:00
Targets that build from this tag:
  hyperscale9s-spin_media-experimental-el9s
External repos:
    5 centos9s-baseos (http://mirror.stream.centos.org/9-stream/BaseOS/$arch/os/, merge mode: bare), arches: inherited from tag
   10 centos9s-appstream (http://mirror.stream.centos.org/9-stream/AppStream/$arch/os/, merge mode: bare), arches: inherited from tag
   15 centos9s-crb (http://mirror.stream.centos.org/9-stream/CRB/$arch/os/, merge mode: bare), arches: inherited from tag
   20 epel9 (https://cbs.centos.org/kojifiles/repos/epel/9/Everything/$arch/, merge mode: bare), arches: inherited from tag
   25 epel9-next (https://cbs.centos.org/kojifiles/repos/epel/next/9/Everything/$arch/, merge mode: bare), arches: inherited from tag
Inheritance:
  0    .... hyperscale9s-packages-main-release [2378]
  5    .... buildsys9s-release [2363]
  10   .... hyperscale9s-spin_media-experimental-candidate [2662]
  15   .... hyperscale9s-packages-hotfixes-release [2414]
  20   .... hyperscale9s-packages-spin-release [2382]
  25   .... hyperscale9s-packages-experimental-release [2410]

Can you verify and validate please ?

@tkopecek So I tried to run a build and Koji seems to choke on it? https://cbs.centos.org/koji/taskinfo?taskID=3032622

The command I ran was:

$ cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#f7475459b17b75adaf3b048bf4ec166830da3218 ./ --kiwi-profile=OpenStack --arch=x86_64

Hmm, that patch was improved a lot during testing. Isn't it the original version? + Now you have to specify path with the xml file ./ -> ./xyz.kiwi to not allow kiwi to choose something else.

 
$ cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#f7475459b17b75adaf3b048bf4ec166830da3218 ./CentOS-Stream-Hyperscale-Spin.kiwi --kiwi-profile=OpenStack --arch=x86_64
Created task: 3032636
Task info: https://cbs.centos.org/koji/taskinfo?taskID=3032636
Watching tasks (this may be safely interrupted)...
3032636 kiwiBuild (noarch): free
3032636 kiwiBuild (noarch): free -> open (x86-5.cbs.centos.org)
3032636 kiwiBuild (noarch): open (x86-5.cbs.centos.org) -> FAILED: GenericError: Repo must contain only one .kiwi file.
  Relevant logs:
  http://cbs.centos.org/kojifiles/work/tasks/2636/3032636/checkout.log
  0 free  0 open  0 done  1 failed

3032636 kiwiBuild (noarch) failed

Yes - it is definitely the old code - This error message is not there anymore. https://pagure.io/koji/pull-request/3498#request_diff

@arrfab, did you deploy koji 1.30.0-2 for CBS?

yes, and built from your src.rpm : https://cbs.centos.org/koji/buildinfo?buildID=41314 (so with your backported patches, not verified)

@ngompa check the 3498 patch - testing found a bunch of problems there.

Submitted PR to update patches in the package: https://src.fedoraproject.org/rpms/koji/pull-request/14

Revisiting ticket without feedback and realizing that your PR was merged.
Submitting the build at the cbs side and we can deploy on stg and update cbs.centos.org.

@ngompa : built and tested yesterday update on https://cbs.stg.centos.org and working so promoted the build to -release and bumped version to 1.30.0-3 in ansible inventory.
https://cbs.centos.org is now running that version, both on hub and kojid level.

Can you test and let me know if that works for you ?

@tkopecek New interesting error: https://cbs.centos.org/koji/taskinfo?taskID=3048455

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/koji/daemon.py", line 1468, in runTask
    response = (handler.run(),)
  File "/usr/lib/python3.6/site-packages/koji/tasks.py", line 335, in run
    return koji.util.call_with_argcheck(self.handler, self.params, self.opts)
  File "/usr/lib/python3.6/site-packages/koji/util.py", line 271, in call_with_argcheck
    return func(*args, **kwargs)
  File "/usr/lib/koji-builder-plugins/kiwi.py", line 362, in handler
    desc, types = self.prepareDescription(path, name, version, repos, arch)
  File "/usr/lib/koji-builder-plugins/kiwi.py", line 205, in prepareDescription
    inc = xml.dom.minidom.parse(path)  # nosec
  File "/usr/lib64/python3.6/xml/dom/minidom.py", line 1958, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python3.6/xml/dom/expatbuilder.py", line 910, in parse
    with open(file, 'rb') as fp:
NotADirectoryError: [Errno 20] Not a directory: '/var/lib/mock/hyperscale9s-spin_media-experimental-el9s-build-159345-971833/root/chroot_tmpdir/kiwi-descriptions/./CentOS-Stream-Hyperscale-Spin.kiwi/repositories/core.xml'

Command used:

$> cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#f7475459b17b75adaf3b048bf4ec166830da3218 ./CentOS-Stream-Hyperscale-Spin.kiwi --kiwi-profile=OpenStack --arch=x86_64
Edited 2 years ago by ngompa

Let us know when it will be all working and only then we'll consider updating again (cbs.stg.centos.org should be used first to validate this works before we update cbs.centos.org again)

I'm waiting on this PR getting merged: https://src.fedoraproject.org/rpms/koji/pull-request/15

Looking at this and it was merged .
Rebuilt on cbs for infra tags : https://cbs.centos.org/koji/buildinfo?buildID=41715
Modified ansible to deploy/upgrade on STG : https://cbs.stg.centos.org/koji/api

As it seems trivial, let me schedule the cbs.centos.org upgrade but ideally we'd validate that it works on STG through some scratch builds

Promoted to release tag and also applied by ansible on cbs : https://cbs.centos.org/koji/api
@ngompa : let me know if that works for you and if it does we'll (finally :-) ) be able to close this ticket.
We'll just have to create another one for the releng part and decide on tags and how to automatically promote content to mirror network (as it can't be actual process for rpms)

Uhh...

$> cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#f7475459b17b75adaf3b048bf4ec166830da3218 ./CentOS-Stream-Hyperscale-Spin.kiwi --kiwi-profile=OpenStack --arch=x86_64
2022-10-27 08:12:59,584 [ERROR] koji: ParameterError: Invalid type for value '['OpenStack']': <class 'list'>, expected type <class 'str'>

@tkopecek, this seems goofy?

@ngompa so, my suggestion : I'll let you work with @tkopecek until you have something that works. Then we can give it a try on .stg. and only then consider deploying on cbs.centos.org.

So just update this ticket when you have something that can be tested and that is also tested upstream (koji side)

@arrfab koji 1.31 should have everything we need to do this properly now: https://koji.fedoraproject.org/koji/buildinfo?buildID=2091321

Can you get this deployed so we can try it?

@ngompa rebuilt : https://cbs.centos.org/koji/buildinfo?buildID=42022 and tagged to -testing so I'll give it a quick deploy on cbs.stg. Ideally I would have asked to test an image build on cbs.stg env but I guess all your complex examples would require building from hyperscale tags, which don't exist on cbs.stg.
So let me just do a quick deploy/check and then I'll update cbs.centos.org too (by looking at koji Release Notes it shouldn't be introducing big changes)

@ngompa , @tkopecek : tested an update and Release Notes are mentioning the traditional schema update for postgres, but it's not in that rpm package , but it exists in upstream repo

Would you mind having that fixed and have a new rpm ?

@ngompa, @tkopecek : just gave it a quick try and the error Neal reported is still the same : 

cbs-stg kiwi-build --scratch hyperscale9s-packages-spin-el9s git+https://git.centos.org/centos/infra-playground#f1a4a7c0b9c8cec74e836f0050b4208205565078 kiwi/complex-build/centos-test.kiwi --kiwi-profile=Openstack
2022-11-22 08:34:52,905 [ERROR] koji: ParameterError: Invalid type for value '['Openstack']': <class 'list'>, expected type <class 'str'>

@arrfab It is there: https://cbs.centos.org/koji/fileinfo?rpmID=411028&filename=/usr/share/doc/koji/docs/schema-update-1.30-1.31.sql

And build fails me on something else:

INFO: Running in chroot: ['kiwi-ng', '--profile', 'Openstack', '--kiwi-file', 'centos-test.x86_64.kiwi', 'system', 'build', '--description', 'infra-playground/kiwi/complex-build', '--target-dir', '/builddir/result/image']
Start: chroot ['kiwi-ng', '--profile', 'Openstack', '--kiwi-file', 'centos-test.x86_64.kiwi', 'system', 'build', '--description', 'infra-playground/kiwi/complex-build', '--target-dir', '/builddir/result/image']
[ INFO    ]: 11:54:51 | Reading runtime config file: '/etc/kiwi.yml'
[ INFO    ]: 11:54:51 | Loading XML description
[ INFO    ]: 11:54:51 | Support for XML markup available
[ ERROR   ]: 11:54:51 | KiwiProfileNotFound: profile Openstack not found for host arch x86_64
Finish: chroot ['kiwi-ng', '--profile', 'Openstack', '--kiwi-file', 'centos-test.x86_64.kiwi', 'system', 'build', '--description', 'infra-playground/kiwi/complex-build', '--target-dir', '/builddir/result/image']

Profile names are case sensitive, I believe. It's OpenStack, not Openstack.

I know and I saw it and forgot to update my comment here as I obviously tried then with correct name : 

cbs-stg kiwi-build --scratch hyperscale9s-packages-spin-el9s git+https://git.centos.org/centos/infra-playground#f1a4a7c0b9c8cec74e836f0050b4208205565078 kiwi/complex-build/centos-test.kiwi --kiwi-profile=OpenStack
2022-11-22 12:04:32,062 [ERROR] koji: ParameterError: Invalid type for value '['OpenStack']': <class 'list'>, expected type <class 'str'>

Did you update the CLI locally? Once I did, it seems to have started:

ngompa@Belldandy-Slimbook ~> cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#4f30f9025a02b18585717d6d037e1c78a43da626 ./CentOS-Stream-Hyperscale-Spin.kiwi --kiwi-profile=OpenStack --arch=x86_64
Created task: 3100877
Task info: https://cbs.centos.org/koji/taskinfo?taskID=3100877
Watching tasks (this may be safely interrupted)...
3100877 kiwiBuild (noarch): free
3100877 kiwiBuild (noarch): free -> open (x86-5.cbs.centos.org)
  3100878 createKiwiImage (x86_64): free
  3100878 createKiwiImage (x86_64): free -> open (x86-5.cbs.centos.org)

@arrfab It is there: https://cbs.centos.org/koji/fileinfo?rpmID=411028&filename=/usr/share/doc/koji/docs/schema-update-1.30-1.31.sql

And build fails me on something else:
INFO: Running in chroot: ['kiwi-ng', '--profile', 'Openstack', '--kiwi-file', 'centos-test.x86_64.kiwi', 'system', 'build', '--description', 'infra-playground/kiwi/complex-build', '--target-dir', '/builddir/result/image'] Start: chroot ['kiwi-ng', '--profile', 'Openstack', '--kiwi-file', 'centos-test.x86_64.kiwi', 'system', 'build', '--description', 'infra-playground/kiwi/complex-build', '--target-dir', '/builddir/result/image'] [ INFO ]: 11:54:51 | Reading runtime config file: '/etc/kiwi.yml' [ INFO ]: 11:54:51 | Loading XML description [ INFO ]: 11:54:51 | Support for XML markup available [ ERROR ]: 11:54:51 | KiwiProfileNotFound: profile Openstack not found for host arch x86_64 Finish: chroot ['kiwi-ng', '--profile', 'Openstack', '--kiwi-file', 'centos-test.x86_64.kiwi', 'system', 'build', '--description', 'infra-playground/kiwi/complex-build', '--target-dir', '/builddir/result/image']

When trying to update the schema : 

psql koji < /usr/share/doc/koji/docs/schema-update-1.30-1.31.sql
BEGIN
ERROR:  syntax error at or near "INCLUDE"
LINE 1: ...sion || '-' || release || '.' || arch || '.rpm')) INCLUDE (i...
                                                             ^
ROLLBACK

@tkopecek ^

Did you update the CLI locally? Once I did, it seems to have started:

ngompa@Belldandy-Slimbook ~> cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#4f30f9025a02b18585717d6d037e1c78a43da626 ./CentOS-Stream-Hyperscale-Spin.kiwi --kiwi-profile=OpenStack --arch=x86_64 Created task: 3100877 Task info: https://cbs.centos.org/koji/taskinfo?taskID=3100877 Watching tasks (this may be safely interrupted)... 3100877 kiwiBuild (noarch): free 3100877 kiwiBuild (noarch): free -> open (x86-5.cbs.centos.org) 3100878 createKiwiImage (x86_64): free 3100878 createKiwiImage (x86_64): free -> open (x86-5.cbs.centos.org)

It's true that I haven't locally, as it's not available (yet) in epel9 .. let me try on the cbs.stg setup

Did you update the CLI locally? Once I did, it seems to have started:

ngompa@Belldandy-Slimbook ~> cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#4f30f9025a02b18585717d6d037e1c78a43da626 ./CentOS-Stream-Hyperscale-Spin.kiwi --kiwi-profile=OpenStack --arch=x86_64 Created task: 3100877 Task info: https://cbs.centos.org/koji/taskinfo?taskID=3100877 Watching tasks (this may be safely interrupted)... 3100877 kiwiBuild (noarch): free 3100877 kiwiBuild (noarch): free -> open (x86-5.cbs.centos.org) 3100878 createKiwiImage (x86_64): free 3100878 createKiwiImage (x86_64): free -> open (x86-5.cbs.centos.org)

It's true that I haven't locally, as it's not available (yet) in epel9 .. let me try on the cbs.stg setup

I enabled updates-testing temporarily to upgrade the koji client.

We're now failing on something that isn't from KIWI, but from the Koji repos?

DEBUG util.py:444: [ ERROR ]: 11:08:34 | KiwiInstallPhaseFailed: System package installation failed: Module or Group 'core' is not available.

Are the comps group metadata not available through the external repos merged into the build repo?

@ngompa : per koji documentation, and @tkopecek can confirm, it seems that koji will just replace all references to repo with the ones from koji so internal. But I don't know how koji is importing comps.xml and groups into koji

Also, koji is still not updated on cbs.centos.org as long as we can't have it properly working on cbs.stg.centos.org
Edited a year ago by arrfab

upstream koji issue for the schema update : https://pagure.io/koji/issue/3594
  • INCLUDE part - it is in postgres from version 11, not sure what is running in cbs. Feel free to ignore this update or drop include part of that (created https://pagure.io/koji/issue/3595)
  • comps are always overriden by koji's groups, so external comps are lost during the process. Anyway, easy way to import needed ones is via 'koji import-comps' command (to buildtag).
  • comps are always overriden by koji's groups, so external comps are lost during the process. Anyway, easy way to import needed ones is via 'koji import-comps' command (to buildtag).

Importing the CentOS and EPEL comps groups into the build tag should fix the problem I'm having now, since DNF is failing on resolving comps groups.

It's another step that we'll have to take into accounts but we can automate that part when a SIG will ask for some tags and kiwi support.
FWIW, I kicked a scratch build on cbs.stg env and it worked then : 

koji kiwi-build --scratch hyperscale9s-packages-spin-el9s git+https://git.centos.org/centos/infra-playground#f1a4a7c0b9c8cec74e836f0050b4208205565078 kiwi/complex-build/centos-test.kiwi --kiwi-profile=OpenStack
Created task: 339
Task info: https://cbs.stg.centos.org/koji/taskinfo?taskID=339
Watching tasks (this may be safely interrupted)...
339 kiwiBuild (noarch): free
339 kiwiBuild (noarch): free -> open (kojid1.stg.centos.org)
  340 createKiwiImage (x86_64): free
  340 createKiwiImage (x86_64): free -> open (kojid1.stg.centos.org)
  340 createKiwiImage (x86_64): open (kojid1.stg.centos.org) -> closed
  0 free  1 open  1 done  0 failed
339 kiwiBuild (noarch): open (kojid1.stg.centos.org) -> closed
  0 free  0 open  2 done  0 failed

339 kiwiBuild (noarch) completed successfully

So eventually I can consider upgrading now the cbs.centos.org env with that version too, and document the .sql issue for koji upgrade (mentioned above)

@tkopecek Where's the $target_dir/build/image-root.log file? I thought we were tailing/archiving that log file per architecture? I don't see it in https://cbs.stg.centos.org/koji/taskinfo?taskID=339
Edited a year ago by ngompa

We're pretty much set aside from missing that log file, so we should be good for production use, though I'd like that log file captured asap.

@ngompa : cbs.centos.org updated to 1.31 (including builders) today : https://cbs.centos.org/koji/api

For the comps to import, I realized that our internal mirror for epel doens't provide/import comps but that's something I can have a look at later
In the meantime, I just imported manually the epel9/epel9-next/baseos/appstream/crb comps in the hyperscale9s-packages-spin-el9s-build build tag and resubmited a scratch build : 

cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#4f30f9025a02b18585717d6d037e1c78a43da626 ./CentOS-Stream-Hyperscale-Spin.kiwi --kiwi-profile=OpenStack --arch=x86_64
Created task: 3105214
Task info: https://cbs.centos.org/koji/taskinfo?taskID=3105214
Watching tasks (this may be safely interrupted)...
3105214 kiwiBuild (noarch): free
3105214 kiwiBuild (noarch): free -> open (x86-5.cbs.centos.org)
  3105223 createKiwiImage (x86_64): free
  3105223 createKiwiImage (x86_64): free -> open (x86-5.cbs.centos.org)
  3105223 createKiwiImage (x86_64): open (x86-5.cbs.centos.org) -> FAILED: GenericError: Kiwi failed
  Relevant logs:
  https://cbs.centos.org/kojifiles/work/tasks/5223/3105223/checkout-x86_64.log
  https://cbs.centos.org/kojifiles/work/tasks/5223/3105223/mock_output.log
  https://cbs.centos.org/kojifiles/work/tasks/5223/3105223/hw_info.log
  https://cbs.centos.org/kojifiles/work/tasks/5223/3105223/build.log
  https://cbs.centos.org/kojifiles/work/tasks/5223/3105223/state.log
  https://cbs.centos.org/kojifiles/work/tasks/5223/3105223/root.log
  0 free  1 open  0 done  1 failed
3105214 kiwiBuild (noarch): open (x86-5.cbs.centos.org) -> FAILED: GenericError: Kiwi failed
  Relevant logs:
  https://cbs.centos.org/kojifiles/work/tasks/5214/3105214/checkout.log
  0 free  0 open  0 done  2 failed

By looking at the log it seems that it's because it's referencing pkgs not present in repos/tags : 

[ INFO    ]: Processing: [########################################] 100%
DEBUG util.py:444:  [ ERROR   ]: 11:17:27 | KiwiInstallPhaseFailed: System package installation failed: Error: Unable to find a match: centos-release-hyperscale centos-release-hyperscale-experimental centos-release-hyperscale-spin
DEBUG util.py:446:  [ INFO    ]: 11:17:27 | Cleaning up SystemPrepare instance
DEBUG util.py:598:  Child return code was: 1

Oh, because those packages are in the centos extras repo, which we also need in our spin media creation tags. Can that repo also be added?
Edited a year ago by ngompa

Added and resubmitted same scratch build (after the regen-repo task) and it worked : https://cbs.centos.org/koji/taskinfo?taskID=3105510

This is awesome, thanks!

Metadata Update from @ngompa:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
a year ago

Looks like it works great!

ngompa@Belldandy-Slimbook ~> cbs kiwi-build --scratch hyperscale9s-spin_media-experimental-el9s git+https://gitlab.com/CentOS/Hyperscale/releng/kiwi-descriptions.git#4f30f9025a02b18585717d6d037e1c78a43da626 ./CentOS-Stream-Hyperscale-Spin.kiwi --kiwi-profile=OpenStack
Created task: 3105781
Task info: https://cbs.centos.org/koji/taskinfo?taskID=3105781
Watching tasks (this may be safely interrupted)...
3105781 kiwiBuild (noarch): free
3105781 kiwiBuild (noarch): free -> open (x86-5.cbs.centos.org)
  3105783 createKiwiImage (aarch64): free
  3105782 createKiwiImage (x86_64): free
  3105782 createKiwiImage (x86_64): free -> open (x86-5.cbs.centos.org)
  3105782 createKiwiImage (x86_64): open (x86-5.cbs.centos.org) -> closed
  1 free  1 open  1 done  0 failed
  3105783 createKiwiImage (aarch64): free -> open (aarch64-01.rdu2.centos.org)

It seems that none of the environment groups from CentOS Stream nor EPEL are present:

The GNOME image failure in the logs is:

DEBUG util.py:444:  [ ERROR   ]: 14:10:03 | KiwiInstallPhaseFailed: System package installation failed: Module or Group 'workstation-product-environment' is not available.

The KDE image failure in the logs is:

DEBUG util.py:444:  [ ERROR   ]: 14:17:47 | KiwiInstallPhaseFailed: System package installation failed: Module or Group 'kde-desktop-environment' is not available.

According to my local environment, these both exist:

ngompa@fedora ~> podman run --rm -it quay.io/centoshyperscale/centos:stream9
bash-5.1# dnf grouplist -v
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync
DNF version: 4.14.0
cachedir: /var/cache/dnf
User-Agent: constructed: 'libdnf (CentOS Stream 9; hyperscale; Linux.x86_64)'
repo: downloading from remote: centos-hyperscale
countme: event triggered for centos-hyperscale: bucket 2
CentOS Stream 9 - Hyperscale                                                                                                                                                463 kB/s | 248 kB     00:00
centos-hyperscale: using metadata from Thu Dec  8 00:04:34 2022.
repo: downloading from remote: centos-hyperscale-experimental
countme: event triggered for centos-hyperscale-experimental: bucket 1
CentOS Stream 9 - Hyperscale Experimental                                                                                                                                   5.5 MB/s | 2.4 MB     00:00
centos-hyperscale-experimental: using metadata from Wed Dec  7 18:00:01 2022.
repo: downloading from remote: centos-hyperscale-spin
countme: event triggered for centos-hyperscale-spin: bucket 1
CentOS Stream 9 - Hyperscale Spin                                                                                                                                            42 kB/s |  36 kB     00:00
centos-hyperscale-spin: using metadata from Wed Apr  6 23:34:08 2022.
repo: downloading from remote: baseos
countme: no event for baseos: budget to spend: 3
CentOS Stream 9 - BaseOS                                                                                                                                                     13 MB/s | 6.0 MB     00:00
baseos: using metadata from Mon Dec  5 14:09:20 2022.
repo: downloading from remote: appstream
countme: event triggered for appstream: bucket 2
CentOS Stream 9 - AppStream                                                                                                                                                  10 MB/s |  16 MB     00:01
appstream: using metadata from Mon Dec  5 14:11:11 2022.
repo: downloading from remote: crb
countme: event triggered for crb: bucket 1
CentOS Stream 9 - CRB                                                                                                                                                       1.4 MB/s | 5.1 MB     00:03
crb: using metadata from Mon Dec  5 14:12:03 2022.
repo: downloading from remote: extras-common
countme: no event for extras-common: budget to spend: 1
CentOS Stream 9 - Extras packages                                                                                                                                            20 kB/s | 9.2 kB     00:00
extras-common: using metadata from Fri Nov 25 07:08:56 2022.
repo: downloading from remote: epel
countme: no event for epel: budget to spend: 1
Extra Packages for Enterprise Linux 9 - x86_64                                                                                                                              6.5 MB/s |  12 MB     00:01
epel: using metadata from Sat Dec 10 00:29:42 2022.
repo: downloading from remote: epel-next
countme: no event for epel-next: budget to spend: 2
Extra Packages for Enterprise Linux 9 - Next - x86_64                                                                                                                       1.2 MB/s | 1.4 MB     00:01
epel-next: using metadata from Thu Nov 24 03:28:46 2022.
Completion plugin: Generating completion cache...
Available Environment Groups:
   Server with GUI (graphical-server-environment)
   Server (server-product-environment)
   Minimal Install (minimal-environment)
   Workstation (workstation-product-environment)
   KDE Plasma Workspaces (kde-desktop-environment)
   Custom Operating System (custom-environment)
   Virtualization Host (virtualization-host-environment)
Available Groups:
   Legacy UNIX Compatibility (legacy-unix)
   Console Internet Tools (console-internet)
   Container Management (container-management)
   Development Tools (development)
   .NET Development (dotnet)
   Graphical Administration Tools (graphical-admin-tools)
   Headless Management (headless-management)
   Network Servers (network-server)
   RPM Development Tools (rpm-development-tools)
   Scientific Support (scientific)
   Security Tools (security-tools)
   Smart Card Support (smart-card)
   System Tools (system-tools)
   Fedora Packager (fedora-packager)
   Xfce (xfce-desktop)
bash-5.1# exit
Edited a year ago by ngompa

Metadata Update from @ngompa:
- Issue status updated to: Open (was: Closed)
a year ago

The comps.xml were imported with koji import-comps <comps-file.xml> hyperscale9s-spin_media-experimental-el9s-build and if you use cbs taginfo hyperscale9s-spin_media-experimental-el9s-build|grep Groups you'll see groups koji imported.

As you can see, there is gnome-desktop or kde-desktop listed as groups there, but what you're referring to (workstation-product-environment and kde-desktop-environment) are in fact environment categories in comps.xml, which seems to be ignored by "koji import-comps" : it seems to import groups but not environments (you can browse the appstream comps.xml to see the difference)

So maybe you should just reference in your kickstart the groups declared in these environments ? Example for workstation-product-environment from c9s appstream comps : 

  <id>workstation-product-environment</id>
    <name>Workstation</name>
    <description>Workstation is a user-friendly desktop system for laptops and PCs.</description>
    <display_order>4</display_order>
    <grouplist>
      <groupid>base-x</groupid>
      <groupid>core</groupid>
      <groupid>fonts</groupid>
      <groupid>gnome-desktop</groupid>
      <groupid>guest-desktop-agents</groupid>
      <groupid>hardware-support</groupid>
      <groupid>internet-browser</groupid>
      <groupid>multimedia</groupid>
      <groupid>networkmanager-submodules</groupid>
      <groupid>print-client</groupid>
      <groupid>standard</groupid>
      <groupid>workstation-product</groupid>
    </grouplist>
Edited a year ago by arrfab

Ugh, alright. I'm going to file a bug about this, because Koji shouldn't ignore them.

ack so nothing to be done at the infra level for now, so let me close this one again

Metadata Update from @arrfab:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)
a year ago

Login to comment on this ticket.

Boards 1
CBS Status: Backlog
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.