#696 Enable kiwi plugin for Hyperscale
Opened 7 months ago by ngompa. Modified 5 days ago

For upcoming Hyperscale work for building images, I need CBS to run Koji 1.28 or newer. Specifically because Koji 1.28 introduces full(ish) support for KIWI:

Kiwi plugin for building images based on XML description files was extended and refactored a bit, so it is now almost production-ready. We expect that in one or two releases we can flip it to first-class plugin.

We'd like to use this in Hyperscale and so we'd like CBS upgraded and the kiwiBuild task enabled for Hyperscale so we can use it.


Metadata Update from @arrfab:
- Issue tagged with: cbs, feature-request

7 months ago

[backlog refinement]
There is another ticket that needs koji to be updated to newer version.

Koji 1.29.0 is out now, so we should update to that directly.

After chatting with Davide and Neal at Devconf, they are ok to maintain and troubleshoot the kiwi plugin in CBS. Since this is included with koji, the Infra team can install the packages on the builders and the hub. The Hyperscale SIG will contribute any necessary config items to the staging branches on https://github.com/centos/ansible-role-kojihub and https://github.com/centos/ansible-role-kojid

We expect to have a CBS staging environment soon so that we can play public contributions for review.

@arrfab, @dcavalca and @ngompa does this match what you expect?

Metadata Update from @arrfab:
- Issue assigned to arrfab

a month ago

Metadata Update from @arrfab:
- Issue tagged with: high-gain, high-trouble

a month ago

the plan is to have a cbs.stg where we can test this, and on VMs (no bare-metal so no hypervisor, and I guess it's fine if kiwi doesn't need this).
Rebuilding needed pkgs that will be needed on kojihub and kojid builders is also needed.
Can we come with a list ? I've never played with kiwi so if you can come with an overview and help , we'd be happy to do that with you :)

The packages needed for the kiwiBuild command are:

  • kiwi-cli
  • kiwi-systemdeps

Both packages come from the kiwi source package in EPEL. I recommend sourcing from EPEL rather than rebuilding the dependency chain again.

For some Hyperscale deliverables, we also need btrfs-progs and btrfs usable on the builder (either through the btrfs kmod from the kmods SIG or using the hyperscale kernel)

wrt btrfs-progs, let me paste what was said on irc in #centos-meeting :

(17:59:51) Eighth_Doctor: oh, and "btrfs-progs" for hyperscale kiwi builds                                                      
(18:00:00) Eighth_Doctor: and the builder host needs to have btrfs available                                                    
(18:00:23) arrfab: so we just need to rebuild these in in the infra tags                                                       
(18:00:31) arrfab: hmm                                                                                                          
(18:00:38) davide: btrfs-progs is not in EPEL, but we do have it in hyperscale if that helps                                    
(18:00:47) arrfab: that was not mentioned (the need for host to be running hyperscale)                                          
(18:01:00) arrfab: as clearly bstinson told me that the goal is to migrate to imagebuilder and kiwi is temporary                
(18:01:16) arrfab: having one plugin is easy, modifying more for a tmp solution would need to be reconsidered then :/           
18:01:26) davide: I don't think it needs to necessarily run our variant -- it just needs to be able to handle btrfs     filesystems                                                                                                                     │
(18:01:38) davide: so e.g. the kmod-btrfs from the kmods sig + btrfs-progrs would do the trick                                  (18:01:40) bstinson: we discussed starting with existing builders                                                               (18:01:50) bstinson: let's do enablement there first                                                                            │
(18:02:05) davide: yeah I think we can take this in two stages                                                                  
(18:02:08) arrfab: bstinson: but if their image need btrfs directly, that will not work ?                                       
(18:02:16) davide: first let's get kiwi working, then we can figure out the btrfs side of things                                
(18:02:25) bstinson: there are non-btrfs variants that can be used                                                              
(18:02:32) arrfab: *ack*                                                                                          

So let's try first with default setup as it will also run in AWS, on CentOS Stream 8 images, so with vanilla kernel

WRT kiwi, I kicked a rebuild in the infra8-buildtools-common tags (inherited through our roles for all koji environments)

Can you also validate that https://docs.pagure.org/koji/plugins/?highlight=kiwi#image-builds-using-kiwi is enough ? not a lot of info but seems that it just needs to be installed without any configuration for the plugin, and then create a kiwi-build group for the build tag and then adding needed rpm pkgs in that group

Yes, that should be sufficient.

Realizing that because it needs to happen within the buildroot itself, we probably need to have it for 8s and 9s ? and should be tagged for all tags (through inheritance)
We don't support (yet) 9s in our infra so if we need that we should start first looking at supporting 9s/el8 in our infra and tags ...

But otoh, if you enable epel{8,9} in your tags, that's possible to then call kiwi-build task assuming that we also ensure that pkgs are added with add-group-pkg in thekiwi-build group in your tag. Something to test ?

@ngompa : it seems it's not possible to have deps in epel8 anyway :

dnf install kiwi-cli kiwi-systemdeps
Last metadata expiration check: 0:00:14 ago on Thu 01 Sep 2022 08:36:08 AM UTC.
Error: 
 Problem: package kiwi-cli-9.24.44-1.el8.noarch requires python3-kiwi = 9.24.44-1.el8, but none of the providers can be installed
  - conflicting requests
  - nothing provides python3.6dist(pyxattr) needed by python3-kiwi-9.24.44-1.el8.noarch
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Would you mind working on missing deps there and then we can revisit ?

Metadata Update from @arrfab:
- Issue priority set to: Waiting on Reporter (was: Needs Review)

a month ago

Metadata Update from @arrfab:
- Issue tagged with: blocked

a month ago

@arrfab That's in PowerTools for c8s.

oh ! completely forgot about that (disabled by default) ... thanks ! .. something to verify for our build tags but iirc it's enabled everywhere.

hey, I just closed #907 so we have now publicly available https://cbs.stg.centos.org on which we can play with new features and have community members interact with it.
I'll so start working on some kojid/kojihub role[s] change[s] to test that it's doable to have kiwi plugin working.
Once validated we'll just be able to replay on cbs.centos.org

Just for my own tests, can you point me to some profiles that we can use for kiwi ? Let's start with something basic/straight forward

I meant for koji and kiwi-build task :-)
It seems koji kiwi-build is expecting a profile or scm (so is that expecting path to .xml ?)

koji kiwi-build --help
Usage: koji kiwi-build [options] <target> <description_scm> <description_path>
(Specify the --help global option for a list of other help options)

Options:
  -h, --help            show this help message and exit
  --scratch             Perform a scratch build
  --release=RELEASE     Release of the output image
  --repo=REPO           Specify a repo that will override the repo used to
                        install RPMs in the image. May be used multiple times.
                        The build tag repo associated with the target is the
                        default.
  --noprogress          Do not display progress of the upload
  --kiwi-profile=KIWI_PROFILE
                        Select profile from description file
  --make-prep           Run 'make prep' in checkout before starting the build
  --can-fail=ARCH1,ARCH2,...
                        List of archs which are not blocking for build
                        (separated by commas.
  --arch=ARCHES         Limit arches to this subset
  --nowait              
  --wait                Wait on the image creation, even if running in the
                        background

path is to the directory containing config.xml, I believe. Try the complex-build one with OpenStack?

@ngompa trying to look at this but can you come with some trees/options that we can test with koji ? (not your standalone wrapper)
also, worth knowing that we can't use metalink= for repositories as kojid builders don't have access to internet and also we have to point to valid but static repositories (like mirror.stream.centos.org which is then available)

@ngompa trying to look at this but can you come with some trees/options that we can test with koji ? (not your standalone wrapper)
also, worth knowing that we can't use metalink= for repositories as kojid builders don't have access to internet and also we have to point to valid but static repositories (like mirror.stream.centos.org which is then available)

Koji is supposed to rewrite the repos automatically. Is it not doing that?

there is no koji doc about how to use it btw :-)

Just a little bit of progress :

I created a git repo to have some modified/simplified .xml for kiwi (based on yours) and not using metalink (as it will not work in our infra) and then kicked :

cbs-stg kiwi-build --scratch hyperscale9s-packages-spin-el9s git+https://git.centos.org/centos/infra-playground#95c4d5d2e15b35255ccf213fec4d06f92d8e8ffc kiwi/simple-build/
Created task: 123
Task info: https://cbs.stg.centos.org/koji/taskinfo?taskID=123
Watching tasks (this may be safely interrupted)...
123 kiwiBuild (noarch): free
123 kiwiBuild (noarch): free -> open (kojid1.stg.centos.org)
  124 createKiwiImage (x86_64): free
  124 createKiwiImage (x86_64): free -> open (kojid1.stg.centos.org)
  124 createKiwiImage (x86_64): open (kojid1.stg.centos.org) -> closed
  0 free  1 open  1 done  0 failed
123 kiwiBuild (noarch): open (kojid1.stg.centos.org) -> closed
  0 free  0 open  2 done  0 failed

123 kiwiBuild (noarch) completed successfully

=> https://cbs.stg.centos.org/koji/taskinfo?taskID=124

I tried to just build a cloud image but had to merge all .xml files into one otherwise koji kiwi plugin fails with :

Sep 13 07:14:49 kojid1.stg.euw1.centos.org kojid[267560]:     with open(file, \'rb\') as fp:
Sep 13 07:14:49 kojid1.stg.euw1.centos.org kojid[267560]: FileNotFoundError: [Errno 2] No such file or directory: \'this://./repositories/core.xml\'
Sep 13 07:14:49 kojid1.stg.euw1.centos.org kojid[267560]: '>

I tried once again but it will be probably my last test (you're now welcome to test it yourself on cbs.stg with a TLS cert from accounts.stg.centos.org and pointing to cbs.stg.centos.org for koji profile)

it fails again but different issue (from root.log ):

DEBUG util.py:446:  [ INFO    ]: 07:26:54 | Processing SELinux file security contexts
DEBUG util.py:446:  [ WARNING ]: 07:26:54 | Could not parse setfiles output
DEBUG util.py:444:  [ ERROR   ]: 07:27:05 | KiwiCommandError: chroot: stderr: setfiles: Could not set context for /run/motd:  Invalid argument
DEBUG util.py:444:  setfiles: Could not set context for /run/motd.d:  Invalid argument
DEBUG util.py:444:  setfiles: Could not set context for /var/lib/systemd/coredump:  Invalid argument
DEBUG util.py:444:  setfiles: Could not set context for /usr/lib/systemd/user/dbus-broker.service:  Invalid argument
DEBUG util.py:444:  setfiles: Could not set context for /usr/lib/systemd/systemd-network-generator:  Invalid argument
DEBUG util.py:444:  setfiles: Could not set context for /usr/bin/rpmdb:  Invalid argument
DEBUG util.py:444:  , stdout: 
1k
2k
3k
4k
5k
6k
7k
8k
9k
10k
11k
12k
13k
14k
15k
16k
17k
18k
19k
20k
21k
22k
23k
24k
25k
26k
27k
28k
29k
/ 100.0%
DEBUG util.py:446:  [ INFO    ]: 07:27:05 | Cleaning up SystemPrepare instance
DEBUG util.py:598:  Child return code was: 1

Worth knowing that host is a centos stream 8 VM (we don't support yet stream 9 in centos infra) and it seems it doens't allow kojid kiwi plugin to set correct selinux context either :

From audit.log :

type=AVC msg=audit(1663054015.307:54713): avc:  denied  { mac_admin } for  pid=279225 comm="setfiles" capability=33  scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0
type=AVC msg=audit(1663054015.308:54714): avc:  denied  { mac_admin } for  pid=279225 comm="setfiles" capability=33  scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0
type=AVC msg=audit(1663054015.813:54715): avc:  denied  { mac_admin } for  pid=279225 comm="setfiles" capability=33  scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0
type=AVC msg=audit(1663054016.057:54717): avc:  denied  { mac_admin } for  pid=279225 comm="setfiles" capability=33  scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0
type=AVC msg=audit(1663054016.169:54718): avc:  denied  { mac_admin } for  pid=279225 comm="setfiles" capability=33  scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0
type=AVC msg=audit(1663054019.551:54719): avc:  denied  { mac_admin } for  pid=279225 comm="setfiles" capability=33  scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=capability2 permissive=0

We run with selinux in enforcing mode everywhere and it should stay like that, so can you verify with kiwi and koji about how to support that ? we can then revisit later if we want to enable it on centos cbs builders ?

I just tried with setenforce 0 on that isolated builder (and temporary for that test) and it then works :

https://cbs.stg.centos.org/koji/taskinfo?taskID=138

So, apart from the selinux issue (already trying to see how to rebuild a custom .pp eventually), I gave the same build another try (no difference) but it now fails : https://cbs.stg.centos.org/kojifiles/work/tasks/142/142/root.log

DEBUG util.py:446:  [ INFO    ]: 08:28:55 | Creating volume group systemVG
DEBUG util.py:444:  [ ERROR   ]: 08:28:55 | KiwiCommandError: vgcreate: stderr:   /dev/systemVG: already exists in filesystem
DEBUG util.py:444:    Run `vgcreate --help' for more information.
DEBUG util.py:444:  , stdout: (no output on stdout)

From what I can see it initially created /dev/loop devices but it seems it doesn't clean these after builds, so new job would complain about some existing VG/LV on it.
Something to report to upstream koji ?

Interesting, I'll try it locally. I'm running it definitely with enforcing selinux. dev leak seems to me like a kiwi issue, but will check it.

Finally found why : it's just that kojid by default doesn't clean-up directly the buildroots, but if you wait 2 minutes (default for buildroot_basic_cleanup_delay=120) it then just works fine.

For the selinux policy, I also built a custom policy and it seems to work :

cat /root/selinux/centos-kojid-kiwi.te ; echo ====== ; echo "selinux status is : $(getenforce)" 

module centos-kojid-kiwi 1.0;

require {
    type unconfined_service_t;
    class capability2 mac_admin;
}

#============= unconfined_service_t ==============
allow unconfined_service_t self:capability2 mac_admin;
======
selinux status is : Enforcing

I'll modify our ansible kojid role to distribute (and load) that .pp file (I don't like it but better than whole setenforce 0 afaics)

I submitted some scratch builds and they all worked : https://cbs.stg.centos.org/koji/index

@tkopecek I think the Koji kiwi config flattener doesn't recognize this:// as a protocol yet. It's basically a relative version of file:// so you can translate this:// to file://${PWD} and resolve the URLs.

Metadata Update from @arrfab:
- Issue untagged with: blocked

19 days ago

@tkopecek It also seems that based on the output from this build log that it's not using the kiwi description koji generates: https://cbs.stg.centos.org/koji/taskinfo?taskID=160

We should probably be renaming any existing config.xml or *.kiwi files out of the way so that the generated one is used.

Maybe better to use directly --kiwi-file instead of --description.

When using --kiwi-file, you still need --description. But yeah, it may make sense to set that.

@ngompa happy to just deploy kiwi plugin on cbs.centos.org (we'll have one host in the image channel) as it seems to work at this stage. Thoughts ?

I pushed that change and it's now enabled on cbs (kojihub and kojid) but nothing is configured for tags (as we need to add the kiwi-build group on the build tags ..
Waiting for feedback on which one you'd like to see it enabled

Let's configure the kiwi-build group to the hyperscale8s-spin_media-experimental-el8s-build and hyperscale8s-spin_media-main-el8s-build tags.

I also noticed that the hyperscale8s-spin_media-experimental-el8s-build tag doesn't have hyperscale8s-packages-experimental-release in its inheritance and it should be added.

The kiwi-build group needs the following: kiwi-cli, kiwi-systemdeps, distribution-gpg-keys, btrfs-progs.

(Having the btrfs-progs userspace in the group will prepare us for later when we have Btrfs-enabled CBS builders)

We also need versions of the hyperscale8s-spin_media-* tags for c9s.

The formula we want here is the same as for c8s: hyperscale9s-spin_media-{main,experimental}-{el9s-build,candidate,testing,release}.

The hyperscale9s-spin_media-*-el9s-build tags should be configured the same way the hyperscale8s-spin_media-*-el8s-build tags were.

Let's try first with the 8s ones :

koji add-group hyperscale8s-spin_media-experimental-el8s-build kiwi-build
koji add-group-pkg hyperscale8s-spin_media-experimental-el8s-build kiwi-build kiwi-cli kiwi-systemdeps distribution-gpg-keys btrfs-progs

koji add-group hyperscale8s-spin_media-main-el8s-build kiwi-build
koji add-group-pkg hyperscale8s-spin_media-main-el8s-build kiwi-build kiwi-cli kiwi-systemdeps distribution-gpg-keys btrfs-progs

Also added the inheritance on hyperscale8s-spin_media-experimental-el8s-build :

cbs taginfo hyperscale8s-spin_media-experimental-el8s-build
Tag: hyperscale8s-spin_media-experimental-el8s-build [2341]
Arches: x86_64 aarch64
Groups: build, livecd-build, livemedia-build, srpm-build
Tag options:
  mock.package_manager : 'dnf'
  mock.yum.module_hotfixes : 1
  rpm.macro.vendor : 'CentOS Hyperscale SIG'
This tag is a buildroot for one or more targets
Current repo: repo#964125: 2022-09-22 03:47:51.050014+00:00
Targets that build from this tag:
  hyperscale8s-spin_media-experimental-el8s
External repos:
    5 centos8s-cr (http://mirror.centos.org/centos/8-stream//cr/$arch/os/, merge mode: bare), arches: inherited from tag
   10 centos8s-extras (http://mirror.centos.org/centos/8-stream//extras/$arch/os/, merge mode: bare), arches: inherited from tag
   15 centos8s-powertools (http://mirror.centos.org/centos/8-stream//PowerTools/$arch/os/, merge mode: bare), arches: inherited from tag
   20 centos8s-appstream (http://mirror.centos.org/centos/8-stream//AppStream/$arch/os/, merge mode: bare), arches: inherited from tag
   25 centos8s-baseos (http://mirror.centos.org/centos/8-stream//BaseOS/$arch/os/, merge mode: bare), arches: inherited from tag
   30 epel8 (https://cbs.centos.org/kojifiles/repos/epel/8/Everything/$arch/, merge mode: bare), arches: inherited from tag
   35 epel8-next (https://cbs.centos.org/kojifiles/repos/epel/next/8/Everything/$arch/, merge mode: bare), arches: inherited from tag
Inheritance:
  0    .... hyperscale8s-packages-main-release [2249]
  5    .... buildsys8s-release [1866]
  10   .... hyperscale8s-spin_media-experimental-candidate [2338]
  20   .... hyperscale8s-packages-hotfixes-release [2305]
  25   .... hyperscale8s-packages-spin-release [2301]
  30   .... hyperscale8s-packages-experimental-release [2245]

Can you give it now a try ? worth knowing that it should stay in -candidate for now as current signing+push releng process doesn't know anything about non RPM repositories (we recently added a function for DuD .iso artifacts for kmods SIG as one example) but that can be then another ticket

Per discussion with @ngompa on irc, I built https://src.fedoraproject.org/rpms/koji/c/c15b855e4e075e2d5ffa586c9ed7fc870796393f?branch=rawhide , which is koji 1.30.0-2 , which has the backported patches/fixes for kiwi (merged in upstream koji)
It's now deployed on CBS : https://cbs.centos.org/koji/api

Per discussion with @ngompa on irc (follow-up) I already also created the CentOS Stream 9 tags :

cbs taginfo hyperscale9s-spin_media-main-el9s-build
Tag: hyperscale9s-spin_media-main-el9s-build [2661]
Arches: x86_64 aarch64
Groups: build, kiwi-build, srpm-build
Tag options:
  mock.new_chroot : 0
  mock.package_manager : 'dnf'
  mock.yum.module_hotfixes : 1
  rpm.macro.vendor : 'CentOS Hyperscale SIG'
This tag is a buildroot for one or more targets
Current repo: repo#965929: 2022-09-27 07:19:42.905150+00:00
Targets that build from this tag:
  hyperscale9s-spin_media-main-el9s
External repos:
    5 centos9s-baseos (http://mirror.stream.centos.org/9-stream/BaseOS/$arch/os/, merge mode: bare), arches: inherited from tag
   10 centos9s-appstream (http://mirror.stream.centos.org/9-stream/AppStream/$arch/os/, merge mode: bare), arches: inherited from tag
   15 centos9s-crb (http://mirror.stream.centos.org/9-stream/CRB/$arch/os/, merge mode: bare), arches: inherited from tag
   20 epel9 (https://cbs.centos.org/kojifiles/repos/epel/9/Everything/$arch/, merge mode: bare), arches: inherited from tag
   25 epel9-next (https://cbs.centos.org/kojifiles/repos/epel/next/9/Everything/$arch/, merge mode: bare), arches: inherited from tag
Inheritance:
  0    .... hyperscale9s-packages-main-release [2378]
  5    .... buildsys9s-release [2363]
  10   .... hyperscale9s-spin_media-main-candidate [2658]
  15   .... hyperscale9s-packages-hotfixes-release [2414]
  20   .... hyperscale9s-packages-spin-release [2382]


cbs taginfo hyperscale9s-spin_media-experimental-el9s-build
Tag: hyperscale9s-spin_media-experimental-el9s-build [2665]
Arches: x86_64 aarch64
Groups: build, kiwi-build, srpm-build
Tag options:
  mock.new_chroot : 0
  mock.package_manager : 'dnf'
  mock.yum.module_hotfixes : 1
  rpm.macro.vendor : 'CentOS Hyperscale SIG'
This tag is a buildroot for one or more targets
Current repo: repo#965930: 2022-09-27 07:21:13.216410+00:00
Targets that build from this tag:
  hyperscale9s-spin_media-experimental-el9s
External repos:
    5 centos9s-baseos (http://mirror.stream.centos.org/9-stream/BaseOS/$arch/os/, merge mode: bare), arches: inherited from tag
   10 centos9s-appstream (http://mirror.stream.centos.org/9-stream/AppStream/$arch/os/, merge mode: bare), arches: inherited from tag
   15 centos9s-crb (http://mirror.stream.centos.org/9-stream/CRB/$arch/os/, merge mode: bare), arches: inherited from tag
   20 epel9 (https://cbs.centos.org/kojifiles/repos/epel/9/Everything/$arch/, merge mode: bare), arches: inherited from tag
   25 epel9-next (https://cbs.centos.org/kojifiles/repos/epel/next/9/Everything/$arch/, merge mode: bare), arches: inherited from tag
Inheritance:
  0    .... hyperscale9s-packages-main-release [2378]
  5    .... buildsys9s-release [2363]
  10   .... hyperscale9s-spin_media-experimental-candidate [2662]
  15   .... hyperscale9s-packages-hotfixes-release [2414]
  20   .... hyperscale9s-packages-spin-release [2382]
  25   .... hyperscale9s-packages-experimental-release [2410]

Can you verify and validate please ?

Login to comment on this ticket.

Boards 1
CBS Status: Backlog