PR#1248: Switch Fedora Cloud and Container builds to use kiwibuild - pungi-fedora

pungi-fedora

#1248 Switch Fedora Cloud and Container builds to use kiwibuild

Merged 10 months ago by kevin. Opened 10 months ago by ngompa.

ngompa/pungi-fedora sweeten-cloud-containers into main

Restore image_build run for Fedora Cloud on s390x for now

Neal Gompa • 10 months ago

fdad9c7

Switch Fedora Cloud and Container builds to use kiwibuild

Neal Gompa • 10 months ago

9996561

fedora-cloud.conf

file modified

+59 -37

		`@@ -135,6 +135,7 @@`
		`image_volid_formats = [`
		`'Fedora-%(variant)s-%(disc_type)s-%(arch)s-%(version)s'`
		`]`
		`+`
		`image_build = {`
		`'^Cloud$': [`
		`{`
		`@@ -148,7 +149,8 @@`
		`'kickstart': 'fedora-cloud-base.ks',`
		`'distro': 'Fedora-22',`
		`'disk_size': 4,`
		`- 'arches': ['aarch64', 'ppc64le', 's390x', 'x86_64'],`
		`+ 'arches': ['s390x'],`
		`+ 'failable': ['s390x'],`
		`'repo': [`
		`'https://kojipkgs.fedoraproject.org/compose/updates/f41-updates/compose/Everything/$arch/os/'`
		`'https://kojipkgs.fedoraproject.org/compose/41/latest-Fedora-41/compose/Everything/$arch/os/'`
		`@@ -156,47 +158,67 @@`
		`'install_tree_from': 'https://kojipkgs.fedoraproject.org/compose/41/latest-Fedora-41/compose/Everything/$arch/os/',`
		`'subvariant': 'Cloud_Base'`
		`}`
		`+ }`
		`+ ],`
		`+ }`
		`+`
		`+ kiwibuild = {`
		`+ '^Cloud$': [`
		`+ {`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Cloud-Base-Generic',`
		`+ 'arches': ['aarch64', 'ppc64le', 'x86_64'],`
		`+ 'repos': ['Everything'],`
		`+ 'subvariant': 'Cloud_Base',`
		`+ 'failable': ['ppc64le'],`
		`},`
		`{`
		`- 'image-build': {`
		`- 'format': [('vagrant-libvirt','vagrant-libvirt.box'), ('vagrant-virtualbox','vagrant-virtualbox.box')]`
		`- 'name': 'Fedora-Cloud-Base-Vagrant',`
		`- 'target': 'f41',`
		`- 'version': '41',`
		`- 'release': None,`
		`- 'ksurl': 'git+https://pagure.io/fedora-kickstarts.git?#origin/f41',`
		`- 'kickstart': 'fedora-cloud-base-vagrant.ks',`
		`- 'distro': 'Fedora-22',`
		`- 'disk_size': 40,`
		`- 'arches': ['x86_64'],`
		`- 'repo': [`
		`- 'https://kojipkgs.fedoraproject.org/compose/updates/f41-updates/compose/Everything/$arch/os/'`
		`- 'https://kojipkgs.fedoraproject.org/compose/41/latest-Fedora-41/compose/Everything/$arch/os/'`
		`- ],`
		`- 'install_tree_from': 'https://kojipkgs.fedoraproject.org/compose/41/latest-Fedora-41/compose/Everything/$arch/os/',`
		`- 'subvariant': 'Cloud_Base',`
		`- }`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Cloud-Base-AmazonEC2',`
		`+ 'arches': ['aarch64', 'x86_64'],`
		`+ 'repos': ['Everything'],`
		`+ 'subvariant': 'Cloud_Base',`
		`+ 'failable': ['*'],`
		`},`
		`{`
		`- 'image-build': {`
		`- 'format': [('vpc','vpc')],`
		`- 'name': 'Fedora-Cloud-Base-Azure',`
		`- 'target': 'f41',`
		`- 'version': '41',`
		`- 'release': None,`
		`- 'ksurl': 'git+https://pagure.io/fedora-kickstarts.git?#origin/f41',`
		`- 'kickstart': 'fedora-cloud-base-azure.ks',`
		`- 'distro': 'Fedora-30',`
		`- 'disk_size': 10,`
		`- 'arches': ['aarch64', 'x86_64'],`
		`- 'repo': [`
		`- 'https://kojipkgs.fedoraproject.org/compose/updates/f41-updates/compose/Everything/$arch/os/'`
		`- 'https://kojipkgs.fedoraproject.org/compose/41/latest-Fedora-41/compose/Everything/$arch/os/'`
		`- ],`
		`- 'install_tree_from': 'https://kojipkgs.fedoraproject.org/compose/41/latest-Fedora-41/compose/Everything/$arch/os/',`
		`- 'subvariant': 'Cloud_Base'`
		`- }`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Cloud-Base-Azure',`
		`+ 'arches': ['aarch64', 'x86_64'],`
		`+ 'repos': ['Everything'],`
		`+ 'subvariant': 'Cloud_Base',`
		`+ 'failable': ['*'],`
		`+ },`
		`+ {`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Cloud-Base-GCE',`
		`+ 'arches': ['aarch64', 'x86_64'],`
		`+ 'repos': ['Everything'],`
		`+ 'subvariant': 'Cloud_Base',`
		`+ 'failable': ['*'],`
		`},`
		`+ {`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Cloud-Base-Vagrant-libvirt',`
		`+ 'arches': ['aarch64', 'x86_64'],`
		`+ 'repos': ['Everything'],`
		`+ 'subvariant': 'Cloud_Base',`
		`+ 'failable': ['*'],`
		`+ },`
		`+ {`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Cloud-Base-Vagrant-VirtualBox',`
		`+ 'arches': ['x86_64'],`
		`+ 'repos': ['Everything'],`
		`+ 'subvariant': 'Cloud_Base',`
		`+ 'failable': ['*'],`
		`+ }`
		`],`
		`}`
		`+`
		`koji_profile = 'compose_koji'`

fedora-container.conf

file modified

+29 -63

		`@@ -134,79 +134,45 @@`
		`'Fedora-%(variant)s-%(disc_type)s-%(arch)s-%(version)s'`
		`]`

		`- # These will be inherited by live_media, live_images and image_build`
		`- global_ksurl = 'git+https://pagure.io/fedora-kickstarts.git?#origin/f41'`
		`+ # These will be inherited by live_media, live_images, image_build, and kiwibuild`
		`global_release = '!RELEASE_FROM_LABEL_DATE_TYPE_RESPIN'`
		`global_version = '41'`
		`global_target = 'f41'`

		`- image_build = {`
		`+ kiwibuild = {`
		`'^Container$': [`
		`{`
		`- 'image-build': {`
		`- 'format': [('docker', 'tar.xz')],`
		`- 'name': 'Fedora-Container-Base',`
		`- 'kickstart': 'fedora-container-base.ks',`
		`- 'distro': 'Fedora-22',`
		`- 'disk_size': 5,`
		`- 'arches': ['aarch64', 'ppc64le', 's390x', 'x86_64'],`
		`- 'install_tree_from': 'https://kojipkgs.fedoraproject.org/compose/41/latest-Fedora-41/compose/Everything/$arch/os/',`
		`- 'repo': [`
		`- 'https://kojipkgs.fedoraproject.org/compose/updates/f41-updates/compose/Everything/$arch/os/'`
		`- 'https://kojipkgs.fedoraproject.org/compose/41/latest-Fedora-41/compose/Everything/$arch/os/'`
		`- ],`
		`- 'subvariant': 'Container_Base'`
		`- },`
		`- 'factory-parameters': {`
		`- 'dockerversion': "1.10.1",`
		`- 'docker_cmd': '[ "/bin/bash" ]',`
		`- 'docker_env': '[ "DISTTAG=f41container", "FGC=f41", "container=oci" ]',`
		`- 'docker_label': '{ "name": "fedora", "license": "MIT", "vendor": "Fedora Project", "version": "41"}',`
		`- },`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Container-Base-Generic',`
		`+ 'arches': ['aarch64', 'ppc64le', 's390x', 'x86_64'],`
		`+ 'repos': [`
		`+ 'https://kojipkgs.fedoraproject.org/compose/updates/f41-updates/compose/Everything/$arch/os/'`
		`+ 'https://kojipkgs.fedoraproject.org/compose/41/latest-Fedora-41/compose/Everything/$arch/os/'`
		`+ ],`
		`+ 'subvariant': 'Container_Base',`
		`},`
		`{`
		`- 'image-build': {`
		`- 'format': [('docker', 'tar.xz')],`
		`- 'name': 'Fedora-Container-Minimal-Base',`
		`- 'kickstart': 'fedora-container-base-minimal.ks',`
		`- 'distro': 'Fedora-22',`
		`- 'disk_size': 5,`
		`- 'arches': ['aarch64', 'ppc64le', 's390x', 'x86_64'],`
		`- 'install_tree_from': 'https://kojipkgs.fedoraproject.org/compose/41/latest-Fedora-41/compose/Everything/$arch/os/',`
		`- 'repo': [`
		`- 'https://kojipkgs.fedoraproject.org/compose/updates/f41-updates/compose/Everything/$arch/os/'`
		`- 'https://kojipkgs.fedoraproject.org/compose/41/latest-Fedora-41/compose/Everything/$arch/os/'`
		`- ],`
		`- 'subvariant': 'Container_Minimal_Base',`
		`- },`
		`- 'factory-parameters': {`
		`- 'dockerversion': "1.10.1",`
		`- 'docker_cmd': '[ "/bin/bash" ]',`
		`- 'docker_env': '[ "DISTTAG=f41container", "FGC=f41", "container=oci" ]',`
		`- 'docker_label': '{ "name": "fedora", "license": "MIT", "vendor": "Fedora Project", "version": "41"}',`
		`- },`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Container-Base-Generic-Minimal',`
		`+ 'arches': ['aarch64', 'ppc64le', 's390x', 'x86_64'],`
		`+ 'repos': [`
		`+ 'https://kojipkgs.fedoraproject.org/compose/updates/f41-updates/compose/Everything/$arch/os/'`
		`+ 'https://kojipkgs.fedoraproject.org/compose/41/latest-Fedora-41/compose/Everything/$arch/os/'`
		`+ ],`
		`+ 'subvariant': 'Container_Minimal_Base',`
		`},`
		`{`
		`- 'image-build': {`
		`- 'format': [('docker', 'tar.xz')],`
		`- 'name': 'Fedora-Container-Toolbox',`
		`- 'kickstart': 'fedora-container-toolbox.ks',`
		`- 'distro': 'Fedora-22',`
		`- 'disk_size': 5,`
		`- 'arches': ['aarch64', 'ppc64le', 's390x', 'x86_64'],`
		`- 'install_tree_from': 'https://kojipkgs.fedoraproject.org/compose/41/latest-Fedora-41/compose/Everything/$arch/os/',`
		`- 'repo': [`
		`- 'https://kojipkgs.fedoraproject.org/compose/updates/f41-updates/compose/Everything/$arch/os/'`
		`- 'https://kojipkgs.fedoraproject.org/compose/41/latest-Fedora-41/compose/Everything/$arch/os/'`
		`- ],`
		`- 'subvariant': 'Container_Toolbox',`
		`- },`
		`- 'factory-parameters': {`
		`- 'dockerversion': "1.10.1",`
		`- 'docker_cmd': '[ "/bin/bash" ]',`
		`- 'docker_env': '[ "container=oci" ]',`
		`- 'docker_label': '{ "name": "fedora", "license": "MIT", "vendor": "Fedora Project", "version": "41", "com.github.containers.toolbox": "true"}',`
		`- },`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Container-Toolbox',`
		`+ 'arches': ['aarch64', 'ppc64le', 's390x', 'x86_64'],`
		`+ 'repos': [`
		`+ 'https://kojipkgs.fedoraproject.org/compose/updates/f41-updates/compose/Everything/$arch/os/'`
		`+ 'https://kojipkgs.fedoraproject.org/compose/41/latest-Fedora-41/compose/Everything/$arch/os/'`
		`+ ],`
		`+ 'subvariant': 'Container_Toolbox',`
		`}`
		`],`
		`}`

fedora.conf

file modified

+92 -115

		`@@ -203,69 +203,8 @@`
		`# live_images ignores this in favor of live_target`
		`global_target = 'f41'`

		`+`
		`image_build = {`
		`- '^Container$': [`
		`- {`
		`- 'image-build': {`
		`- 'format': [('docker', 'tar.xz')],`
		`- 'name': 'Fedora-Container-Base',`
		`- 'kickstart': 'fedora-container-base.ks',`
		`- 'distro': 'Fedora-22',`
		`- 'disk_size': 5,`
		`- 'arches': ['aarch64', 'ppc64le', 's390x', 'x86_64'],`
		`- 'repo': 'Everything',`
		`- 'install_tree_from': 'Everything',`
		`- 'subvariant': 'Container_Base',`
		`- 'failable': ['*'],`
		`- },`
		`- 'factory-parameters': {`
		`- 'dockerversion': "1.10.1",`
		`- 'docker_cmd': '[ "/bin/bash" ]',`
		`- 'docker_env': '[ "DISTTAG=f41container", "FGC=f41", "container=oci" ]',`
		`- 'docker_label': '{ "name": "fedora", "license": "MIT", "vendor": "Fedora Project", "version": "41"}',`
		`- },`
		`- },`
		`- {`
		`- 'image-build': {`
		`- 'format': [('docker', 'tar.xz')],`
		`- 'name': 'Fedora-Container-Minimal-Base',`
		`- 'kickstart': 'fedora-container-base-minimal.ks',`
		`- 'distro': 'Fedora-22',`
		`- 'disk_size': 5,`
		`- 'arches': ['aarch64', 'ppc64le', 's390x', 'x86_64'],`
		`- 'repo': 'Everything',`
		`- 'install_tree_from': 'Everything',`
		`- 'subvariant': 'Container_Minimal_Base',`
		`- 'failable': ['*'],`
		`- },`
		`- 'factory-parameters': {`
		`- 'dockerversion': "1.10.1",`
		`- 'docker_cmd': '[ "/bin/bash" ]',`
		`- 'docker_env': '[ "DISTTAG=f41container", "FGC=f41", "container=oci" ]',`
		`- 'docker_label': '{ "name": "fedora", "license": "MIT", "vendor": "Fedora Project", "version": "41"}',`
		`- },`
		`- },`
		`- {`
		`- 'image-build': {`
		`- 'format': [('docker', 'tar.xz')],`
		`- 'name': 'Fedora-Container-Toolbox',`
		`- 'kickstart': 'fedora-container-toolbox.ks',`
		`- 'distro': 'Fedora-22',`
		`- 'disk_size': 5,`
		`- 'arches': ['aarch64', 'ppc64le', 's390x', 'x86_64'],`
		`- 'repo': 'Everything',`
		`- 'install_tree_from': 'Everything',`
		`- 'subvariant': 'Container_Toolbox',`
		`- 'failable': [''],`
		`- },`
		`- 'factory-parameters': {`
		`- 'dockerversion': "1.10.1",`
		`- 'docker_cmd': '[ "/bin/bash" ]',`
		`- 'docker_env': '[ "container=oci" ]',`
		`- 'docker_label': '{ "name": "fedora", "license": "MIT", "vendor": "Fedora Project", "version": "41", "com.github.containers.toolbox": "true"}',`
		`- },`
		`- }`
		`- ],`
		`'^Cloud$': [`
		`{`
		`'image-build': {`
		`@@ -274,63 +213,13 @@`
		`'kickstart': 'fedora-cloud-base.ks',`
		`'distro': 'Fedora-22',`
		`'disk_size': 5,`
		`- 'arches': ['aarch64', 'ppc64le', 's390x', 'x86_64'],`
		`+ 'arches': ['s390x'],`
		`'repo': 'Everything',`
		`'install_tree_from': 'Everything',`
		`'subvariant': 'Cloud_Base',`
		`- 'failable': ['ppc64le', 's390x'],`
		`+ 'failable': ['s390x'],`
		`}`
		`- },`
		`- {`
		`- 'image-build': {`
		`- 'format': [('tar-gz', 'tar.gz')],`
		`- 'name': 'Fedora-Cloud-Base-GCP',`
		`- 'kickstart': 'fedora-cloud-base-gcp.ks',`
		- # Use `Fedora-30` profile here because we need it
		`- # to be a UEFI image and currently OZ does that if`
		- # we use the `Fedora-30` profile.
		`- 'distro': 'Fedora-30',`
		`- # GCP engineers have told me the disk images have bad`
		`- # performance if they are anything less than 10G`
		`- 'disk_size': 10,`
		`- 'arches': ['x86_64'],`
		`- 'repo': 'Everything',`
		`- 'install_tree_from': 'Everything',`
		`- 'subvariant': 'Cloud_Base',`
		`- 'failable': ['x86_64'],`
		`- }`
		`- },`
		`- {`
		`- 'image-build': {`
		`- 'format': [('vpc','vpc')],`
		`- 'name': 'Fedora-Cloud-Base-Azure',`
		`- 'kickstart': 'fedora-cloud-base-azure.ks',`
		- # Use `Fedora-30` profile here because we need it
		`- # to be a UEFI image and currently OZ does that if`
		- # we use the `Fedora-30` profile.
		`- 'distro': 'Fedora-30',`
		`- 'disk_size': 5,`
		`- 'arches': ['aarch64', 'x86_64'],`
		`- 'repo': 'Everything',`
		`- 'install_tree_from': 'Everything',`
		`- 'subvariant': 'Cloud_Base',`
		`- 'failable': ['*'],`
		`- }`
		`- },`
		`- {`
		`- 'image-build': {`
		`- 'format': [('vagrant-libvirt','vagrant-libvirt.box'), ('vagrant-virtualbox','vagrant-virtualbox.box')],`
		`- 'name': 'Fedora-Cloud-Base-Vagrant',`
		`- 'kickstart': 'fedora-cloud-base-vagrant.ks',`
		`- 'distro': 'Fedora-22',`
		`- 'disk_size': 40,`
		`- 'arches': ['x86_64'],`
		`- 'repo': 'Everything',`
		`- 'install_tree_from': 'Everything',`
		`- 'subvariant': 'Cloud_Base',`
		`- 'failable': ['*'],`
		`- },`
		`- },`
		`+ }`
		`],`
		`'^Labs$': [`
		`{`
		`@@ -501,6 +390,94 @@`
		`],`
		`}`

		`+ kiwibuild = {`
		`+ '^Cloud$': [`
		`+ {`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Cloud-Base-Generic',`
		`+ 'arches': ['aarch64', 'ppc64le', 'x86_64'],`
		`+ 'repos': ['Everything'],`
		`+ 'subvariant': 'Cloud_Base',`
		`+ 'failable': ['ppc64le'],`
		`+ },`
		`+ {`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Cloud-Base-AmazonEC2',`
		`+ 'arches': ['aarch64', 'x86_64'],`
		`+ 'repos': ['Everything'],`
		`+ 'subvariant': 'Cloud_Base',`
		`+ 'failable': ['*'],`
		`+ },`
		`+ {`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Cloud-Base-Azure',`
		`+ 'arches': ['aarch64', 'x86_64'],`
		`+ 'repos': ['Everything'],`
		`+ 'subvariant': 'Cloud_Base',`
		`+ 'failable': ['*'],`
		`+ },`
		`+ {`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Cloud-Base-GCE',`
		`+ 'arches': ['aarch64', 'x86_64'],`
		`+ 'repos': ['Everything'],`
		`+ 'subvariant': 'Cloud_Base',`
		`+ 'failable': ['*'],`
		`+ },`
		`+ {`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Cloud-Base-Vagrant-libvirt',`
		`+ 'arches': ['aarch64', 'x86_64'],`
		`+ 'repos': ['Everything'],`
		`+ 'subvariant': 'Cloud_Base',`
		`+ 'failable': ['*'],`
		`+ },`
		`+ {`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Cloud-Base-Vagrant-VirtualBox',`
		`+ 'arches': ['x86_64'],`
		`+ 'repos': ['Everything'],`
		`+ 'subvariant': 'Cloud_Base',`
		`+ 'failable': ['*'],`
		`+ }`
		`+ ],`
		`+ '^Container$': [`
		`+ {`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Container-Base-Generic',`
		`+ 'arches': ['aarch64', 'ppc64le', 's390x', 'x86_64'],`
		`+ 'repos': ['Everything'],`
		`+ 'subvariant': 'Container_Base',`
		`+ 'failable': ['*'],`
		`+ },`
		`+ {`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Container-Base-Generic-Minimal',`
		`+ 'arches': ['aarch64', 'ppc64le', 's390x', 'x86_64'],`
		`+ 'repos': ['Everything'],`
		`+ 'subvariant': 'Container_Minimal_Base',`
		`+ 'failable': ['*'],`
		`+ },`
		`+ {`
		`+ 'description_scm': 'git+https://pagure.io/fedora-kiwi-descriptions.git?#HEAD',`
		`+ 'description_path': 'Fedora.kiwi',`
		`+ 'kiwi_profile': 'Container-Toolbox',`
		`+ 'arches': ['aarch64', 'ppc64le', 's390x', 'x86_64'],`
		`+ 'repos': ['Everything'],`
		`+ 'subvariant': 'Container_Toolbox',`
		`+ 'failable': [''],`
		`+ }`
		`+ ],`
		`+ }`
		`+`
		`live_media = {`
		`'^Workstation$': [`
		`{`

ngompa commented 10 months ago

Reference: https://fedoraproject.org/wiki/Changes/KiwiBuiltCloudImages

ngompa commented 10 months ago

I don't know if I'm doing this right, I would certainly appreciate some real reviews here...

CC: @adamwill @davdunc @dcavalca

rebased onto 30b907efe3db80b8c66751d9c1739e7fee503e58

10 months ago

rebased onto 6714e9fbe4562d4beb0834ce1cb00fc7eed76e7b

10 months ago

2 new commits added

Restore image_build run for Fedora Cloud on s390x for now
Switch Fedora Cloud and Container builds to use kiwibuild

10 months ago

adamwill commented 10 months ago

This will need at least https://pagure.io/pungi/pull-request/1735 , I believe, to work.

rebased onto 02a9fa9e54246fb955212cd8c055dc4ccb0a5f99

10 months ago

adamwill commented 10 months ago

We also worked out that the factory-parameters dicts do nothing for kiwi builds. how those work is:

pungi writes them into a config file it passes to koji - e.g. in today's compose, pungi ran koji --profile=compose_koji image-build --config=/mnt/koji/compose/rawhide/Fedora-Rawhide-20240306.n.0/work/image-build/Container/docker_Fedora-Container-Base_aarch64-ppc64le-s390x-x86_64.cfg --wait. That file can be seen here, and you can see it has a [factory-parameters] section
The Koji code that handles the image-build subcommand reads that config file and passes the factory-parameters values into the options for the task as a list called task_options.factory_parameter
That code then calls _build_image_oz() with those task options, which creates a buildImageOz instance with the factory_parameter opt list in its opts
buildImageOz (here) passes the opts through to make_task, which...does a lot of stuff! It all gets complicated here and I am not gonna trace it all the way out, but it's clear that we wind up with a BaseImageTask (here), which has a _mergeFactoryParams() method that actually reads in the factory parameters. It's called by _buildBase() and _do_target_image() methods, which pass them on to ImageFactory methods (imgfac.BuildDispatcher.builder_for_base_image() or imgfac.BuildDispatcher.builder_for_target_image())

Point being, all of this is completely oz/ImageFactory-specific code paths.

Looking into how this works on the Kiwi codepath, well, this factory-parameters stuff just is not handled at any level on that codepath. The Pungi kiwibuild phase definition does not do anything with it. The Koji kiwi plugin's CLI command parser does not read config files or do anything at all with the word 'factory' in it. The Kiwi image build tasks similarlly don't.

But also, uh...it means this PR is actually wrong in another way. We can't have the variant lists be lists of dicts with kiwibuild and optional factory-parameters keys like they are in this PR, that is not what the kiwibuild phase expects. It's one level too much nesting. If you look at the config @lsedlar used for the test compose, it doesn't look like that. The variant lists need to just be lists of image dicts. So it should be something like:

kiwibuild = {
    '^Cloud$': [
        {
            'kiwi_profile': 'Cloud-Base-Generic',
            'arches': ['aarch64', 'ppc64le', 'x86_64'],
            'repos': ['Everything'],
            'subvariant': 'Cloud_Base',
            'failable': ['ppc64le'],
        },
        {
            'kiwi_profile': 'Cloud-Base-AmazonEC2',
            'arches': ['aarch64', 'x86_64'],
            'repos': ['Everything'],
            'subvariant': 'Cloud_Base',
            'failable': ['*'],
        },

rather than the way it is now.

rebased onto 6ca7221d879c0b6b771f1bb59ebdd08083ff2994

10 months ago

adamwill commented 10 months ago

This also needs https://pagure.io/pungi/pull-request/1736 (thanks @jcline ).

rebased onto 72b6c5715e36bfc5c39a1a1e6c40c76c6902a736

10 months ago

ngompa commented 10 months ago

All the requested changes have been made. Corresponding changes have been made to fedora-kiwi-descriptions too.

adamwill commented 10 months ago

So, what's the plan here? Should we ask @lsedlar to do a release and build of Pungi, then merge this for Rawhide and see what happens? edit: or do we want to do a bigger 'test compose' first? The test compose we ran before only covered the Cloud images and didn't have the latest changes...

Edited 10 months ago by adamwill

kevin commented 10 months ago

I think getting rawhide to work would be best and much easier than trying to do a complete test compose... but if folks really want to do that, we will need to create a fedora-test-compose.conf pungi config for it...

ngompa commented 10 months ago

Let's just start with Rawhide. Once it's good, I'll branch kiwi-descriptions for f40 and make the corresponding PR.

kevin commented 10 months ago

seeing some validation errors...

Failed validation in kiwibuild.^Cloud$.0: 'description_scm' is a required property
Failed validation in kiwibuild.^Cloud$.0: 'description_path' is a required property
Failed validation in kiwibuild.^Cloud$.1: 'description_scm' is a required property
Failed validation in kiwibuild.^Cloud$.1: 'description_path' is a required property
Failed validation in kiwibuild.^Cloud$.2: 'description_scm' is a required property
Failed validation in kiwibuild.^Cloud$.2: 'description_path' is a required property
Failed validation in kiwibuild.^Cloud$.3: 'description_scm' is a required property
Failed validation in kiwibuild.^Cloud$.3: 'description_path' is a required property
Failed validation in kiwibuild.^Cloud$.4: 'description_scm' is a required property
Failed validation in kiwibuild.^Cloud$.4: 'description_path' is a required property
Failed validation in kiwibuild.^Cloud$.5: 'description_scm' is a required property
Failed validation in kiwibuild.^Cloud$.5: 'description_path' is a required property
Failed validation in kiwibuild.^Container$.0: 'description_scm' is a required property
Failed validation in kiwibuild.^Container$.0: 'description_path' is a required property
Failed validation in kiwibuild.^Container$.1: 'description_scm' is a required property
Failed validation in kiwibuild.^Container$.1: 'description_path' is a required property
Failed validation in kiwibuild.^Container$.2: 'description_scm' is a required property
Failed validation in kiwibuild.^Container$.2: 'description_path' is a required property
WARNING: Unrecognized config option: global_description_scm. Did you mean global_version?
WARNING: Unrecognized config option: global_description_path.

rebased onto 9996561

10 months ago

kevin commented 10 months ago

ok, that fixed that. Lets gooooo...

Pull-Request has been merged by kevin

10 months ago

ngompa commented 10 months ago

:tada:

adamwill commented 10 months ago

okay, so good news, we got a compose, bad news, all the kiwibuilds failed. This means we have two problems: one, all the kiwibuilds are failing, two, the compose isn't dying when that happens (like it should).

The kiwibuilds are dying because their "package names" (remember, everything's a package so far as Koji is concerned) aren't listed somewhere they should be: "BuildError: package (image) Fedora-Cloud-Base-AmazonEC2 not in list for tag f41-updates-candidate" - see https://koji.fedoraproject.org/koji/taskinfo?taskID=114675502 . I presume we have some mechanism somewhere for adding these notional "package names" to the appropriate tags, I'll have to find out the 'package name' for imagefactory builds and grep around for it.

The compose isn't failing (as it should, because some of the images aren't supposed to be "failable") because the kiwibuild phase of pungi doesn't handle "failability" right. I'll file a Pungi issue on that.

adamwill commented 10 months ago

https://pagure.io/pungi/pull-request/1738 should fix the failability issue, I hope.

adamwill commented 10 months ago

So the "package name" for ImageFactory cloud base images is "Fedora-Cloud-Base", but I can't yet find anywhere in an obvious repo where we script adding that package name to Koji tags. Is this maybe just done on an ad hoc basis and we need @kevin or someone to do it for all the Kiwi "package names"? I think the list is:

Fedora-Cloud-Base-Generic
Fedora-Cloud-Base-AmazonEC2
Fedora-Cloud-Base-GCE
Fedora-Cloud-Base-Vagrant-libvirt
Fedora-Cloud-Base-Vagrant-VirtualBox
Fedora-Container-Base-Generic
Fedora-Container-Base-Generic-Minimal

Or we could, I guess, find out what names ImageFactory used for all of those and get Kiwi to use the same ones.

There are two cases where Kiwi used the same name as ImageFactory - Fedora-Container-Toolbox and Fedora-Cloud-Base-Azure . In those cases, there was yet a different failure: we tried to do dnf5 group install kiwi-build and it failed.

DEBUG package_manager.py:289:  ['/usr/bin/dnf5', '--installroot', '/var/lib/mock/f41-build-49556467-5901287/root/', 'group', 'install', 'kiwi-build', '--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes']
DEBUG util.py:636:  child environment: None
DEBUG util.py:553:  Using nspawn with args ['--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.aqfgvrhs:/etc/resolv.conf']
DEBUG util.py:558:  Executing command: ['/usr/bin/systemd-nspawn', '-q', '-M', '5df3ab93122943c1ae84aebc3e29ec84', '-D', '/var/lib/mock/f41-build-49556467-5901287-bootstrap/root', '-a', '--capability=cap_ipc_lock', '--bind=/tmp/mock-resolv.aqfgvrhs:/etc/resolv.conf', '--console=pipe', '--setenv=TERM=vt100', '--setenv=SHELL=/bin/bash', '--setenv=HOME=/var/lib/mock/f41-build-49556467-5901287/root/installation-homedir', '--setenv=HOSTNAME=mock', '--setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin', '--setenv=PROMPT_COMMAND=printf "\\033]0;<mock-chroot>\\007"', '--setenv=PS1=<mock-chroot> \\s-\\v\\$ ', '--setenv=LANG=C.UTF-8', '--setenv=LC_MESSAGES=C.UTF-8', '--resolv-conf=off', '/usr/bin/dnf5', '--installroot', '/var/lib/mock/f41-build-49556467-5901287/root/', 'group', 'install', 'kiwi-build', '--setopt=deltarpm=False', '--setopt=allow_vendor_change=yes', '--setopt=tsflags=nocontexts'] with env {'TERM': 'vt100', 'SHELL': '/bin/bash', 'HOME': '/var/lib/mock/f41-build-49556467-5901287/root/installation-homedir', 'HOSTNAME': 'mock', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'PROMPT_COMMAND': 'printf "\\033]0;<mock-chroot>\\007"', 'PS1': '<mock-chroot> \\s-\\v\\$ ', 'LANG': 'C.UTF-8', 'LC_MESSAGES': 'C.UTF-8', 'SYSTEMD_NSPAWN_TMPFS_TMP': '0', 'SYSTEMD_SECCOMP': '0'} and shell False
DEBUG util.py:463:  Updating and loading repositories:
DEBUG util.py:463:   build                                  100% |  73.0 MiB/s |  17.2 MiB |  00m00s
DEBUG util.py:463:  Repositories loaded.
DEBUG util.py:461:  Failed to resolve the transaction:
DEBUG util.py:461:  No match for argument: kiwi-build
DEBUG util.py:610:  Child return code was: 1

looks like the 'build' repo we are using does not have that package group in it.

adamwill commented 10 months ago

Kiwi                                        ImageFactory

Fedora-Cloud-Base-Generic                   Fedora-Cloud-Base
Fedora-Cloud-Base-AmazonEC2                 Fedora-Cloud-Base [0]
Fedora-Cloud-Base-GCE                       Fedora-Cloud-Base-GCP
Fedora-Cloud-Base-Vagrant-libvirt           Fedora-Cloud-Base-Vagrant
Fedora-Cloud-Base-Vagrant-VirtualBox        Fedora-Cloud-Base-Vagrant [1]
Fedora-Cloud-Base-Azure                     N/A [2]
Fedora-Container-Base-Generic               Fedora-Container-Base
Fedora-Container-Base-Generic-Minimal       Fedora-Container-Minimal-Base

so we can't reuse the ImageFactory names perfectly, but we could get closer if we wanted to...

[0] The ImageFactory Fedora-Cloud-Base task builds both .qcow2 and .raw.xz images. For Kiwi, Fedora-Cloud-Base-Generic builds .qcow2, Fedora-Cloud-Base-AmazonEC2 builds .raw.xz (I think)
[1] The ImageFactory Fedora-Cloud-Base-Vagrant task builds both libvirt and VirtualBox images.
[2] We don't build an Azure image with ImageFactory.

adamwill commented 10 months ago

sorry, I was wrong about Fedora-Cloud-Base-Azure in the match list above, it should be the same name for ImageFactory, as I said in the previous comment. Can't edit comments on this repo for some reason.

kevin commented 10 months ago

I'm fine just adding the new names... unless that will cause problems anywhere looking for/using them?

ngompa commented 10 months ago

I'd prefer to add the new names if we can.

kevin commented 10 months ago

Yeah, just done that. We can adjust later if there's any problems doing that...

adamwill commented 10 months ago

So I think we're finally done cleaning up all the consequences of this. Well. It might be good to compare package manifests before and after the switch for each image, I guess.

adamwill commented 10 months ago

@jcline did that for container_base at least (thanks jcline!) and the results are at https://pagure.io/fedora-kiwi-descriptions/pull-request/29 . We should do the same for container base minimal, toolbox, and cloud images, I guess.

jcline commented 10 months ago

@jcline did that for container_base at least (thanks jcline!) and the results are at https://pagure.io/fedora-kiwi-descriptions/pull-request/29 . We should do the same for container base minimal, toolbox, and cloud images, I guess.

That should also cover the minimal image, and I can take a look at the toolbox and cloud images next.

adamwill commented 10 months ago

were you able to do those comparisons, @jcline ?

jcline commented 10 months ago

Ah, whoops, forgot to update. The toolbox looked very similar, a few unnecessary libraries went away. I haven’t looked at the Amazon, Google, or vagrant box yet.