#12014 container sync: try and handle different filenames with Kiwi
Merged 2 months ago by kevin. Opened 2 months ago by adamwill.
adamwill/releng update-container-sync-kiwi-2  into  main

@@ -66,14 +66,16 @@ 

      minname="Fedora-Container-Minimal-Base"

      format="docker-archive"

  fi

+ # this one is always the same

+ toolname="Fedora-Container-Toolbox"

  

  # Obtain the latest build

  #

  #   Need fXX-updates-canddiate to get actual latest nightly

  #

- build_name=$(koji -q latest-build --type=image f${1}-updates-candidate ${basename} | awk '{print $1}')

- minimal_build_name=$(koji -q latest-build --type=image f${1}-updates-candidate ${minname} | awk '{print $1}')

- toolbox_build_name=$(koji -q latest-build --type=image f${1}-updates-candidate Fedora-Container-Toolbox | awk '{print $1}')

+ base_build_nvr=$(koji -q latest-build --type=image f${1}-updates-candidate ${basename} | awk '{print $1}')

+ minimal_build_nvr=$(koji -q latest-build --type=image f${1}-updates-candidate ${minname} | awk '{print $1}')

+ toolbox_build_nvr=$(koji -q latest-build --type=image f${1}-updates-candidate ${toolname} | awk '{print $1}')

  

  

  if [[ ${1} -eq "$current_stable" ]]; then
@@ -119,64 +121,62 @@ 

      done

  }

  

- #

- # Version should not be higher than rawhide

- # Either there is a mistake or script is out of date

- #

- if [[ ${1} -gt "$current_rawhide" ]]; then

-     printf "ERROR: VERSION HIGHER THAN RAWHIDE"

-     exit 1

- fi

- 

- # For fedora-container-base

- if [[ -n ${build_name} ]]; then

+ find_and_copy_images() {

+     local nvr=$1

+     local registry_name=$2

+     local release=$3

+     local name

+     if [[ ${registry_name} = "fedora" ]]; then

+         name=${basename}

+     elif [[ ${registry_name} = "fedora-minimal" ]]; then

+         name=${minname}

+     elif [[ ${registry_name} = "fedora-toolbox" ]]; then

+         name=${toolname}

+     else

+         printf "Unexpected image type! If we added a new one, it needs mapping here"

+         exit 1

+     fi

      # Download the image

      work_dir=$(mktemp -d)

      pushd ${work_dir} &> /dev/null

-     koji download-build --type=image  ${build_name}

+     koji download-build --type=image  ${nvr}

      # Import the image

      for arch in "${ARCHES[@]}"; do

-         xz -d ${build_name}.${arch}.tar.xz

-         copy_image ${format}:${build_name}.${arch}.tar fedora:${1}-${arch}

+         local filename

+         if [[ ${release} -gt "39" ]]; then

+             filename=$(echo ${nvr} | sed -e "s,${name},${name}.${arch},g").oci.tar

+         else

+             filename=${nvr}.${arch}.tar

+         fi

+         xz -d ${filename}.xz

+         copy_image ${format}:${filename}.tar fedora:${registry_name}-${arch}

      done

  

      popd &> /dev/null

  

-     generate_manifest_list fedora ${1}

+     generate_manifest_list ${registry_name} ${release}

      printf "Removing temporary directory\n"

      rm -rf $work_dir

+ }

+ 

+ #

+ # Version should not be higher than rawhide

+ # Either there is a mistake or script is out of date

+ #

+ if [[ ${1} -gt "$current_rawhide" ]]; then

+     printf "ERROR: VERSION HIGHER THAN RAWHIDE"

+     exit 1

  fi

- # For fedora-container-minimal-base

- if [[ -n ${minimal_build_name} ]]; then

-     # Download the image

-     work_dir=$(mktemp -d)

-     pushd ${work_dir} &> /dev/null

-     koji download-build --type=image  ${minimal_build_name}

-     # Import the image

-     for arch in "${ARCHES[@]}"; do

-         xz -d ${minimal_build_name}.${arch}.tar.xz

-         copy_image ${format}:${minimal_build_name}.${arch}.tar fedora-minimal:${1}-${arch}

-     done

-     popd &> /dev/null

  

-     generate_manifest_list fedora-minimal ${1}

-     printf "Removing temporary directory\n"

-     rm -rf $work_dir

+ # For fedora-container-base

+ if [[ -n ${base_build_nvr} ]]; then

+     find_and_copy_images ${base_build_nvr} fedora ${1}

+ fi

+ # For fedora-container-minimal-base

+ if [[ -n ${minimal_build_nvr} ]]; then

+     find_and_copy_images ${minimal_build_nvr} fedora-minimal ${1}

  fi

  # For fedora-container-toolbox

- if [[ -n ${toolbox_build_name} ]]; then

-     # Download the image

-     work_dir=$(mktemp -d)

-     pushd ${work_dir} &> /dev/null

-     koji download-build --type=image  ${toolbox_build_name}

-     # Import the image

-     for arch in "${ARCHES[@]}"; do

-         xz -d ${toolbox_build_name}.${arch}.tar.xz

-         copy_image ${format}:${toolbox_build_name}.${arch}.tar fedora-toolbox:${1}-${arch}

-     done

-     popd &> /dev/null

- 

-     generate_manifest_list fedora-toolbox ${1}

-     printf "Removing temporary directory\n"

-     rm -rf $work_dir

+ if [[ -n ${toolbox_build_nvr} ]]; then

+     find_and_copy_images ${minimal_build_nvr} fedora-toolbox ${1}

  fi

Kiwi doesn't just result in different Koji NVRs for container
images, but the actual filenames differ too.

This problem really shows the limits of the current approach to
this; this script should really be fed the compose metadata and
use that to find images, and it should probably be a toddler,
and it should be in a real language, not shell. Fixing this in
shell script with the current 'poke the Koji task' approach is
somewhat painful and I'm not super confident this is 100% correct
(I will test it when I'm home on a regular internet connection),
but I wanted to get something done at least before diving into a
complete rewrite.

The reference jobs I tried to test this against are:
https://koji.fedoraproject.org/koji/buildinfo?buildID=2421063
https://koji.fedoraproject.org/koji/buildinfo?buildID=2421062
https://koji.fedoraproject.org/koji/buildinfo?buildID=2421079

From those, it looks like the filename format for Kiwi is more
or less (name).(arch)-(version)-(release).oci.tar.xz, and that's
what we try to use here, for releases higher than 39, keeping
the old (name)-(version)-(release).(arch).tar.xz format for older
releases. This is much more awkward in shell script than it would
be in a real language.

Signed-off-by: Adam Williamson awilliam@redhat.com

This seems right, but I also agree that this is awful and I have trouble understanding what's going on...

So... :thumbsup: ?

If I read this right, it should push the images as fedora(-minimal|-toolbox):(40|41) right?

Yep.

Looks mostly great, but... drop the '-i' from sed... it causes it to fail with no filename passed.

Ugh, right. I tried that at a console, found it didn't work, fixed it in the console and forgot to fix it in the script 😅 will fix once I'm off this plane.

rebased onto 8b95c97

2 months ago

Fixed that, and one other totally intentional mistake I put in there to see if you would catch it. That's my story and I'm sticking with it.

Looks ok now to me.

Ah yes, I didn't test the 'old' path there. Good catch.

I'll go ahead and merge this and run it for 40/41?

Pull-Request has been merged by kevin

2 months ago
Metadata