#11180 scripts: Add sync-ostree-base-containers.sh
Merged a year ago by kevin. Opened a year ago by walters.
walters/releng sync-ostree-container  into  main

@@ -0,0 +1,123 @@ 

+ #!/bin/bash

+ 

+ set -euo pipefail

+ 

+ # Poll the Fedora OSTree repository and convert the silverblue and kinoite refs

+ # to container images.

+ 

+ # Execute a command verbosely, i.e. echoing its arguments to stderr

+ runv () {

+     ( set -x ; $@ )

+ }

+ 

+ fedora_version=${1}

+ shift

+ 

+ ARCHES=("x86_64" "aarch64" "ppc64le")

+ # This is the release of Fedora that is currently stable, it will define if we

+ # need to move the fedora:latest tag

+ current_stable="37"

+ # Define what is rawhide so we know to push that tag

+ current_rawhide="38"

+ 

+ ostree_base_images=(silverblue kinoite)

+ 

+ if [[ ${fedora_version} -eq "$current_stable" ]]; then

+     tagname="latest"

+ fi

+ if [[ ${fedora_version} -eq "$current_rawhide" ]]; then

+     tagname="rawhide"

+ fi

+ if [[ -z "${REGISTRY_STAGE:-}" ]]; then

+     registries=("registry.fedoraproject.org" "candidate-registry.fedoraproject.org" "quay.io/fedora")

+ else

+     registries=("registry.stg.fedoraproject.org" "candidate-registry.stg.fedoraproject.org")

+ fi

+ canonical_registry=${registries[-1]}

+ secondary_registries=(${registries[@]})

+ unset secondary_registries[-1]

+ 

+ arch_to_goarch() {

+     local a=$1; shift

+     # See https://github.com/coreos/stream-metadata-go/blob/c5fe1b98ac1b1e6ab62a606b7580dc1f30703f83/arch/arch.go#L14

+     case "$a" in

+         aarch64) echo arm64 ;;

+         x86_64) echo amd64 ;;

+         *) echo $a ;;

+     esac

+ }

+ 

+ for name in "${ostree_base_images[@]}"; do

+     echo "Processing: ${name}"

+     imgname=fedora-"${name}"

+     imgtag=${imgname}:${fedora_version}

+     primary_imgtag=${canonical_registry}/${imgtag}

+     work_dir=$(mktemp -d -p /var/tmp --suffix=.sync-ostree)

+     pushd ${work_dir} &> /dev/null

+     # Note that this command will use the current architecture

+     arch=$(arch)

+     if ! runv skopeo inspect -n docker://${primary_imgtag} > inspect.json; then

+         echo "Failed to invoke skopeo, assuming container does not exist"

+     fi

+     container_commit=

+     if test -s inspect.json; then

+         container_commit=$(jq -r '.Labels["ostree.commit"]' inspect.json)

+     fi

+     mkdir repo

+     ostree --repo=repo init --mode=bare-user

+     echo fsync=0 >> repo/config

+     if test -f /etc/ostree/remotes.d/fedora.conf; then

+         cat /etc/ostree/remotes.d/fedora.conf >> repo/config

+     else

+         cat >> repo/config <<'EOF'

+ [remote "fedora"]

+ url=https://ostree.fedoraproject.org

+ gpg-verify=true

+ gpgkeypath=/etc/pki/rpm-gpg/

+ contenturl=mirrorlist=https://ostree.fedoraproject.org/mirrorlist

+ EOF

+     fi

+     ostree_ref=fedora:fedora/${fedora_version}/${arch}/${name}

+     runv ostree --repo=repo pull --commit-metadata-only $ostree_ref

+     current_commit=$(ostree --repo=repo rev-parse $ostree_ref)

+     echo "current ${ostree_ref}: $current_commit"

+     echo "current ${arch} container image ${imgtag}: $container_commit"

+     # IMPORTANT: We want to no-op if the sync is already done, otherwise we'll end up

+     # regenerating the image, which can cause people to apply pointless updates.

+     # This logic here is a bit hacky because we're only querying the current architecture

+     # which is likely to be x86_64 - i.e. we will ignore any updates that only affect aarch64

+     # for example.

+     if test "$current_commit" == "$container_commit"; then

+         echo "Generated container image for arch=$arch is up to date at commit $current_commit"

+         continue

+     fi

+ 

+     buildah rmi "${imgtag}" &>/dev/null || true

+     runv buildah manifest create "${imgtag}"

+ 

+     # Download the ostree commit for each architecture, and turn them into container images,

+     # then push to the registry

+     for arch in "${ARCHES[@]}"; do

+         ostree_ref=fedora:fedora/${fedora_version}/${arch}/${name}

+         imgtag_arch=oci:${imgtag}-${arch}

+         # To optimize local testing, set REPO_CACHE to a path to a repository which has

+         # pre-pulled the ostree data.

+         runv ostree --repo=repo pull "${REPO_CACHE:+-L ${REPO_CACHE}}" $ostree_ref

+         rev=$(ostree --repo=repo rev-parse $ostree_ref)

+         runv rpm-ostree compose container-encapsulate --repo repo $rev ${imgtag_arch}

+         runv buildah manifest add --arch=$(arch_to_goarch ${arch}) "${imgtag}" ${imgtag_arch}

+     done

+ 

+     target=docker://${primary_imgtag}

+     echo "Pushing to ${target}"

+     runv buildah manifest push --all "${imgtag}" ${target}

+     echo "Copying to secondary registries"

+     for registry in "${secondary_registries[@]}"; do

+         runv skopeo copy ${target} docker://${registry}/${imgname}

+     done

+     echo "done synchronizing ${name}"

+     popd

+     rm "${work_dir}" -rf

+ done

+ 

+ echo "done"

This new script implements https://pagure.io/releng/issue/11047
for Fedora Silverblue and Kinoite. Fedora CoreOS already
implements this logic in a different way (and one that is
integrated with automated testing). Fedora IoT has not requested
this yet, but it would be easy to add.

integrated with automated testing). Fedora IoT has not requested
this yet, but it would be easy to add.

We are not intending to add it yet, we don't believe it to be ready for our use case and the feature owners haven't bought it to the SIG/meetings to begin to socialise it either.

rebased onto 31e4789

a year ago

Sorry for the delay here. This looks reasonable to me... we should also call it from the nightly.sh script to make it run after every rawhide compose right?

although we should be able to test it manually also right?

lets go ahead and merge it tho and we can improve from there.

Pull-Request has been merged by kevin

a year ago

we should also call it from the nightly.sh script to make it run after every rawhide compose right?

Right, that's my understanding.

although we should be able to test it manually also right?

Yep! Anything I can do to help with that please let me know!

Just going to add a cookie/note to myself (and others) here that we'll want to change this script to use https://github.com/coreos/rpm-ostree/issues/4247 once it's ready - i.e. pass in the prior build

@humaton can you add this to rawhide / nightly over in pungi-fedora?

Anything I can do to help this along?

It should be live now. It was added to the end of the rawhide compose.

See https://lists.fedoraproject.org/archives/list/releng-cron@lists.fedoraproject.org/thread/K3TG5JREOUBBTC32LKTGLXNRUDHSDMGR/ for todays for example.

Can you see if the contents all look right?

It failed, but for a reason that I think may be rawhide-specific:

  • ostree --repo=repo pull --commit-metadata-only fedora:fedora/39/x86_64/silverblue
    error: No such branch 'fedora/39/x86_64/silverblue' in repository summary

It's fedora/rawhide/x86_64/silverblue, not fedora/39/x86_64/silverblue. Now I get a little uncertain how the aliasing is set up here. And honestly this is exactly an example of the problem I'm trying to get away from...on the order of 100 people could confidently talk about ostree ref aliases, but 10,000,000 could talk about how to run skopeo tag to alias images etc.

Yeah. We deliberate moved away from the number for rawhide a while back because it caused other issues.

But no good deed goes unpunished. ;)

Merging your pr... thanks!

OK next problem is ./sync-ostree-base-containers.sh: line 10: rpm-ostree: command not found...not sure where to fix that, somewhere in the ansible code?

ok. Should be fixed tomorrow.

Cool, we got farther in https://lists.fedoraproject.org/archives/list/releng-cron@lists.fedoraproject.org/thread/MSBB2WE6PLNNBBMFNIONICQJQ7YWD74Z/

Failure is now:

Error: copying image 1/3 from manifest list: trying to reuse blob
sha256:b77d5051829d56431cb68a5481139cd07ae85f771f0c592151c4e5d7c79392e2 at destination:
checking whether a blob
sha256:b77d5051829d56431cb68a5481139cd07ae85f771f0c592151c4e5d7c79392e2 exists in
quay.io/fedora/fedora-silverblue: authentication required

Hmm. We should be honoring the standard paths for container authfiles. One issue here is quay.io always gives a 403 and not 404 to avoid info leaks.

I suspect that these repositories (quay.io/fedora/fedora-silverblue and quay.io/fedora/fedora-kinoite) may need to be explicitly created on quay (whether via the UI or API) and have whatever bot account is doing the push granted permission to them.

Is there something that needs to be set up on the Fedora quay account to initialize these repositories and allow the account to push to them?

I don't know that I have access to that off hand. ;(

Could you file a releng ticket and I will track down who can fix it?

Possibly @cverna ? or @humaton might have it.

Metadata