#2 Enable two-week Atomic ISO builds until we get pungi4 up and running
Closed 5 years ago by ausil. Opened 5 years ago by maxamillion.
maxamillion/releng make-updates-atomic  into  master

file added
+102
@@ -0,0 +1,102 @@ 

+ #!/bin/bash

+ 

+ # Fedora Atomic related functions

+ #

+ # NOTE: This should not be run at face value, it is meant to be source'd

+ #       into other code and functions called accordingly.

+ #

+ #####

+ # Functions that "return" a value will do so in the __RETURN__ variable

+ #####

+ 

+ 

+ ##############################################################################

+ # function build-atomic

+ #   - build Fedora Atomic OSTree Commit, ISO and virt/cloud images

+ #   - In the future: pxe-to-live content

+ #

+ #   The required globally defined variables are:

+ #   DESTDIR     - Base directory for resulting build artifacts (for run-pungi)

+ #               - Also required for logs dir

+ #   FINALDEST   - Base directory for resulting build artifacts (for make-updates)

+ #

+ #   ATOMIC      - Location of the Atomic tdl, host json and rpm repo configs

+ #   ATOMICDEST  - Destination directory for the ostree tree builds

+ #   VERSION     - Fedora compose version (Example: 22_Beta_RC3)

+ #   SHORTCOMPOSE: a short reference to the compmose it is 1 for RC1 and TC1 for TC1

+ #   SHORTVERSION: is a shortened version of the Version passed in to the 

+ #                 control script (example: drops off _TCX)

+ 

+ function build-atomic(){

+     # This is kind of a dirty hack but because the directory structures stored

+     # in vars for make-updates vs run-pungi are pretty different, this is the

+     # least bad choice.

+     if [[ -n "$MAKEUPDATES" ]]; then

+         BUILDATOMIC_DIR="$FINALDEST"

+     else

+         BUILDATOMIC_DIR="$DESTDIR/$VERSION"

+     fi

+ 

+     # $DESTDIR gets bindmounted from the mock config for the compose actions.

+     # This allows log files to persist outside of individual builds/chroots

+     sg releng "mkdir -p $DESTDIR/logs"

+ 

+     pushd $ATOMIC

+         git reset --hard

+         git pull --rebase

+ 

+     sed -i -e 's|mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-22&arch=$basearch|baseurl=http://kojipkgs.fedoraproject.org/mash/branched/$basearch/os/|g' fedora*repo

+     mock -r fedora-branched-compose-x86_64 --init || exit 1

+     mock -r fedora-branched-compose-x86_64 --install rpm-ostree lorax || exit 1

+     mock -r fedora-branched-compose-x86_64 --shell "mkdir -p $BUILDATOMIC_DIR/Cloud_Atomic/x86_64/iso"

+     mock -r fedora-branched-compose-x86_64 --shell "mkdir -p $DESTDIR/logs/"

+ 

+     if [[ -n "$MAKEUPDATES" ]]; then

+         mock -r fedora-branched-compose-x86_64 --shell "lorax --nomacboot -p Fedora -v $SHORTVERSION -r $SHORTCOMPOSE \

+                                                         -s http://kojipkgs.fedoraproject.org/mash/branched/x86_64/os/ \

+                                                         -i fedora-productimg-atomic \

+                                                         -t Cloud_Atomic \

+                                                         --tmp /tmp/ \

+                                                         --logfile=$DESTDIR/logs/atomic \

+                                                         --add-template /srv/pungi/spin-kickstarts/atomic-installer/lorax-configure-repo.tmpl \

+                                                         --add-template-var=ostree_osname=fedora-atomic \

+                                                         --add-arch-template-var=ostree_repo=http://kojipkgs.fedoraproject.org/mash/atomic/$SHORTVERSION/ \

+                                                         --add-template-var=ostree_ref=fedora-atomic/f$SHORTVERSION/x86_64/docker-host \

+                                                         --add-arch-template /srv/pungi/spin-kickstarts/atomic-installer/lorax-embed-repo.tmpl \

+                                                         --add-arch-template-var=ostree_osname=fedora-atomic \

+                                                         --add-arch-template-var=ostree_ref=fedora-atomic/f$SHORTVERSION/x86_64/docker-host \

+                                                         $BUILDATOMIC_DIR/Cloud_Atomic/x86_64/os/ " || exit 1

+     else

+         # make sure we add the bleed repo to the json file

+         sed -i -e 's|\["fedora-22"\]|\["fedora-22", "fedora-bleed"\]|g' *json

+         mock -r fedora-branched-compose-x86_64  --shell "if [ ! -d $ATOMICDEST ]; then ostree init --repo=$ATOMICDEST --mode=archive-z2;fi"

+ 

+         #FIXME

+         # In the future, this is where we should invoke ostree static-delta

+         # generate or the like; track what rawhide is doing.

+         mock -r fedora-branched-compose-x86_64 --shell "rpm-ostree compose tree --repo=$ATOMICDEST $ATOMIC/fedora-atomic-docker-host.json" || exit 1

+ 

+         mock -r fedora-branched-compose-x86_64 --shell "lorax --nomacboot -p Fedora -v $SHORTVERSION -r $SHORTCOMPOSE \

+                                                         -s http://kojipkgs.fedoraproject.org/mash/branched/x86_64/os/ \

+                                                         -s http://kojipkgs.fedoraproject.org/mash/bleed/x86_64/ \

+                                                         -i fedora-productimg-atomic \

+                                                         -t Cloud_Atomic \

+                                                         --tmp /tmp/ \

+                                                         --logfile=$DESTDIR/logs/atomic \

+                                                         --add-template /srv/pungi/spin-kickstarts/atomic-installer/lorax-configure-repo.tmpl \

+                                                         --add-template-var=ostree_osname=fedora-atomic \

+                                                         --add-arch-template-var=ostree_repo=http://kojipkgs.fedoraproject.org/mash/atomic/22/ \

+                                                         --add-template-var=ostree_ref=fedora-atomic/f22/x86_64/docker-host \

+                                                         --add-arch-template /srv/pungi/spin-kickstarts/atomic-installer/lorax-embed-repo.tmpl \

+                                                         --add-arch-template-var=ostree_osname=fedora-atomic \

+                                                         --add-arch-template-var=ostree_ref=fedora-atomic/f22/x86_64/docker-host \

+                                                         $BUILDATOMIC_DIR/Cloud_Atomic/x86_64/os/ " || exit 1

+     fi

+ 

+     mock -r fedora-branched-compose-x86_64 --shell "cp -l $BUILDATOMIC_DIR/Cloud_Atomic/x86_64/os/images/boot.iso $BUILDATOMIC_DIR/Cloud_Atomic/x86_64/iso/fedora-atomic-x86_64-$VERSION.iso" || exit 1

+     mock -r fedora-branched-compose-x86_64 --shell "pushd $BUILDATOMIC_DIR/Cloud_Atomic/x86_64/iso/ ;sha256sum -b --tag *iso >fedora-atomic-$VERSION-x86_64-CHECKSUM; popd" || exit 1

+ 

+     #mock -r fedora-branched-compose-x86_64 --shell "livemedia-creator " || exit 1

+ 

+     popd

+ } # end build-atomic

file modified
+12 -1
@@ -44,9 +44,20 @@ 

  

  sg releng "mkdir -p $FINALDEST"

  pushd $FINALDEST/

- sg releng "mkdir -p Cloud-Images/i386/Images Cloud-Images/x86_64/Images Docker/x86_64"

+ sg releng "mkdir -p Cloud-Images/i386/Images Cloud-Images/x86_64/Images Docker/x86_64 Cloud_Atomic/x86_64"

  popd

  

+ ###

+ # BEGIN build-atomic

+ # build-atomic with current variable settings

+ ##

+ MAKEUPDATES="true" #just needs to be non-zero length string

+ . $FULLPATH/lib-atomic

+ build-atomic

+ 

+ ##

+ # END build-atomic

+ ##

  

  USER=$(whoami)

  # stage images

file modified
+6 -38
@@ -34,6 +34,8 @@ 

         MILESTONE=Final

  }

  

+ # $DESTDIR gets bindmounted from the mock config for the compose actions.

+ # This allows log files to persist outside of individual builds/chroots

  sg releng "mkdir -p $DESTDIR/logs"

  

  FULLPATH=$(pwd)
@@ -47,44 +49,10 @@ 

  $FULLPATH/build-arm-images "$SHORTVERSION" "$SHORTCOMPOSE" "$BRANCH"

  pushd

  

- pushd $ATOMIC

-  git reset --hard

-  git pull --rebase

-  echo '[fedora-bleed]

- name=fedora-bleed

- gpgcheck=1

- gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-22-$basearch

- enabled=1

- baseurl=http://kojipkgs.fedoraproject.org/mash/bleed/$basearch/' > fedora-bleed.repo

-  sed -i -e 's|mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-22&arch=$basearch|baseurl=http://kojipkgs.fedoraproject.org/mash/branched/$basearch/os/|g' fedora*repo

-  # make sure we add the bleed repo to the json file

-  sed -i -e 's|\["fedora-22"\]|\["fedora-22", "fedora-bleed"\]|g' *json

-  mock -r fedora-branched-compose-x86_64 --init || exit 1

-  mock -r fedora-branched-compose-x86_64 --install rpm-ostree lorax || exit 1

-  mock -r fedora-branched-compose-x86_64  --shell "if [ ! -d $ATOMICDEST ]; then ostree init --repo=$ATOMICDEST --mode=archive-z2;fi"

-  mock -r fedora-branched-compose-x86_64 --shell "rpm-ostree compose tree --repo=$ATOMICDEST $ATOMIC/fedora-atomic-docker-host.json" || exit 1

-  mock -r fedora-branched-compose-x86_64 --shell "mkdir -p $DESTDIR/$VERSION/Cloud_Atomic/x86_64/iso"

-  mock -r fedora-branched-compose-x86_64 --shell "lorax --nomacboot -p Fedora -v $SHORTVERSION -r $SHORTCOMPOSE \

-                                                  -s http://kojipkgs.fedoraproject.org/mash/branched/x86_64/os/ \

-                                                  -s http://kojipkgs.fedoraproject.org/mash/bleed/x86_64/ \

-                                                  -i fedora-productimg-atomic \

-                                                  -t Cloud_Atomic \

-                                                  --tmp /tmp/ \

-                                                  --logfile=$DESTDIR/logs/atomic \

-                                                  --add-template /srv/pungi/spin-kickstarts/atomic-installer/lorax-configure-repo.tmpl \

-                                                  --add-template-var=ostree_osname=fedora-atomic \

-                                                  --add-arch-template-var=ostree_repo=http://compose-x86-01.phx2.fedoraproject.org/compose/atomic/ \

-                                                  --add-template-var=ostree_ref=fedora-atomic/f22/x86_64/docker-host \

-                                                  --add-arch-template /srv/pungi/spin-kickstarts/atomic-installer/lorax-embed-repo.tmpl \

-                                                  --add-arch-template-var=ostree_osname=fedora-atomic \

-                                                  --add-arch-template-var=ostree_ref=fedora-atomic/f22/x86_64/docker-host \

-                                                  $DESTDIR/$VERSION/Cloud_Atomic/x86_64/os/ " || exit 1

-  mock -r fedora-branched-compose-x86_64 --shell "cp -l $DESTDIR/$VERSION/Cloud_Atomic/x86_64/os/images/boot.iso $DESTDIR/$VERSION/Cloud_Atomic/x86_64/iso/Fedora-Cloud_Atomic-x86_64-$VERSION.iso" || exit 1

-  mock -r fedora-branched-compose-x86_64 --shell "pushd $DESTDIR/$VERSION/Cloud_Atomic/x86_64/iso/ ;sha256sum -b --tag *iso >Fedora-Cloud_Atomic-$VERSION-x86_64-CHECKSUM; popd" || exit 1

- 

-  #mock -r fedora-branched-compose-x86_64 --shell "livemedia-creator " || exit 1

- 

- popd

+ ###

+ # build-atomic with current variable settings

+ . $FULLPATH/lib-atomic

+ build-atomic

  

  for arch in armhfp i386 x86_64

  do

no initial comment

Just a quick note, I've not tested this because I'm not exactly sure how best to do that or if I have permissions to run run-pungi.

I don't like submitting without testing ... really looking forward to getting stage koji in a state that we can actually test with it :)

I don't like submitting without testing ... really looking forward to getting stage koji in a state that we can actually test with it :)

random test of the "reply" pagure functionality

bleed repo is only used when doing TC and RC composes. it will not be used when doing updates composes

log lovcation is wrong. the logs go at a lower location. everything that goes in the directories made here ends up on the public mirrors

I did some refactoring and I believe I took care of the review points mentioned previously. Please let me know if there are other issues. Thanks :)

One of the reasons I made the config file JSON is so that it can be both generated and edited by programs sanely. For example:

https://github.com/cgwalters/playground/blob/master/scripts/rpmostree-set-repos.py

is a tiny Python program that could set this.

Can you add a stub line here:

In the future, this is where we should invoke ostree static-delta generate or the like; track what rawhide is doing.

Does the bleed repo still come into play after a release? I don't know.

OSTree commit, ISO and virt/cloud images

are what I would expect for deliverables.

In the future, PXE-to-Live.

This will likely need some change. since we will be using the updates atomic tree and not building a new atomic tree as part of the update compose. It is definitely an area that needs some improvement.

bleed repo is unused post release.

Updated with previous comments taken into account. There was one set of comments that I wasn't able to address and wanted to discuss further but it appears they were lost when I commit. The comment was around how bleed repo is involved and how we will handle updates post-release.

Another note was about the json configs, I left the sed lines in for now just because that's what's being done already. However, I think refactoring in a different pull request to remove the sed lines out of the rel-eng scripts would be a solid idea. I'll add it to my todo list.

Looking forward to more feedback, thank you.