It would be super-useful to have an archive of all of the current specfiles. They're nice to have around as examples, and a5t least the FPC folks find themselves needing to grep through them pretty often. Keeping a local checkout of everything just for this (and keeping it updated) is kind of wasteful, and so is downloading the checkout seed.
Since pkgs already has a git checkout (in /srv/git_seed/git-checkout) and I believe it should be pretty easy to tar up all of the specfiles.
The script which generates the checkout seed is at https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/git/make_checkout_seed/files/make-git-checkout-seed.sh and it should be pretty easy to add a bit there to copy out all of the specfiles and tar them up.
fixed line-wraps
here is a patch for the script. kindly, test and report back.
{{{ diff --git a/roles/git/make_checkout_seed/files/make-git-checkout-seed.sh b/role s/git/make_checkout_seed/files/make-git-checkout-seed.sh index e873523..fbba669 100644 --- a/roles/git/make_checkout_seed/files/make-git-checkout-seed.sh +++ b/roles/git/make_checkout_seed/files/make-git-checkout-seed.sh @@ -3,6 +3,7 @@ # mkdir /srv/git_seed # cron job to invoke file daily # Need to setup OUTPUT_DIR to be served by apache +# modified by doteast
# Where the git repos live. These are bare repos ORIGIN_DIR=/srv/git/rpms @@ -12,12 +13,16 @@ WORK_DIR=/srv/git_seed # Subdirectory makes cleanup easier SEED_DIR=$WORK_DIR/git-checkout
+# subdirectory to collect rpm speciles +SPEC_DIR=$WORK_DIR/rpm-specs + # Where to store the seed tarball for download OUTPUT_DIR=/srv/cache/lookaside/
# Instead of starting fresh each time, we'll try to use git pull to keep things synced #rm -rf $WORK_DIR/* mkdir -p $SEED_DIR +mkdir -p $SPEC_DIR
# Give people an indication of when this checkout was last synced TIMESTAMP=date --rfc-3339=seconds @@ -33,15 +38,21 @@ for repo in $ORIGIN_DIR/.git ; do git pull --all &> /dev/null sed -i "s@url = .@url = git://pkgs.fedoraproject.org/$(basename $repo .git )@" $working_tree/.git/config popd &>/dev/null + cp $working_tree/$(basename $repo .git).spec $SPEC_DIR/ else pushd $SEED_DIR &>/dev/null git clone $repo &> /dev/null popd &>/dev/null sed -i "s@url = .*@url = git://pkgs.fedoraproject.org/$(basename $repo .git )@" $working_tree/.git/config + cp $working_tree/$(basename $repo .git).spec $SPEC_DIR/ fi done
date --rfc-3339=seconds
tar -cf - -C$WORK_DIR $(basename $SEED_DIR)|xz -2 > $OUTPUT_DIR/.git-seed-$DATE .tar.xz +tar -cf - -C$WORK_DIR $(basename $SPEC_DIR)|xz -2 > $OUTPUT_DIR/.rpm-specs-$DAT E.tar.xz rm $OUTPUT_DIR/git-seedtar.xz +rm $OUTPUT_DIR/rpm-specstar.xz mv $OUTPUT_DIR/.git-seed-$DATE.tar.xz $OUTPUT_DIR/git-seed-$DATE.tar.xz +mv $OUTPUT_DIR/.rpm-specs-$DATE.tar.xz $OUTPUT_DIR/rpm-specs-$DATE.tar.xz ln -s git-seed-$DATE.tar.xz $OUTPUT_DIR/git-seed-latest.tar.xz +ln -s rpm-specs-$DATE.tar.xz $OUTPUT_DIR/rpm-specs-latest.tar.xz
}}}
This looks like a good start, but will need to handle the case where the package has no .spec file (if it's dead.package or nothing at all).
Here is a patch which handles package has no .spec file.
{{{ --- make-git-checkout-seed.sh 2015-11-19 10:27:04.232135631 +0100 +++ test.sh 2015-11-19 10:33:08.938761005 +0100 @@ -3,6 +3,8 @@ # mkdir /srv/git_seed # cron job to invoke file daily # Need to setup OUTPUT_DIR to be served by apache +# modified by doteast +# modified by alda
# Where the git repos live. These are bare repos ORIGIN_DIR=/srv/git/rpms @@ -12,12 +14,16 @@ # Subdirectory makes cleanup easier SEED_DIR=$WORK_DIR/git-checkout
# Give people an indication of when this checkout was last synced TIMESTAMP=date --rfc-3339=seconds @@ -33,15 +39,25 @@ git pull --all &> /dev/null sed -i "s@url = .@url = git://pkgs.fedoraproject.org/$(basename $repo .git)@" $working_tree/.git/config popd &>/dev/null + if [ -e $working_tree/$(basename $repo .git).spec ]; then + cp $working_tree/$(basename $repo .git).spec $SPEC_DIR/ + fi else pushd $SEED_DIR &>/dev/null git clone $repo &> /dev/null popd &>/dev/null - sed -i "s@url = .@url = git://pkgs.fedoraproject.org/$(basename $repo .git)@" $working_tree/.git/config + sed -i "s@url = .*@url = git://pkgs.fedoraproject.org/$(basename $repo .git)@" $working_tree/.git/config + if [ -e $working_tree/$(basename $repo .git).spec ]; then + cp $working_tree/$(basename $repo .git).spec $SPEC_DIR/ + fi fi done
tar -cf - -C$WORK_DIR $(basename $SEED_DIR)|xz -2 > $OUTPUT_DIR/.git-seed-$DATE.tar.xz +tar -cf - -C$WORK_DIR $(basename $SPEC_DIR)|xz -2 > $OUTPUT_DIR/.rpm-specs-$DATE.tar.xz rm $OUTPUT_DIR/git-seedtar.xz +rm $OUTPUT_DIR/rpm-specstar.xz mv $OUTPUT_DIR/.git-seed-$DATE.tar.xz $OUTPUT_DIR/git-seed-$DATE.tar.xz +mv $OUTPUT_DIR/.rpm-specs-$DATE.tar.xz $OUTPUT_DIR/rpm-specs-$DATE.tar.xz ln -s git-seed-$DATE.tar.xz $OUTPUT_DIR/git-seed-latest.tar.xz +ln -s rpm-specs-$DATE.tar.xz $OUTPUT_DIR/rpm-specs-latest.tar.xz }}}
This looked good in stg, so I pushed it out to prod too. Should take effect tomorrow.
Thanks!
Seems to be working fine. Thanks!
Oops, found a bug. If a package gets dead.package'd, the spec sticks around in the spec tarball forever. For example, nspluginwrapper was retired on Nov. 27 but its spec is still in the archive.
{{{ diff --git a/roles/git/make_checkout_seed/files/make-git-checkout-seed.sh b/roles/git/make_checkout_seed/files/make-git-checkout-seed.sh index 26fd055..c4b76be 100644 --- a/roles/git/make_checkout_seed/files/make-git-checkout-seed.sh +++ b/roles/git/make_checkout_seed/files/make-git-checkout-seed.sh @@ -39,7 +39,11 @@ for repo in $ORIGIN_DIR/.git ; do git pull --all &> /dev/null sed -i "s@url = .@url = git://pkgs.fedoraproject.org/$(basename $repo .git)@" $working_tree/.git/config popd &>/dev/null - if [ -e $working_tree/$(basename $repo .git).spec ]; then + + # If the package is dead, we might need to delete an existing spec + if [ -e $working_tree/dead.package ]; then + rm -f $working_tree/$(basename $repo .git).spec + elif [ -e $working_tree/$(basename $repo .git).spec ]; then cp $working_tree/$(basename $repo .git).spec $SPEC_DIR/ fi else }}}
I think we also need to remove the spec file from the spec collection directory. also, unless bringing a package back from the dead entails removing dead.package from the repo, we can also skip processing directories having dead.package earlier in the loop (commented in the patch provided).
kindly, test and prompt for push
{{{
diff --git a/roles/git/make_checkout_seed/files/make-git-checkout-seed.sh b/roles/git/make_checkout_seed/files/make-git-checkout-seed.sh index 6419b48..f7c4eff 100644 --- a/roles/git/make_checkout_seed/files/make-git-checkout-seed.sh +++ b/roles/git/make_checkout_seed/files/make-git-checkout-seed.sh @@ -3,14 +3,13 @@ # mkdir /srv/git_seed # cron job to invoke file daily # Need to setup OUTPUT_DIR to be served by apache -# modified by doteast -# modified by alda
# Where the git repos live. These are bare repos ORIGIN_DIR=/srv/git/rpms
# Where we'll create the repos to tar up WORK_DIR=/srv/git_seed + # Subdirectory makes cleanup easier SEED_DIR=$WORK_DIR/git-checkout
@@ -32,25 +31,36 @@ echo "$TIMESTAMP" > $SEED_DIR/TIMESTAMP
for repo in $ORIGIN_DIR/.git ; do - working_tree=$SEED_DIR/$(basename $repo .git) + bname=$(basename $repo .git) + working_tree=$SEED_DIR/$bname +# uncomment to skip processing dead.package repos +# if [ -e $working_tree/dead.package ]; then +# continue +# fi if [ -d $working_tree ] ; then pushd $working_tree &> /dev/null sed -i "s@url = .@url = $repo@" $working_tree/.git/config git pull --all &> /dev/null - sed -i "s@url = .@url = git://pkgs.fedoraproject.org/$(basename $repo .git)@" $working_tree/.git/config + sed -i "s@url = .@url = git://pkgs.fedoraproject.org/$bname@" $working_tree/.git/config popd &>/dev/null - if [ -e $working_tree/$(basename $repo .git).spec ]; then - cp $working_tree/$(basename $repo .git).spec $SPEC_DIR/ + if [ -e $working_tree/dead.package ]; then + rm -f $working_tree/$bname.spec + rm -f $SPEC_DIR/$bname.spec + elif [ -e $working_tree/$bname.spec ]; then + cp $working_tree/$bname.spec $SPEC_DIR/ fi else pushd $SEED_DIR &>/dev/null git clone $repo &> /dev/null popd &>/dev/null - sed -i "s@url = .@url = git://pkgs.fedoraproject.org/$(basename $repo .git)@" $working_tree/.git/config - if [ -e $working_tree/$(basename $repo .git).spec ]; then - cp $working_tree/$(basename $repo .git).spec $SPEC_DIR/ - fi + sed -i "s@url = .@url = git://pkgs.fedoraproject.org/$bname@" $working_tree/.git/config + if [ -e $working_tree/dead.package ]; then + rm -f $working_tree/$bname.spec + rm -f $SPEC_DIR/$bname.spec + elif [ -e $working_tree/$bname.spec ]; then + cp $working_tree/$bname.spec $SPEC_DIR/ fi + fi done
tar -cf - -C$WORK_DIR $(basename $SEED_DIR)|xz -2 > $OUTPUT_DIR/.git-seed-$DATE.tar.xz
Trying this out in staging:
https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=401e9b0f6d4430e1b7b14c7b3d98bc89ba4b854c
It seems to be working fine (not breaking) in staging:
http://pkgs.stg.fedoraproject.org/lookaside/
But nspluginwrapper isn't dead.packaged in staging... so I couldn't quite verify the correctness.
So, since this wasn't working right in the first place in prod, I've moved it up to prod pkgs in this commit here:
https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=4971401cec6fa0e2d9f1483b7bf045fc2966b91f
We'll need to check on the result on February 23rd to see if nspluginwrapper is handled correctly.
OK, this looks good to me now. Thanks for all the work on this doteast, tibbs, and alda!
for internet searchers, the result lives here: https://pkgs.fedoraproject.org/repo/rpm-specs-latest.tar.xz
Login to comment on this ticket.