From a1548d2aacedb1809ebfdeb45e46f6f53faf1b18 Mon Sep 17 00:00:00 2001 From: Jakub Kadlčík Date: Mar 12 2019 08:14:42 +0000 Subject: [PATCH 1/16] [doc] run copr-frontend-cloud.yml playbook instead of copr-frontend.yml --- diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst index d9fa113..c659038 100644 --- a/doc/how_to_release_copr.rst +++ b/doc/how_to_release_copr.rst @@ -35,7 +35,7 @@ Check that .repo files correctly points to copr-dev. And run on batcave01.phx2.f sudo rbac-playbook -l copr-keygen-dev.cloud.fedoraproject.org groups/copr-keygen.yml - sudo rbac-playbook -l copr-fe-dev.cloud.fedoraproject.org groups/copr-frontend.yml + sudo rbac-playbook -l copr-fe-dev.cloud.fedoraproject.org groups/copr-frontend-cloud.yml sudo rbac-playbook -l copr-dist-git-dev.fedorainfracloud.org groups/copr-dist-git.yml @@ -149,7 +149,7 @@ Run on batcave01.phx2.fedoraproject.org (if you do not have account there ask Mi sudo rbac-playbook -l copr-keygen.cloud.fedoraproject.org groups/copr-keygen.yml - sudo rbac-playbook -l copr-fe.cloud.fedoraproject.org groups/copr-frontend.yml + sudo rbac-playbook -l copr-fe.cloud.fedoraproject.org groups/copr-frontend-cloud.yml sudo rbac-playbook -l copr-dist-git.fedorainfracloud.org groups/copr-dist-git.yml From eebbda637ff01b69aa089cb14d6b7e6c1db5c308 Mon Sep 17 00:00:00 2001 From: Jakub Kadlčík Date: Mar 12 2019 08:14:42 +0000 Subject: [PATCH 2/16] [doc] remove trailing whitespace --- diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst index c659038..335bfe2 100644 --- a/doc/how_to_release_copr.rst +++ b/doc/how_to_release_copr.rst @@ -52,7 +52,7 @@ Call for QA Move MODIFIED+ bugzillas to ON_QA. -Ask people to test, verify bugs, and generally help with QA. They will ignore it but you will feel good about giving them a chance. +Ask people to test, verify bugs, and generally help with QA. They will ignore it but you will feel good about giving them a chance. Test ---- From 9b6bfae38b582bacb312763034a0c571759083cc Mon Sep 17 00:00:00 2001 From: Jakub Kadlčík Date: Mar 12 2019 08:14:42 +0000 Subject: [PATCH 3/16] [doc] show link to bodhi --- diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst index 335bfe2..422a016 100644 --- a/doc/how_to_release_copr.rst +++ b/doc/how_to_release_copr.rst @@ -116,7 +116,7 @@ Run:: rpkg --path common srpm --outdir /tmp/rpkg releng/fedora-release git /tmp/rpkg/python-copr-common*.src.rpm -And create erratas in Bodhi. +And create erratas in `Bodhi `_. Generate documentation ---------------------- From 63d5f108cd8f4ab7cfcefb93a4752ce5266fe294 Mon Sep 17 00:00:00 2001 From: Jakub Kadlčík Date: Mar 12 2019 08:14:42 +0000 Subject: [PATCH 4/16] [doc] the correct file is releasers.ini not releasers.conf --- diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst index 422a016..204a1c6 100644 --- a/doc/how_to_release_copr.rst +++ b/doc/how_to_release_copr.rst @@ -81,7 +81,7 @@ If you cannot run that, tell somebody with access to run that (msuchy has access Release package to Fedora ------------------------- -Make sure that ``releng/releasers.conf`` has up to date list of branches. +Make sure that ``releng/releasers.ini`` has up to date list of branches. Make sure you are co-maintainer of those packages in Fedora. From aa4e67ef33037bf98a726173e6ecc18877275922 Mon Sep 17 00:00:00 2001 From: Jakub Kadlčík Date: Mar 12 2019 08:14:42 +0000 Subject: [PATCH 5/16] [doc] alembic is run from playbooks --- diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst index 204a1c6..1816c42 100644 --- a/doc/how_to_release_copr.rst +++ b/doc/how_to_release_copr.rst @@ -153,7 +153,7 @@ Run on batcave01.phx2.fedoraproject.org (if you do not have account there ask Mi sudo rbac-playbook -l copr-dist-git.fedorainfracloud.org groups/copr-dist-git.yml -Note: if need run manually DB upgrade. +Note: You shouldn't need to upgrade DB manually, playbook covers it. Test production machine ----------------------- From 90a2072a71bfeea69729400f51f3aa4546809d36 Mon Sep 17 00:00:00 2001 From: Jakub Kadlčík Date: Mar 12 2019 08:14:42 +0000 Subject: [PATCH 6/16] [doc] remove ugly links to bugzilla Those links are kilometers long and ubreakable. The page layout can't handle it and adds horizontal scrollbar, so they can float infinitely to the right. --- diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst index 1816c42..e38a868 100644 --- a/doc/how_to_release_copr.rst +++ b/doc/how_to_release_copr.rst @@ -168,12 +168,12 @@ Send email to copr-dev mailing list. If there is some important change you can s Post release ------------ -Check if the MODIFIED bugs (that are not ON_QA) are fixed in released Copr or not, move them ON_QA if they are: +Check if the `MODIFIED bugs `_ +(that are not ON_QA) are fixed in released Copr or not, move them ON_QA. -https://bugzilla.redhat.com/buglist.cgi?bug_status=POST&bug_status=MODIFIED&classification=Community&list_id=4678039&product=Copr&query_format=advanced -Change status of all ON_DEV, ON_QA, VERIFIED, and RELEASE_PENDING bugs to CLOSED/CURRENTRELEASE with comment like 'New Copr has been released.': +Change status of all `ON_DEV, ON_QA, VERIFIED, and RELEASE_PENDING bugs `_ +to CLOSED/CURRENTRELEASE with comment like 'New Copr has been released.' -https://bugzilla.redhat.com/buglist.cgi?bug_status=ON_QA&bug_status=VERIFIED&bug_status=RELEASE_PENDING&classification=Community&list_id=4678045&product=Copr&query_format=advanced Fix this document to make it easy for the release nanny of the next release to use it. From 82676d1672ffbd6b88c9467d7ccaf25495e5f32d Mon Sep 17 00:00:00 2001 From: Jakub Kadlčík Date: Mar 12 2019 08:14:42 +0000 Subject: [PATCH 7/16] [doc] describe how to generate the general documentation --- diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst index e38a868..c6146ab 100644 --- a/doc/how_to_release_copr.rst +++ b/doc/how_to_release_copr.rst @@ -121,7 +121,14 @@ And create erratas in `Bodhi `_. Generate documentation ---------------------- -Go to: +Generate `Copr project documentation `_ + +:: + + cd doc + ./update_docs.sh + +Generate package specific documentation by going to: * https://readthedocs.org/projects/copr-rest-api/ @@ -131,7 +138,7 @@ Go to: * https://readthedocs.org/projects/python-copr/ -And hit "Build" button for each of those projects. +And hitting "Build" button for each of those projects. If schema was modified you should generate new Schema documentation. From 027964d73162c183d8a135902e2d585a4aedd509 Mon Sep 17 00:00:00 2001 From: Jakub Kadlčík Date: Mar 12 2019 08:14:42 +0000 Subject: [PATCH 8/16] [doc] describe how to use new version of copr-rpmbuild on builders --- diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst index c6146ab..2ad11e9 100644 --- a/doc/how_to_release_copr.rst +++ b/doc/how_to_release_copr.rst @@ -46,6 +46,13 @@ Note: if need manually run DB upgrade on frontend:: alembic upgrade head +Note: If there is a new version of copr-rpmbuild, run this on backend:: + + redis-cli + 127.0.0.1:6379> FLUSHALL + cleanup_vm_nova.py + systemctl restart copr-backend + Call for QA ----------- @@ -162,6 +169,8 @@ Run on batcave01.phx2.fedoraproject.org (if you do not have account there ask Mi Note: You shouldn't need to upgrade DB manually, playbook covers it. +Note: If there is a new version of copr-rpmbuild, throw away all builders and reprovision them. + Test production machine ----------------------- From 35dec7d12e34614d6683ed8a0c5126122da9c556 Mon Sep 17 00:00:00 2001 From: Jakub Kadlčík Date: Mar 12 2019 08:14:43 +0000 Subject: [PATCH 9/16] [doc] advise to stop copr-backend.service before running playbooks --- diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst index 2ad11e9..720f3b8 100644 --- a/doc/how_to_release_copr.rst +++ b/doc/how_to_release_copr.rst @@ -157,6 +157,9 @@ Go over bugs, which were resolved. Write some nice announce. Upgrade production machines --------------------------- +It is advised to stop ``copr-backend.service`` before upgrading production machines to avoid failing +builds due to temporarily having installed incompatible versions of Copr packages. + Run on batcave01.phx2.fedoraproject.org (if you do not have account there ask Mirek or somebody from fedora-infra):: sudo rbac-playbook -l copr-be.cloud.fedoraproject.org groups/copr-backend.yml From a640c3f1c93cd6767b9a288ab34d3a56beedd860 Mon Sep 17 00:00:00 2001 From: Jakub Kadlčík Date: Mar 12 2019 08:14:43 +0000 Subject: [PATCH 10/16] [doc] build also copr-rpmbuild in koji --- diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst index 720f3b8..3898baa 100644 --- a/doc/how_to_release_copr.rst +++ b/doc/how_to_release_copr.rst @@ -123,6 +123,9 @@ Run:: rpkg --path common srpm --outdir /tmp/rpkg releng/fedora-release git /tmp/rpkg/python-copr-common*.src.rpm + rpkg --path rpmbuild srpm --outdir /tmp/rpkg + releng/fedora-release git /tmp/rpkg/copr-rpmbuild*.src.rpm + And create erratas in `Bodhi `_. Generate documentation From b89e830711954cd9f448b06babf9f98b4cc2a989 Mon Sep 17 00:00:00 2001 From: Jakub Kadlčík Date: Mar 12 2019 08:14:43 +0000 Subject: [PATCH 11/16] [doc] checkout a tag before running rpkg srpm It is necessary because otherwise we get commit hash in the package filename. --- diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst index 3898baa..3f0bb6c 100644 --- a/doc/how_to_release_copr.rst +++ b/doc/how_to_release_copr.rst @@ -96,33 +96,43 @@ Run:: rm -r /tmp/rpkg + git checkout rpkg --path python srpm --outdir /tmp/rpkg releng/fedora-release git-all /tmp/rpkg/python-copr*.src.rpm + git checkout rpkg --path cli srpm --outdir /tmp/rpkg releng/fedora-release git-all /tmp/rpkg/copr-cli*.src.rpm + git checkout rpkg --path frontend srpm --outdir /tmp/rpkg releng/fedora-release git /tmp/rpkg/copr-frontend*.src.rpm + git checkout rpkg --path backend srpm --outdir /tmp/rpkg releng/fedora-release git /tmp/rpkg/copr-backend*.src.rpm + git checkout rpkg --path dist-git srpm --outdir /tmp/rpkg releng/fedora-release git /tmp/rpkg/copr-dist-git*.src.rpm + git checkout rpkg --path keygen srpm --outdir /tmp/rpkg releng/fedora-release git /tmp/rpkg/copr-keygen*.src.rpm + git checkout rpkg --path selinux srpm --outdir /tmp/rpkg releng/fedora-release git /tmp/rpkg/copr-selinux*.src.rpm + git checkout rpkg --path prunerepo srpm --outdir /tmp/rpkg releng/fedora-release git /tmp/rpkg/prunerepo*.src.rpm + git checkout rpkg --path common srpm --outdir /tmp/rpkg releng/fedora-release git /tmp/rpkg/python-copr-common*.src.rpm + git checkout rpkg --path rpmbuild srpm --outdir /tmp/rpkg releng/fedora-release git /tmp/rpkg/copr-rpmbuild*.src.rpm From 4967be7d587e096539c1c0785a59672876483e4e Mon Sep 17 00:00:00 2001 From: Jakub Kadlčík Date: Mar 12 2019 08:14:43 +0000 Subject: [PATCH 12/16] [doc] build dev packages into @copr/copr The purpose of @copr/copr-dev project is to build and test packages from commits, that are not yet merged into the `master` branch. Devel packages from the `master` branch should be built in the @copr/copr project. Which is the one, that should be used during release. --- diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst index 3f0bb6c..3e001b8 100644 --- a/doc/how_to_release_copr.rst +++ b/doc/how_to_release_copr.rst @@ -24,12 +24,12 @@ Build packages Build all packages:: - releng/build-packages @copr/copr-dev + releng/build-packages @copr/copr Upgrade -dev machines --------------------- -Check that .repo files correctly points to copr-dev. And run on batcave01.phx2.fedoraproject.org (if you do not have account there ask Mirek or somebody from fedora-infra):: +Check that .repo files correctly points to ``@copr/copr``. And run on batcave01.phx2.fedoraproject.org (if you do not have account there ask Mirek or somebody from fedora-infra):: sudo rbac-playbook -l copr-be-dev.cloud.fedoraproject.org groups/copr-backend.yml From 64c866adb61a311885c43d72b852e42320b10d45 Mon Sep 17 00:00:00 2001 From: Jakub Kadlčík Date: Mar 12 2019 08:14:43 +0000 Subject: [PATCH 13/16] [doc] we don't build our production packages in Copr anymore --- diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst index 3e001b8..ed9cd75 100644 --- a/doc/how_to_release_copr.rst +++ b/doc/how_to_release_copr.rst @@ -67,13 +67,6 @@ Test Run :ref:`beaker_tests` and check the results. -Build packages for production ------------------------------ - -Build all packages for @copr projects:: - - releng/build-packages @copr/copr - Release python-copr to PyPi --------------------------- @@ -85,8 +78,8 @@ Substitute `` with the current package version. If you cannot run that, tell somebody with access to run that (msuchy has access). -Release package to Fedora -------------------------- +Build packages for production +----------------------------- Make sure that ``releng/releasers.ini`` has up to date list of branches. From c9853cab20d15dc163dab218b2723827fc56058d Mon Sep 17 00:00:00 2001 From: Jakub Kadlčík Date: Mar 12 2019 08:14:43 +0000 Subject: [PATCH 14/16] [doc] describe how to schedule an outage --- diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst index ed9cd75..2da5c42 100644 --- a/doc/how_to_release_copr.rst +++ b/doc/how_to_release_copr.rst @@ -160,6 +160,13 @@ Prepare release notes Go over bugs, which were resolved. Write some nice announce. +Schedule outage +--------------- + +Schedule outage even if it has to happen in the next 5 minutes! + +Follow the instructions in `Outage SOP `_. + Upgrade production machines --------------------------- From 044b67b21fb5e7ddabff1f6d24aa30a2df1a99f9 Mon Sep 17 00:00:00 2001 From: Jakub Kadlčík Date: Mar 12 2019 08:14:43 +0000 Subject: [PATCH 15/16] [doc] add motivation for writting release notes --- diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst index 2da5c42..845f265 100644 --- a/doc/how_to_release_copr.rst +++ b/doc/how_to_release_copr.rst @@ -158,7 +158,10 @@ If schema was modified you should generate new Schema documentation. Prepare release notes --------------------- -Go over bugs, which were resolved. Write some nice announce. +Go over bugs, which were resolved. Write some nice announce. It is useful to prepare the release notes beforehand +because developers usualy don't remember what they worked on and therefore don't know what to test once +production instances are upgraded. Sharing the prepared notes with team members before doing the actuall release +is appreciated. Schedule outage --------------- From 8677b5d03eef0e1889fb3545a89d7b2284b9b9ad Mon Sep 17 00:00:00 2001 From: Jakub Kadlčík Date: Mar 12 2019 08:14:43 +0000 Subject: [PATCH 16/16] [doc] release from infra tags, not bodhi --- diff --git a/doc/how_to_release_copr.rst b/doc/how_to_release_copr.rst index 845f265..3abc032 100644 --- a/doc/how_to_release_copr.rst +++ b/doc/how_to_release_copr.rst @@ -78,6 +78,8 @@ Substitute `` with the current package version. If you cannot run that, tell somebody with access to run that (msuchy has access). +.. _build_packages_for_production: + Build packages for production ----------------------------- @@ -129,7 +131,9 @@ Run:: rpkg --path rpmbuild srpm --outdir /tmp/rpkg releng/fedora-release git /tmp/rpkg/copr-rpmbuild*.src.rpm -And create erratas in `Bodhi `_. +And submit them into `Infra tags repo `_. +Not even every fedora infra member can to this, ping clime or ask on ``#fedora-admin``. + Generate documentation ---------------------- @@ -195,6 +199,11 @@ Test production machine Just run some build and check if it succeeds. +Submit Bodhi updates +-------------------- + +Create updates in `Bodhi `_ for :ref:`every package built in Koji `. + Announce the release --------------------