PR#955: doc: how_to_manage_chroots: branching howto scripted - copr/copr

copr / copr

#955 doc: how_to_manage_chroots: branching howto scripted

Merged 4 years ago by praiskup. Opened 4 years ago by praiskup.

Unknown source branching-scripted into master

frontend, doc: more scripted fedora branching

Pavel Raiskup • 4 years ago

1523678

doc/how_to_manage_chroots.rst

file modified

+36 -23

		`@@ -17,6 +17,7 @@`
		`copr-frontend create_chroot <name>`
		`copr-frontend alter_chroot --action activate <name>`
		`copr-frontend alter_chroot --action deactivate <name>`
		`+ copr-frontend branch_fedora <new-branched-version>`
		`copr-frontend rawhide_to_release <rawhide-chroot> <newly-created-chroot>`

		However, `enablement process upon Fedora branching <#branching-process>`_ and also
		`@@ -27,39 +28,51 @@`
		`Branching process`
		`-----------------`

		`- Immediately after Fedora branching, you want to do something like this`
		`+ Projects can be configured to follow Fedora branching. That means, that once a`
		`+ chroot for new Fedora release is enabled, it should be automatically turned-on`
		`+ for such projects. Moreover, builds from Rawhide should be forked into this new`
		`+ chroot.`

		`- ::`
		`-`
		`- copr-frontend create_chroot fedora-27-x86_64 fedora-27-i386 fedora-27-ppc64le`
		`+ So immediately after Fedora branching (for exmaple to version 31), you`
		`+ want to do this (the command takes a very long time, be prepared)::`

		`+ copr-frontend branch_fedora 31`

		- Copr uses `Mock <https://github.com/rpm-software-management/mock>`_ for building packages, so you should check if
		`- the mock configs`
		- `are already available <https://github.com/rpm-software-management/mock/tree/devel/mock-core-configs/etc/mock>`_
		- and in which version of the :code:`mock-core-configs` package they were added. If that version is not installed
		`- on builders, you should temporarily disable the chroots.`
		+ This command creates ``fedora-31-*`` chroots from corresponding
		+ ``fedora-rawhide-*`` chroots, and it also copies (duplicates/forks) latest
		`+ successful rawhide package builds into the new chroots. This can be done`
		`+ manually for each architecture by::`

		`- ::`
		`+ copr-frontend create_chroot fedora-31-x86_64 --deactivated`
		`+ copr-frontend rawhide_to_release fedora-rawhide-x86_64 fedora-31-x86_64`

		`- copr-frontend alter_chroot --action deactivate fedora-27-x86_64 fedora-27-i386 fedora-27-ppc64le`
		`+ From the manual steps you can see that the new chroots are deactivated at`
		`+ the beginning.`

		+ It's important to do ``rawhide_to_release`` as soon as possible, because right
		`+ after branching action - Fedora Rawhide starts to live it's own separate life -`
		`+ and the builds in Rawhide become more and more incompatible with the branched`
		`+ Fedora. So - if we copied the packages later - the branched chroot in copr`
		`+ could become unusable. You may consider sending an email to mailing list that`
		`+ rawhide packages were copied.`

		`- Projects can be configured to follow Fedora branching. That means, that once a chroot for new Fedora release is`
		`- enabled, it should be automatically turned-on for such projects. Moreover, builds from Rawhide should be forked into`
		`- this new chroot. It can be done like this`
		`+ The next needs to wait a bit, namely`
		`+ * till there's a working compose for the freshly branched Fedora, and`
		`+ * till the new mock configs are available on the builders.`

		`- ::`
		`-`
		`- copr-frontend rawhide_to_release fedora-rawhide-x86_64 fedora-27-x86_64`
		`- copr-frontend rawhide_to_release fedora-rawhide-i386 fedora-27-i386`
		`- copr-frontend rawhide_to_release fedora-rawhide-ppc64le fedora-27-ppc64le`
		`-`
		`- Once the mock configs are available on the builders, you can re-enable the chroots`
		+ Copr uses `Mock <https://github.com/rpm-software-management/mock>`_ for building packages, so you should check if
		`+ the mock configs`
		+ `are already available <https://github.com/rpm-software-management/mock/tree/devel/mock-core-configs/etc/mock>`_
		+ and in which version of the :code:`mock-core-configs` package they were added. If that version is not installed
		`+ on builders, you should keep the chroots deactivated for now and continue later.`

		`- ::`
		`+ But the sooner we can enable the new chroots, the better -- all the builds that`
		+ happened in the time window between ``rawhide_to_release`` and chroot enablement
		`+ will be missed in the branched chroot later (users will have to rebuild them`
		`+ manually). So as soon as it is possible, do::`

		`- copr-frontend alter_chroot --action activate fedora-27-x86_64 fedora-27-i386 fedora-27-ppc64le`
		`+ copr-frontend alter_chroot --action activate \`
		`+ fedora-31-x86_64 fedora-31-i386 fedora-31-ppc64le fedora-31-aarch64`

		When everything is done, `send an information email to a mailing list <#mailing-lists>`_.

frontend/coprs_frontend/commands/branch_fedora.py

file added

+42

		`@@ -0,0 +1,42 @@`
		`+ from flask_script import Command, Option`
		`+ from coprs.logic import coprs_logic`
		`+`
		`+ from commands.create_chroot import CreateChrootCommand`
		`+ from commands.rawhide_to_release import RawhideToReleaseCommand`
		`+`
		`+`
		`+ class BranchFedoraCommand(Command):`
		`+ """`
		`+ Branch fedora-rawhide-* chroots to fedora-N* and execute rawhide_to_release`
		`+ on them`
		`+ """`
		`+`
		`+ option_list = [`
		`+ Option("fedora_version",`
		`+ help="The version of Fedora to branch Rawhide into, e.g. 32",`
		`+ type=int),`
		`+ Option(`
		`+ "--dist-git-branch",`
		`+ "-b",`
		`+ dest="branch",`
		`+ help="Branch name for this set of new chroots"),`
		`+ ]`
		`+`
		`+ def run(self, fedora_version, branch=None):`
		`+ rawhide_chroots = coprs_logic.MockChrootsLogic.get_from_name(`
		`+ "fedora-rawhide",`
		`+ active_only=True,`
		`+ noarch=True).all()`
		`+`
		`+ chroot_pairs = {`
		`+ 'fedora-{}-{}'.format(fedora_version, rch.arch):`
		`+ 'fedora-rawhide-{}'.format(rch.arch)`
		`+ for rch in rawhide_chroots`
		`+ }`
		`+`
		`+ c_cmd = CreateChrootCommand()`
		`+ c_cmd.run(chroot_pairs.keys(), branch, True)`
		`+`
		`+ r2r_cmd = RawhideToReleaseCommand()`
		`+ for new_chroot, rawhide_chroot in chroot_pairs.items():`
		`+ r2r_cmd.run(rawhide_chroot, new_chroot)`

frontend/coprs_frontend/commands/create_chroot.py

file modified

+15 -7

		`@@ -30,13 +30,20 @@`
		`"Creates a mock chroot in DB"`

		`def __init__(self):`
		`- self.option_list += Option(`
		`- "--dist-git-branch",`
		`- "-b",`
		`- dest="branch",`
		`- help="Branch name for this set of new chroots"),`
		`-`
		`- def run(self, chroot_names, branch=None):`
		`+ self.option_list += (`
		`+ Option(`
		`+ "--dist-git-branch",`
		`+ "-b",`
		`+ dest="branch",`
		`+ help="Branch name for this set of new chroots"),`
		`+ Option(`
		`+ "--deactivated",`
		`+ action="store_true",`
		+ help="Activate the chroot later, manually by `alter_chroot`"
		`+ ),`
		`+ )`
		`+`
		`+ def run(self, chroot_names, branch=None, deactivated=False):`
		`for chroot_name in chroot_names:`
		`if not branch:`
		`branch = chroot_to_branch(chroot_name)`
		`@@ -44,6 +51,7 @@`
		`try:`
		`chroot = coprs_logic.MockChrootsLogic.add(chroot_name)`
		`chroot.distgit_branch = branch_object`
		`+ chroot.is_active = not deactivated`
		`db.session.commit()`
		`except exceptions.MalformedArgumentException:`
		`self.print_invalid_format(chroot_name)`

frontend/coprs_frontend/manage.py

file modified

		`@@ -23,6 +23,7 @@`
		`"alter_chroot": "AlterChrootCommand",`
		`"display_chroots": "DisplayChrootsCommand",`
		`"drop_chroot": "DropChrootCommand",`
		`+ "branch_fedora": "BranchFedoraCommand",`

		`# User commands`
		`"alter_user": "AlterUserCommand",`

no initial comment

praiskup commented on line 37 of doc/how_to_manage_chroots.rst 4 years ago

Btw., I'm running this now on stage, and it is running for 15 minutes for the first architecture now. Is this known problem?

praiskup commented on line 10 of doc/how_to_manage_chroots.rst 4 years ago

missing alter_chroot

rebased onto 62501db6dead88da8ad05a75d9810d998e77b42d

4 years ago

rebased onto 61c2c06c2924fd23fcf27b2229a6005516622765

4 years ago

rebased onto 73c2690ec4793e24e6c33f33ac22b40ae4ac61f6

4 years ago

frostyx commented 4 years ago

I really like the overall improvements for that article, they explain why we do things this way and put them into wider context. Thank you!

However, I am not particularly convinced by the scripting part. When reading documentation I would expect to read an explanation how to do things, not interpret scripts in my head. IMHO if those tasks are annoying/slow to run manually, we could write a proper script/manage.py command and suggest running it here, but personally I would avoid writing the script content directly in the documentation.

Especially I am paranoid about this line

set -- x86_64 i386 ppc64le aarch64 # all architectures are in "$@"

Maybe this won't happen but feels like using such construct on multiple places, it will affect each other ... so we can never be sure, what is in $@.

praiskup commented 4 years ago

Would architectures=(x86_64 i386 ppc64le aarch64), and later using ${architectures[@]} sound better to you?

praiskup commented 4 years ago

The thing is; I really felt like I have to script this -- otherwise I'd definitely forget about running something (especially taking into account there is multi-hour gap between particular commands).

frostyx commented 4 years ago

Would architectures=(x86_64 i386 ppc64le aarch64), and later using ${architectures[@]} sound better to you?

By far. But the rest of the comment still applies. Don't interpret it as a "hard no", but rather a personal preference.

The thing is; I really felt like I have to script this -- otherwise I'd definitely forget about running something (especially taking into account there is multi-hour gap between particular commands).

In this case, I am all for scripting it, but I would prefer to execute the script, not copy-pasting its content from the documentation.

praiskup commented 4 years ago

Ok, let's have a look how this can be better scripted.

Metadata Update from @praiskup:
- Pull-request tagged with: needs-work

4 years ago

rebased onto 8d3cc276dd01d9ed5ecfaf520228c3c42385a185

4 years ago

Metadata Update from @praiskup:
- Pull-request untagged with: needs-work

4 years ago

rebased onto bc191a9e4f1845e094af3b00e79d6e4f3d53d56c

4 years ago

praiskup commented 4 years ago