PR#874: [doc] kill all old builders when upgrading infrastructure - copr/copr

copr / copr

#874 [doc] kill all old builders when upgrading infrastructure

Merged 4 years ago by praiskup. Opened 4 years ago by praiskup.

Unknown source kill-all-vms into master

[doc] kill all old builders when upgrading infrastructure

Pavel Raiskup • 4 years ago

72f55da

doc/how_to_upgrade_persistent_instances.rst

file modified

+17 -1

		`@@ -70,9 +70,11 @@`
		`.. warning:: You might need to backup also letsencrypt certificates.`
		See `Letsencrypt renewal limits`_.

		`- .. warning:: backend - You should terminate existing resalloc resources.`
		`+ .. warning:: backend - You have to terminate existing resalloc resources.`
		See `Terminate resalloc resources`_.

		+ .. warning:: backend - `Terminate OpenStack VMs`_.
		`+`
		`Finally, shut down the instance to avoid storage inconsistency and other possible problems::`

		`$ ssh root@<old_name>.fedorainfracloud.org`
		`@@ -215,6 +217,20 @@`
		`resalloc-maint resource-delete $(resalloc-maint resource-list \| cut -d' ' -f1)`


		`+ Terminate OpenStack VMs`
		`+ .......................`
		`+`
		`+ Make sure you terminate all the OpenStack located builders allocated by`
		+ ``copr-backend.service``::
		`+`
		`+ # systemctl stop copr-backend # ensure that new are not allocated anymore`
		`+ # su - copr`
		`+ $ redis-cli # drop the DB`
		`+ 127.0.0.1:6379> FLUSHALL`
		`+ ^D`
		`+ $ cleanup_vm_nova.py # clean all VMs not in DB`
		`+`
		`+`
		`Downgrade python novaclient`
		`...........................`

praiskup commented 4 years ago

We forgot about this, and it caused that the copr production stack
was extremely slow for several days.

frostyx commented on line 18 of doc/how_to_upgrade_persistent_instances.rst 4 years ago

Wouldn't

$ ssh root@copr-be-dev.cloud.fedoraproject.org
[root@copr-be-dev ~][STG]# cleanup_vm_nova.py

be easier? :-)

praiskup commented on line 18 of doc/how_to_upgrade_persistent_instances.rst 4 years ago

Dunno, I'll take a look later ... it seems like the script needs a bit rewrite before we'll suggest its use.

frostyx commented on line 18 of doc/how_to_upgrade_persistent_instances.rst 4 years ago

Ok, np. But know that it is already suggested in how_to_release_copr.rst. And how_to_upgrade_builders.rst suggests /home/copr/cleanup_vm_nova.py. I can't remember which one is the correct one.