From 72f55da545a9ab0218986cd64f5cf270aa4a12bd Mon Sep 17 00:00:00 2001
From: Pavel Raiskup <praiskup@redhat.com>
Date: Jul 26 2019 13:33:53 +0000
Subject: [doc] kill all old builders when upgrading infrastructure


We forgot about this, and it caused that the copr production stack
was extremely slow for several days.

---

diff --git a/doc/how_to_upgrade_persistent_instances.rst b/doc/how_to_upgrade_persistent_instances.rst
index 00a64e0..90457e5 100644
--- a/doc/how_to_upgrade_persistent_instances.rst
+++ b/doc/how_to_upgrade_persistent_instances.rst
@@ -70,9 +70,11 @@ Backup the old instance by renaming it::
 .. warning:: You might need to backup also letsencrypt certificates.
              See `Letsencrypt renewal limits`_.
 
-.. warning:: backend - You should terminate existing resalloc resources.
+.. warning:: backend - You have to terminate existing resalloc resources.
              See `Terminate resalloc resources`_.
 
+.. warning:: backend - `Terminate OpenStack VMs`_.
+
 Finally, shut down the instance to avoid storage inconsistency and other possible problems::
 
     $ ssh root@<old_name>.fedorainfracloud.org
@@ -215,6 +217,20 @@ Then delete all current resources::
     resalloc-maint resource-delete $(resalloc-maint resource-list | cut -d' ' -f1)
 
 
+Terminate OpenStack VMs
+.......................
+
+Make sure you terminate all the OpenStack located builders allocated by
+``copr-backend.service``::
+
+    # systemctl stop copr-backend # ensure that new are not allocated anymore
+    # su - copr
+    $ redis-cli # drop the DB
+    127.0.0.1:6379> FLUSHALL
+    ^D
+    $ cleanup_vm_nova.py # clean all VMs not in DB
+
+
 Downgrade python novaclient
 ...........................