PR#251: Kill zombie VMs from ImageFactory - taskotron

taskotron

#251 Kill zombie VMs from ImageFactory

Closed 6 years ago by kparal. Opened 6 years ago by frantisekz.

frantisekz/taskotron infra-ansible into infra-ansible

Kill zombie VMs from ImageFactory

František Zatloukal • 6 years ago

50afb7f

roles/taskotron/imagefactory/files/imagefactory-kill-zombie

file added

+20

		`@@ -0,0 +1,20 @@`
		`+ #!/bin/bash`
		`+`
		`+ PROCESS_IDENTIFICATOR='/usr/bin/qemu-system-x86_64 -machine accel=kvm -name guest=factory-build'`
		`+`
		`+ if [ "$1" == "" ]; then`
		`+ echo "Please, specify a time that how long a process has to be running to be killed (in minutes)"`
		`+ exit`
		`+ fi`
		`+`
		`+`
		`+ # Check if we have any process matching PROCESS_IDENTIFICATOR`
		`+ while (( $(pgrep -o -f "$PROCESS_IDENTIFICATOR") )); do`
		`+ # Get PID of the longest (pgrep -o) running matched process`
		`+ if (( $(ps -o etimes= -p $(pgrep -o -f "$PROCESS_IDENTIFICATOR")) >= $1*60)); then`
		`+ pgrep -o -f "$PROCESS_IDENTIFICATOR" \| xargs kill;`
		`+ else`
		`+ # We can exit if the oldest matching process wasn't old enough to be killed :)`
		`+ exit`
		`+ fi`
		`+ done`
		`\ No newline at end of file`

roles/taskotron/imagefactory/tasks/main.yml

file modified

		`@@ -13,6 +13,9 @@`
		`- name: copy oz.cfg config file`
		`copy: src=oz.cfg dest=/etc/oz/oz.cfg owner=root group=root mode=0644`

		`+ - name: copy script to kill zombie ImageFactory VMs`
		`+ copy: src=imagefactory-kill-zombie dest=/usr/local/bin owner=root group=root mode=0744`
		`+`
		`- name: hotfix imagefactory's REST api to allow file download`
		`copy: src=hotfix_imgfac_RESTv2.py dest=/usr/lib/python2.7/site-packages/imgfac/rest/RESTv2.py owner=root group=root mode=0644`

		`@@ -72,3 +75,9 @@`
		`name: "Prune old builds in ImageFactory"`
		`special_time: daily`
		`job: '/var/lib/fedoraqa/base_images/prune_images/prune_images.py imgfac-old --config /etc/taskotron_base_images/config_server.ini --keep 3 > /dev/null'`
		`+`
		`+ - name: Create cronjob to kill zombie VMs from ImageFactory`
		`+ cron:`
		`+ name: "kill zombie VMs from ImageFactory"`
		`+ special_time: daily`
		`+ job: 'imagefactory-kill-zombie'`

frantisekz commented 6 years ago

Fixes https://pagure.io/taskotron/issue/233 .

Script will iterate through all QEMU processes by imagefactory till there is no one running longer than three hours. It iterates from oldest to newest processes (if there is more than one). The script exits if the oldest QEMU processes by imagefactory is younger than 3 hours.

TODO:
- Argument to specify maximum factory run time in minutes - DONE
- Use a variable instead of duplicating the same very long argument three times - DONE

rebased onto a1b2e18

6 years ago

rebased onto 50afb7f

6 years ago