PR#4: [RFC Patch] Added docker support for shim - taskotron/task-standard-interface-shim

taskotron / task-standard-interface-shim

#4 [RFC Patch] Added docker support for shim

Opened 6 years ago by burlak. Modified 6 years ago

taskotron/ burlak/task-standard-interface-shim docker-clean into master

changed dnf->package in playbook

Andrej Manduch • 6 years ago

0a9e6a4

Added docker support for shim

Andrej Manduch • 6 years ago

61752c2

runtask.yml

file modified

+16 -1

		`@@ -39,6 +39,21 @@`
		`artifactsdir: ${artifactsdir}`
		`export: standard_interface_output`

		`- - name: report results to resultsdb`
		`+ - name: run container interface tests`
		`+ python:`
		`+ file: standard-interface-shim.py`
		`+ callable: run_docker_interface_test`
		`+ workdir: ${workdir}`
		`+ koji_build: ${koji_build}`
		`+ artifactsdir: ${artifactsdir}`
		`+ docker_image: "docker.io/library/fedora:rawhide"`
		`+ rpms: ${rpm_data}`
		`+ export: container_interface_output`
		`+`
		`+ - name: report standart results to resultsdb`
		`resultsdb:`
		`results: ${standard_interface_output}`
		`+`
		`+ - name: report container results to resultsdb`
		`+ resultsdb:`
		`+ results: ${container_interface_output}`

standard-interface-shim.py

file modified

+81 -16

		`@@ -1,7 +1,10 @@`
		`import logging`
		`from collections import namedtuple`
		`import subprocess`
		`- import os`
		`+ import os, platform`
		`+`
		`+ import re`
		`+ import urllib`

		`from libtaskotron import check`
		`from libtaskotron.ext.fedora import rpm_utils`
		`@@ -44,33 +47,24 @@`
		`def create_results_file():`
		`pass`

		`- def run_standard_interface_test(koji_build, rpms, workdir='.', artifactsdir='artifacts'):`
		`- outcome = 'PASSED'`
		`-`
		`- name = rpm_utils.rpmformat(koji_build, 'n')`
		`-`
		`- # set envirnment vars`
		`+ def __set_env_vars(name, workdir='.', artifactsdir='artifacts'):`
		`environment = {}`
		`+`
		`inventory_filename = os.path.join(workdir, name, 'inventory')`
		`if os.path.exists(inventory_filename):`
		`environment['ANSIBLE_INVENTORY'] = inventory_filename`
		`else:`
		`environment['ANSIBLE_INVENTORY'] = '/usr/share/ansible/inventory'`

		`- environment['TEST_SUBJECTS'] = ' '.join(rpms['downloaded_rpms'])`
		`environment['TEST_ARTIFACTS'] = artifactsdir`

		`environment['HOME'] = os.environ.get('HOME', '/')`
		`environment['PATH'] = os.environ.get('PATH', '') + ':/usr/bin:/usr/local/bin'`

		`- # create the command`
		`- testdir = os.path.join(workdir, name)`
		`- command = ['ansible-playbook', '-e', 'artifacts='+artifactsdir, '--tags', 'classic', 'tests.yml']`
		`+ return environment`

		`- log.debug("sending environment: {}".format(environment))`
		`-`
		`-`
		`- # execute the tests`
		`+ def __execute_test(command, testdir, environment):`
		`+ outcome = 'PASSED'`
		`log.info('Running command: {} in {}'.format(' '.join(command), testdir))`
		`try:`
		`output, _ = os_utils.popen_rt(' '.join(command), shell=True, cwd=testdir, env=environment)`
		`@@ -88,8 +82,79 @@`

		`# create results file`

		`- result = Result(outcome=outcome, output=output, errors=errors, warnings=warnings)`
		`+ return Result(outcome=outcome, output=output, errors=errors, warnings=warnings)`
		`+`
		`+ def install_rpms_docker_playbook(rpms, artifactsdir='artifacts'):`
tflink commented 6 years ago I'm not as familiar with how the standard interface tests are supposed to work with containers - why do the rpms under test need to be installed prior to running the test? As this is run after the non-container tests, how can a stable environment be guaranteed if the packages do need to be installed prior to running the container tests?
		`+ output_yml = """`
		`+ - hosts: localhost`
		`+ tags:`
		`+ - container`
		`+ tasks:`
		`+ """`
		`+ install_template = """`
		`+ - copy:`
		`+ src: RPM_PATH`
		`+ dest: /root/RPM_NAME`
		`+ - name: install RPM_NAME`
		`+ package: name=/root/RPM_NAME state=present`
		`+ """`
		`+ path_replace = re.compile("RPM_PATH", re.MULTILINE)`
		`+ name_replace = re.compile("RPM_NAME", re.MULTILINE)`
		`+ for path in rpms['downloaded_rpms']:`
		`+ tmp_template = install_template`
		`+ rpm_name = os.path.basename(path)`
		`+ tmp_template = path_replace.sub(path, tmp_template)`
		`+ tmp_template = name_replace.sub(rpm_name, tmp_template)`
		`+ output_yml += tmp_template`
		`+`
		`+ inst_yml_file = os.path.join(artifactsdir, "install_rpms.yml")`
		`+ y_file = open(inst_yml_file, "w")`
		`+ y_file.write(output_yml)`
		`+ y_file.close()`
		`+`
		`+ return inst_yml_file`
		`+`
		`+`
		`+ def run_docker_interface_test(koji_build, rpms, compose=None, workdir='.', \`
		`+ artifactsdir='artifacts', docker_image="docker.io/library/fedora:rawhide"):`
		`+`
		`+ name = rpm_utils.rpmformat(koji_build, 'n')`
		`+ environment = __set_env_vars(name, workdir, artifactsdir)`
		`+ testdir = os.path.join(workdir, name)`
		`+ yaml_rpm_install = install_rpms_docker_playbook(rpms, workdir)`
		`+ command = ['ansible-playbook', '-e', 'artifacts='+artifactsdir, '-e', \`
		`+ 'subjects=docker:'+docker_image, '--tags', 'container', yaml_rpm_install, 'tests.yml']`
		`+`
		`+ log.debug("sending environment: {}".format(environment))`
		`+`
		`+ result = __execute_test(command, testdir, environment)`
		`+ # compute overall results`
		`+ detail = check.CheckDetail(koji_build, check.ReportType.KOJI_BUILD, result.outcome)`
		`+ detail.update_outcome(result.outcome)`
		`+ detail.note = 'dist-git tests for {} finished with result {}'.format(name, detail.outcome)`
		`+`
		`+ # print some summary to console`
		`+ summary = 'dist-git tests %s for %s (%s)' % (detail.outcome, koji_build, detail.note)`
		`+ log.info(summary)`
		`+`
		`+ output = check.export_YAML(detail)`
		`+ return output`
		`+`
		`+ def run_standard_interface_test(koji_build, rpms, workdir='.', artifactsdir='artifacts'):`
		`+ name = rpm_utils.rpmformat(koji_build, 'n')`
		`+`
		`+ # set envirnment vars`
		`+ environment = __set_env_vars(name, workdir, artifactsdir)`
		`+`
		`+ environment['TEST_SUBJECTS'] = ' '.join(rpms['downloaded_rpms'])`
		`+`
		`+ # create the command`
		`+ testdir = os.path.join(workdir, name)`
		`+ command = ['ansible-playbook', '-e', 'artifacts='+artifactsdir, '--tags', 'classic', 'tests.yml']`
		`+`
		`+ log.debug("sending environment: {}".format(environment))`

		`+ result = __execute_test(command, testdir, environment)`
tflink commented 6 years ago This function is missing the output return that was moved to the `run_docker_interface_test`. Without this, no results will be able to be sent to resultsdb
		`# compute overall results`
		`detail = check.CheckDetail(koji_build, check.ReportType.KOJI_BUILD, result.outcome)`
		`detail.update_outcome(result.outcome)`

burlak commented 6 years ago

I was asked in internal jira https://projects.engineering.redhat.com/browse/RHELPLAN-1472 to create pull request which would allow testing tests with --container tag using shim. And this is what I come up with.

I would like to ask you review this patch and give me some feedback or if everything is ok then push it to repository.

My main concerns is:
I changed runtask.yml so it would authomatically run always docker after runing classical test, I'm not if that's desired and maybe better option would be add
another yaml formula for runing just docker taged tests (in that case I would also update readme.rst with example how to run test with docker).

rebased onto 61752c2

6 years ago

1 new commit added

changed dnf->package in playbook

6 years ago

tflink commented on line 59 of standard-interface-shim.py 6 years ago

I'm not as familiar with how the standard interface tests are supposed to work with containers - why do the rpms under test need to be installed prior to running the test? As this is run after the non-container tests, how can a stable environment be guaranteed if the packages do need to be installed prior to running the container tests?

tflink commented on line 129 of standard-interface-shim.py 6 years ago

This function is missing the output return that was moved to the run_docker_interface_test. Without this, no results will be able to be sent to resultsdb

tflink commented 6 years ago

Sorry for the delay on this review - the notification from pagure must have gotten lost in my inbox :(

For folks who may not be able to read the linked ticket, it states:

As a QE engineer, as a workaround until Taskotron in running in production, I would like Taskotron shim to be updated to run inside a container. Currently, the shim only supports classic mode, we need to scan for tests in upstream staging instance with the container tag, and then run the command invoing the standard test interface using the --tags container option. The prereqs needed for docker should already be enabled in Vagrant.

e.g.

sudo ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory) TEST_SUBJECTS=docker:docker.io/library/fedora:rawhide TEST_ARTIFACTS=$PWD/artifacts ansible-playbook --tags container tests.yml

tflink commented 6 years ago

I assume that this is meant to at least mostly follow the docs on testing against containers with the standard interface? https://fedoraproject.org/wiki/Changes/InvokingTests

burlak commented 6 years ago

This function is missing the output return that was moved to the run_docker_interface_test. Without this, no results will be able to be sent to resultsdb

Not sure what you mean, result is processed later and returned.

I'm not as familiar with how the standard interface tests are supposed to work with containers - why do the rpms under test need to be installed prior to running the test? As this is run after the non-container tests, how can a stable environment be guaranteed if the packages do need to be installed prior to running the container tests?

why do the rpms under test need to be installed prior to running the test?

I need to install that rpm to container and then I run test on that container.

As this is run after the non-container tests, how can a stable environment be guaranteed if the packages do need to be installed prior to running the container tests?

Not sure what you mean, of course stable enviroment can't be guaranteed same way as stable enviroment can't be guaranteed without this commit, you could run shim on f24 with f26 package or f24 package on rawhide enviroment.

I'm sorry I feel like I don't understand your concerns. maybe can you schedule meeting on IRC or other platform?

I assume that this is meant to at least mostly follow the docs on testing against containers with the standard interface? https://fedoraproject.org/wiki/Changes/InvokingTests

I'm sorry no, this is first time I'm seeing this document. I briefly reviewed that document and I don't see any inconsistencies between that document and patch.

I'm also not sure how should reporting to resultdb should be done in this case when there are 2 cases run (classic and container), can you please take look to runtask.yml if way how I made reporting is acceptable ?

Edited 6 years ago by burlak