Please provide a tool that given a result directory with RPMs and/or koji scratchbuild, runs the CI test in local container / mock / virtual machine. Such tool could be integrated into fedpkg. This is essential to enable quick debugging, instead of iterating over a PR with ~hours of waiting between every change.
Metadata Update from @bookwar: - Issue tagged with: feature
Metadata Update from @mvadkert: - Issue assigned to mvadkert
Just a note: This is the main blocker for us. Not being able to reproduce the test failure locally is a real deal-breaker of the entire CI thing :(
What can we do to make this a priority? This has been open without a response for 3 months.
@churchyard hi, is using a downstream tool without the question? We have a tool that is able to execute tests quite well with 1 line command on a VM (outside your localhost). It is called 1minutetip.
I believe we could make this priority in next sprint though, but I would like to do it properly.
Who's downstream here? If you mean using "Red Hat employees only" tools, I can live with that as a temporary workaround, however it's not a solution.
yes, Red Hat employees, will send you over on IRC. Of course as a temporary workaround.
Will speak to my teamlead if I could work on this next sprints
Metadata Update from @bookwar: - Issue priority set to: High
I will start with virtualmachine testing here and will try to have something handy until DevConf form a copr repository.
I forgot to talk to you about this at DevConf. I assume there is nothing handy yet, right?
Yeah, we are in process of opensourcing the tool we have. I will let you know ASAP if there is something handy to play with.
Any chance there'll be something to try at Flock?
just a prototype for https://pagure.io/flock/issue/170 :(
we are closer a bit to having a tool for local testing with the flock prototype. We would like to have something more handy in Q3 with @psss
We've started work on a test metadata tool prototype tmt which will be used for handling L1 and L2 test metadata as defined by the specification as well as for running tests according to these metadata in different ways (localhost, container...).
tmt
For now there is just an outline of the command line options / subcommands but expect more in the near future. Also give us feedback if the proposed syntax outline does not look ok.
tmt run now has the first fuctionality built in. It should be able to execute FMF based tests in a local VM or localhost
No artifact installation is currently supported, that is something to work on
Can we specify our Fedora CI tests in FMF?
Curently we are not able to run STI tests, but it will be failrly easy to add that
FTR: https://github.com/psss/tmt/issues/50
Hi, I just ran into this issue now, 2 years after it was opened. There is a weird issue in my tests, I'm trying to debug it and it's a huge pain, because every change I make takes almost an hour before I see the results.
Isn't it time to make this a priority?
@torsava we are working on TMT which will provide a more reasonable way how to run stuff on localhost, but it is primarily NOT targeting STI interface ...
I guess so ... asking some additional details in the github PR and we will finish this via tmt
Submitted support for STI tests for TMT:
https://github.com/psss/tmt/pull/229
The execution in container will work, but we need to patch also standard-test-roles
I just ran into this issue now, 2 years after it was opened. There is a weird issue in my tests, I'm trying to debug it and it's a huge pain, because every change I make takes almost an hour before I see the results.
I see and completely understand.
It is a priority and we are working hard to make the test writing/executing experience much better. Unfortunately so far only a few pepole joined to help or contribute. But we're getting close. The tmt tool is already able to execute tests on local host, in a vm, in a container and connect to an existing host. So debugging a test failure should be as easy as tmt run in your git repo.
tmt run
Around the end of May we should have the first version of Fedora CI support for executing tests based on L2 Metadata configuration. See also documentation for a bunch of examples of how to work with tmt.
@torsava, if you find this helpful, please give it a try. We will be glad for any feedback.
Metadata Update from @psss: - Issue assigned to psss (was: mvadkert)
During the last couple of weeks we've made substantial progress on the tmt implementation. Currently tmt supports five provision methods:
virtual ... use testcloud to provision a vm for testing container ... create a new container using podman connect ... connect to a provisioned guest using ssh minute ... provision guest using 1minutetip backend (internal only) local ... use local host for test execution
Executing all tests for your component can then be as easy as "tmt run" in your component's git repo. Choosing your preferred environment for testing can be done in the following way:
tmt run --all provision --how virtual --image f31 tmt run --all provision --how connect --guest 10.0.136.55 tmt run --all provision --how minute --image 1MT-Fedora-31 tmt run --all provision --how container --image fedora:31 tmt run --all provision --how local
Installing freshly built rpms on the system is easy as well:
tmt run --all prepare --how install --package my-freshly-built.rpm tmt run --all prepare --how install --directory tmp/RPMS/noarch
For experimenting install the latest package from the copr repo, see tmt examples for more inspiration. Please, give it a try and let us know how well it covers your use cases and what's still missing.
If everything goes well, the first version of Fedora CI pipeline support for tmt tests should be available in June.
thanks for the update @psss I will just add that STI support is coming, I just need to rebase my patch finally
Please, give it a try and let us know how well it covers your use cases and what's still missing.
I don't understand how do I try it. I've tried:
[python-tox (rpmbuild_test %)]$ sudo dnf copr enable psss/tmt ... [python-tox (rpmbuild_test %)]$ sudo dnf install tmt ... [python-tox (rpmbuild_test %)]$ tmt run --all provision --how container --image fedora:rawhide Usage: tmt run provision [OPTIONS] Try 'tmt run provision --help' for help. Error: no such option: --image [python-tox (rpmbuild_test %)]$ dnf install tmt-provision-container ... [python-tox (rpmbuild_test %)]$ tmt run --all provision --how container --image fedora:rawhide /var/tmp/tmt/run-008 /plans/default discover how: shell summary: 0 tests selected warning: No tests found, finishing plan. total: no results found [python-tox (rpmbuild_test %)]$ cd tests/ [tests (rpmbuild_test %)]$ tmt run --all provision --how container --image fedora:rawhide /var/tmp/tmt/run-009 /plans/default discover how: shell summary: 0 tests selected warning: No tests found, finishing plan. total: no results found
STI support still coming sorry :(
In that case I am confused. Why do you ask to give it a try when the thing we asked for is not ready to be tested?
@churchyard I did not ask anything :) @psss ? :)
@psss did. I've meant "you" as in plural. Sorry for the confusion.
Sorry, forgot to mention that STI executor is in progress. If you would like to try out how the new configuration would look like have a look at the pull request I've just created:
Here is an example output of tmt run -v:
tmt run -v
Found 1 plan. /plans/venv summary: Test virtual environments discover how: fmf url: https://src.fedoraproject.org/forks/psss/tests/python.git ref: tmt summary: 7 tests selected /smoke/venv/python27 /smoke/venv/python34 /smoke/venv/python35 /smoke/venv/python36 /smoke/venv/python37 /smoke/venv/python38 /smoke/venv/python39 provision how: virtual image: fedora user: root memory: 2048 MB disk: 10 GB qcow: Fedora-Cloud-Base-32-1.6.x86_64.qcow2 name: HdjqbftLwZsphaxX progress: booting... distro: Fedora release 32 (Thirty Two) kernel: 5.6.6-300.fc32.x86_64 summary: 1 guest provisioned prepare how: install summary: Install required packages name: requires package: gcc, python36, python2-devel and 9 more summary: 1 preparation applied execute how: shell.tmt 00:00:25 /smoke/venv/python27 00:00:25 /smoke/venv/python34 00:00:20 /smoke/venv/python35 00:00:19 /smoke/venv/python36 00:00:18 /smoke/venv/python37 00:00:16 /smoke/venv/python38 00:00:28 /smoke/venv/python39 summary: 7 tests executed report how: display pass /smoke/venv/python27 pass /smoke/venv/python34 pass /smoke/venv/python35 pass /smoke/venv/python36 pass /smoke/venv/python37 pass /smoke/venv/python38 pass /smoke/venv/python39 summary: 7 tests passed finish guest: stopped guest: removed summary: 0 tasks completed total: 7 tests passed
tmt is available. it is advised to migrate to it. I would close this. As we would love to have STI support, it is a dead format for us ...
also we now show a reproducer command while running tmt tests, which should help with reproducability (and it will get even better this year once we implement additional features to tmt)
Poor STI :(
Metadata Update from @churchyard: - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.