Hmm, I kind of like how the subprocess output is also captured by py.test currently. It means if my test does something wrong which causes an exception in Greenwave (or one of the others), py.test will show me the traceback right there in the output. With this change, I will have to find the right log file and open it up to see the traceback.

Is this because you find the subprocess output to be too noisy when a test fails?

Is this because you find the subprocess output to be too noisy when a test fails?

Yup, it's very noisy. The first thing I'm interested in is failing test case name but there is all the output from subprocesses after test case failure message.

Is it possible to set less verbose logging for the subprocesses? Maybe it would help to just clear the output before next test case is run.

I think py.test is already smart enough to show you only the captured output which happened during the failing test case, right? Could you pastebin an example of the noise so I can understand what exactly we are talking about?

Here is output for:

TEST=true python -m pytest functional-tests -k test_make_a_decision_with_verbose_flag

Oh yeah I see what you mean. That particular case is bad because we are inserting 72 results into Resultsdb and it's printing the whole message it would send for every single result.

I actually changed that too as part of the "remove errata-related policies, use Fedora policies instead" commit in PR#184, that test only inserts 3 results instead of 72.

Maybe for the Resultsdb and Waiverdb subprocesses, we should run them with just INFO or WARNING log level and discard their DEBUG messages? That way, if there's a traceback as part of a failing test it will still get captured by pytest and displayed (which is useful) but the amount of debug spew before that should be reduced.

That change now lives independently as a part of #196 (the "remove errata-related policies, use Fedora policies instead" change).

I've rebased, updated the code and dropped logging subprocesses to separate files.

Nice!

I needed to make one change to get it to work locally for me:

diff --git a/functional-tests/conftest.py b/functional-tests/conftest.py
index 64282de..3e258cf 100644
--- a/functional-tests/conftest.py
+++ b/functional-tests/conftest.py
@@ -124,8 +124,8 @@ def resultsdb_server(tmpdir_factory):
     DEBUG = False
     """ % dbname
-    init_db_arguments = ['python', 'run_cli.py', 'init_db']
-    start_server_arguments = ['python', 'runapp.py']
+    init_db_arguments = ['python2', 'run_cli.py', 'init_db']
+    start_server_arguments = ['python2', 'runapp.py']
     with server_subprocess(
         name='resultsdb',

Explicit python2 for resultsdb.

Explicit python2 for resultsdb.

Fixed.

Thanks @lholecek!