In test_runner.py we have code like this:

class TestRunnerPrep():
    def test_prep_invalid_commands(self, tmpdir):
        ref_taskdata = {'preparation': {'turboencabulator':
                                            'reduce sinusoidal repleneration'}}
        ref_inputdata = {}

        test_runner = runner.Runner(ref_taskdata, ref_inputdata, tmpdir)

        with pytest.raises(Exception):
            test_runner.do_prep()

The interesting part is that do_prep method doesn't exist (I guess it used to exist in the past). But the tests don't fail (yay!:)), because we catch the whole Exception, which is basically anything, even AttributeError. I assume that is the remnant of the times we didn't have our own exceptions module.

Go through test_runner.py (and check other modules as well) and replace all Exception catching with the proper and targeted exception type. Fix the actual code where necessary. If in doubts what the code or unit test should do, ask the author.