#24 tests all fail when paramiko 2.10.4 is installed
Opened 2 years ago by lelutin. Modified 2 years ago

I've started using this pytest plugin to run tests with commands across hosts. It would seem that when paramiko is installed all of the tests will fail (since paramiko is chosen by default for the ssh connecions).

I'm using pytest-multihost 3.4 and paramiko 2.10.4.

The current workaround is to force a fallback to the builtin ssh transport by adding this to the test files:

import os
os.environ['PYTESTMULTIHOST_SSH_TRANSPORT'] = 'openssh'

The failing tests show this information after the call to run_command(...):

/usr/lib/python3/dist-packages/pytest_multihost/host.py:271: in run_command                                                                                               
    command.wait()                                                                                                                                                        
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pytest_multihost.transport.SSHCommand object at 0x7f15e624c0a0>, raiseonerr = True                                                                                

    def wait(self, raiseonerr=DEFAULT):                                                                                                                                   
        """Wait for the remote process to exit                                                                                                                            

        Raises an exception if the exit code is not 0, unless ``raiseonerr`` is                                                                                           
        true.                                                                                                                                                             

        When ``raiseonerr`` is not specified as argument, the ``raiseonerr``                                                                                              
        attribute is used.                                                                                                                                                
        """                                                                                                                                                               
        if raiseonerr is DEFAULT:                                                                                                                                         
            raiseonerr = self.raiseonerr                                             

        if self._done:                                                               
            return self.returncode        

        self._end_process()                                                          

        self._done = True                                                                                                                                                 

        if raiseonerr and self.returncode:                                                                                                                                
            self.log.error('Exit code: %s', self.returncode)                         
>           raise subprocess.CalledProcessError(self.returncode, self.argv)          
E           subprocess.CalledProcessError: Command '['jc', 'traceroute', '-q', '1', '-w', '2,1,1', '10.1.0.95']' returned non-zero exit status 1.

/usr/lib/python3/dist-packages/pytest_multihost/transport.py:215: CalledProcessError

I've run a little bit of debugging on the test and the only thing that I can see for now is that the command correctly gets executed by paramiko and the output is returned, but two things differ from the OpenSSHTransport:

  1. the commands ends with an exit code of 1, triggering the failure
  2. the output that paramiko retrieves, e.g. stdout_text, contains the motd from the server, whereas stdout_text with OpenSSHTransport does not contain the contents of the motd

I don't know enough about paramiko to press much further in the debugging though

Hi @lelutin

Sorry about the delayed response. Is there any chance you could attach a script/test that reproduces the issue you're seeing?

Also, are you seeing the failure on any command run when using paramiko?

Hi @spoore, no worries, your response came in actually pretty quickly :)

I'm seeing the error on all calls to run_command().

Here's a pruned down version of the script that I'm using. The host "rtr0.test" is a vagrant VM on my laptop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#!/usr/bin/python3
import pytest 
from pytest_multihost import make_multihost_fixture


@pytest.fixture(scope='class')
def multihost(request):
    mh = make_multihost_fixture(
            request,
            descriptions=[
                {
                    "type": "rtr",
                    "hosts": {
                        "master": 1,
                    }
                }
            ])

    return mh


class TestNetworkingLabOptimal(object):
    def test_ping(self, multihost):
        remote_host = multihost.config.domains[0].host_by_name("rtr0")
        remote_host.run_command(['ping', '-c', '1', '-W', '1', '8.8.8.8'])

with the configuration file that looks like:

ssh_key_filename: /home/gabriel/control-repo/.vagrant/machines/rtr0/libvirt/private_key
ssh_username: vagrant
domains:
  - name: test
    type: rtr
    hosts:
      - name: rtr0
        ip: 192.168.121.203
        role: master

Another detail that might be interesting is that, additionally to the motd, stdout_text does contain the output that I would expect to see from the command that's being run.

I couldn't figure out yet why even though the command that I ask it to call seems to be running correctly, I'm getting a return code of 1

p.s.: some of the options that can be used in the pytest-multihost config file are not documented.. I had to read the library's source code in order to know about ssh_username for example

Is it possible that the vagrant user has a .logout or .bash_logout with something that's failing and exiting with a return code of 1?

I am able to reproduce this with a custom .bash_logout file with "exit 1" for the vagrant user on the remote host. If I switch from paramiko to openssh, it starts passing. I suspect paramiko is using more of the environment than openssh but, I'll have to dig deeper to identify the exact issue here.

Does running pytest with --log-cli-level=DEBUG show anything else useful?

oh wow you put your finger right on it apparently!

the vagrant user has a .bash_logout file that comes from debian's default contents in /etc/skel. if I add "exit 0" at the top, then the test succeeds.

The .bash_logout file looks like this:

# ~/.bash_logout: executed by bash(1) when login shell exits.

# when leaving the console clear the screen to increase privacy

if [ "$SHLVL" = 1 ]; then
    [ -x /usr/bin/clear_console ] && /usr/bin/clear_console -q
fi

If I were to take a guess at what's different between the builtin "openssh" transport and the paramiko one, the builtin transport asks ssh to start a command directly (bash) which does not request a tty, whereas paramiko seems to be grabbing a tty and so motd and logout scripts end up interfering.

Do you know if there's a way to avoid running user login and logout scripts with paramiko? If it's not possible, I guess it's a case of "won't fix" but it might be interesting to note this behaviour down in the pytest-multihost readme?

That's what I was thinking too but, looking at the code again, paramiko and openssh are both running an interactive shell. We're using the invoke_shell() call from paramiko for this and we're not specifying the -T option for openssh to disable using a tty.

I can see openssh run the .bashrc on login but, after exit is called, it's not running (or not capturing) the .bash_logout. I'm not sure why but, this is definitely worth documenting at least.

If you have a chance, can you take a look at PR #25 and let me know if the README.rst note makes sense?

Login to comment on this ticket.

Metadata
Related Pull Requests
  • #25 Merged a year ago