I've found out that very often command output from disposable VM is not captured by paramiko (or by our libraries) and it is missing or cropped on the host's stdout and in the host logs. The longer the command output is, the higher chance it has to be included (but often cropped). It seems to be a buffering issue.
This is very easy to test: 1. edit libtaskotron:runner.py 2. in RemoteRunner.run(), change self.exitcode = self.ssh.cmd(' '.join(task_cmd)) to self.exitcode = self.ssh.cmd('your command here'). 3. run a task remotely (ideally over --ssh, so that you don't need to wait to install dependencies every time), see that your command output is not included or cropped on stdout and in saved file logs
libtaskotron:runner.py
RemoteRunner.run()
self.exitcode = self.ssh.cmd(' '.join(task_cmd))
self.exitcode = self.ssh.cmd('your command here')
--ssh
The bug is always reproducible with commands like echo FOOOOO. With dmesg, I see output (because it's long), but I see it cropped. If you experiment with something like echo {1..100} and echo {1..10000}, you'll see that with longer output you start to see it more often.
echo FOOOOO
dmesg
echo {1..100}
echo {1..10000}
I guess the problem is somewhere around libtaskotron.remote_exec.RemoteExec.cmd().
libtaskotron.remote_exec.RemoteExec.cmd()
This ticket had assigned some Differential requests: D588
I played with stdbuf (e.g. stdbuf -o0 echo foo), but it doesn't seem to fix this issue.
stdbuf
stdbuf -o0 echo foo
However, this seems to fix it :-)
echo foo; sync
Login to comment on this ticket.