Issue #183: yamlish crashes with UnicodeDecodeError - libtaskotron

taskotron / libtaskotron

#183 yamlish crashes with UnicodeDecodeError

Closed: Fixed None Opened 9 years ago by kparal.

In this log http://taskotron-dev.fedoraproject.org/taskmaster/builders/x86_64/builds/5450/steps/runtask/logs/stdio there is this crash:

runtask -i marble-4.14.0-1.fc21 -t koji_build -a x86_64 -j x86_64/5450 rpmlint.yml
 in dir /home/buildslave/slave/x86_64/build (timeout 1200 secs)
 watching logfiles {}
 argv: ['runtask', '-i', 'marble-4.14.0-1.fc21', '-t', 'koji_build', '-a', 'x86_64', '-j', 'x86_64/5450', 'rpmlint.yml']
 environment:
  HOME=/home/buildslave
  LANG=en_US.UTF-8
  LOGNAME=buildslave
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
  PWD=/home/buildslave/slave/x86_64/build
  SHELL=/bin/bash
  USER=buildslave
 using PTY: False
[libtaskotron:runner.py:207] 2014-08-15 13:49:05 DEBUG   Parsed arguments: Namespace(arch=['x86_64'], debug=False, item='marble-4.14.0-1.fc21', jobid='x86_64/5450', task=['rpmlint.yml'], type='koji_build')
[libtaskotron:config.py:93] 2014-08-15 13:49:05 DEBUG   Using config file: /etc/taskotron/taskotron.yaml
[libtaskotron:config.py:64] 2014-08-15 13:49:05 DEBUG   Using config profile: production
[libtaskotron:koji_directive.py:133] 2014-08-15 13:49:05 INFO    Getting RPMs for koji build marble-4.14.0-1.fc21 (['x86_64', 'noarch']) and downloading to /var/tmp/taskotron/task-VgVH2I
[libtaskotron:koji_utils.py:116] 2014-08-15 13:49:05 INFO    Querying Koji for a list of RPMS for: marble-4.14.0-1.fc21
[libtaskotron:koji_utils.py:166] 2014-08-15 13:49:05 INFO    Fetching 6 RPMs for: marble-4.14.0-1.fc21
[libtaskotron:python_directive.py:154] 2014-08-15 13:49:06 INFO    Executing Python: run_rpmlint.run() with args {'workdir': '/var/tmp/taskotron/task-VgVH2I'}
[libtaskotron:logger.py:70] 2014-08-15 13:49:18 CRITICAL Traceback (most recent call last):
  File "/usr/bin/runtask", line 9, in <module>
    load_entry_point('libtaskotron==0.3.5', 'console_scripts', 'runtask')()
  File "/usr/lib/python2.7/site-packages/libtaskotron/runner.py", line 223, in main
    task_runner.run()
  File "/usr/lib/python2.7/site-packages/libtaskotron/runner.py", line 45, in run
    self.do_actions()
  File "/usr/lib/python2.7/site-packages/libtaskotron/runner.py", line 129, in do_actions
    self.do_single_action(action)
  File "/usr/lib/python2.7/site-packages/libtaskotron/runner.py", line 109, in do_single_action
    self.envdata)
  File "/usr/lib/python2.7/site-packages/libtaskotron/directives/python_directive.py", line 214, in process
    output = self.execute(task, method_name, input_data)
  File "/usr/lib/python2.7/site-packages/libtaskotron/directives/python_directive.py", line 155, in execute
    output = task_method(**kwargs)
  File "run_rpmlint.py", line 31, in run
    tapdetails = check.export_TAP(results)
  File "/usr/lib/python2.7/site-packages/libtaskotron/check.py", line 331, in export_TAP
    output = tapgen.format_TAP_msg(result=tap_outcome, name=name, data=data)
  File "/usr/lib/python2.7/site-packages/tap.py", line 90, in format_TAP_msg
    return self.translate_msg[result](name, desc, data)
  File "/usr/lib/python2.7/site-packages/tap.py", line 72, in _pass
    return self._format_string("ok", desc, name, data)
  File "/usr/lib/python2.7/site-packages/tap.py", line 64, in _format_string
    yamlish_str = yamlish.dumps(data)
  File "/usr/lib/python2.7/site-packages/yamlish.py", line 214, in dumps
    canonical=False, Dumper=_YamlishDumper)
  File "/usr/lib64/python2.7/site-packages/yaml/__init__.py", line 202, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
  File "/usr/lib64/python2.7/site-packages/yaml/__init__.py", line 190, in dump_all
    dumper.represent(data)
  File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 28, in represent
    node = self.represent_data(data)
  File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 57, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 223, in represent_dict
    return self.represent_mapping(u'tag:yaml.org,2002:map', data)
  File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 123, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 57, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 223, in represent_dict
    return self.represent_mapping(u'tag:yaml.org,2002:map', data)
  File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 123, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/usr/lib64/python2.7/site-packages/yaml/representer.py", line 57, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/lib/python2.7/site-packages/yamlish.py", line 158, in str_representer_compact_multiline
    if '\n' in data:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 59: ordinal not in range(128)

program finished with exit code 1
elapsedTime=12.969183

The problem is in yamlish:

def str_representer_compact_multiline(dumper, data):
    style = None
    if '\n' in data:
        style = '|'
    if isinstance(data, str):
        data = data.decode('utf-8') # assumes all your strings are UTF-8 encoded
    tag = u'tag:yaml.org,2002:str'
    return dumper.represent_scalar(tag, data, style)

I guess these two lines:

    if isinstance(data, str):
        data = data.decode('utf-8') # assumes all your strings are UTF-8 encoded

should be before these two lines:

    if '\n' in data:
        style = '|'

Patch yamlish and test the fix. If it works, send the patch upstream and once a fixed version is released, update libtaskotron requirements and ask @tflink to push this change to dev infra. Thanks.