#4191 [1.34.0] Reading the config files can fail if they contain UTF-8 characters.
Closed: Fixed 3 months ago by tkopecek. Opened 5 months ago by frank-mdc.

The reason will be this call in config.read(cfgs) in line 2342 of koji/init.py
The fix will be:

config.read(cfgs, encoding="utf8")  

instant of

config.read(cfgs) 

Without the change, the hub can crash when an comment in the config files contains an unicode character.


Metadata Update from @tkopecek:
- Custom field Size adjusted to None
- Issue set to the milestone: 1.35.1

5 months ago

Metadata Update from @mikem:
- Issue set to the milestone: 1.35.2 (was: 1.35.1)

4 months ago

Metadata Update from @tkopecek:
- Issue tagged with: testing-ready

4 months ago

Do you have an example that will trigger this issue?

Yes of course:
/etc/koji-hub/plugins/foo.conf

[foo]
# key_file = äöüß

This will trigger it on RHEL-9

I am so far unable to get koji.read_config_files to error on this or similar content in a rhel9 container. Do you have an example traceback?

Yes of course:

Tue Oct 29 08:42:45.144480 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540] 2024-10-29 08:42:45,144 [WARNING] m=build u=fbuettn p=3457742 r=X.X.X.X:36540 koji.xmlrpc: Traceback (m
ost recent call last):                                                                                                                                                                                             
[Tue Oct 29 08:42:45.144538 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib/python3.9/site-packages/koji/plugin.py", line 223, in run_callbacks                         
[Tue Oct 29 08:42:45.144545 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     func(cbtype, *cb_args, **cb_kwargs)                                                                      
[Tue Oct 29 08:42:45.144550 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib/koji-hub-plugins/epost.py", line 69, in receive_event                                       
[Tue Oct 29 08:42:45.144555 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     config = mdc.common.AddonConfig("epost")                                                                 
[Tue Oct 29 08:42:45.144560 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib/python3.9/site-packages/mdc_koji_hub_plugins/common.py", line 8, in __init__                
[Tue Oct 29 08:42:45.144565 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     self._read_config()
[Tue Oct 29 08:42:45.144570 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib/python3.9/site-packages/mdc_koji_hub_plugins/common.py", line 12, in _read_config
[Tue Oct 29 08:42:45.144575 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     self._config = koji.read_config_files([(self._file, True)])
[Tue Oct 29 08:42:45.144580 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib/python3.9/site-packages/koji/__init__.py", line 2342, in read_config_files
[Tue Oct 29 08:42:45.144584 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     config.read(cfgs)
[Tue Oct 29 08:42:45.144589 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib64/python3.9/configparser.py", line 697, in read
[Tue Oct 29 08:42:45.144594 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     self._read(fp, filename)
[Tue Oct 29 08:42:45.144598 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib64/python3.9/configparser.py", line 1020, in _read
[Tue Oct 29 08:42:45.144603 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     for lineno, line in enumerate(fp, start=1):
[Tue Oct 29 08:42:45.144607 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib64/python3.9/encodings/ascii.py", line 26, in decode
[Tue Oct 29 08:42:45.144612 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     return codecs.ascii_decode(input, self.errors)[0]
[Tue Oct 29 08:42:45.144617 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 9: ordinal not in range(128)
[Tue Oct 29 08:42:45.144644 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540] 
[Tue Oct 29 08:42:45.144648 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540] During handling of the above exception, another exception occurred:
[Tue Oct 29 08:42:45.144684 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540] 

[Tue Oct 29 08:42:45.144704 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540] Traceback (most recent call last):
[Tue Oct 29 08:42:45.144713 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib/python3.9/site-packages/kojihub/kojixmlrpc.py", line 273, in _wrap_handler
[Tue Oct 29 08:42:45.144722 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     response = handler(environ)
[Tue Oct 29 08:42:45.144729 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib/python3.9/site-packages/kojihub/kojixmlrpc.py", line 300, in handle_rpc
[Tue Oct 29 08:42:45.144733 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     return self._dispatch(method, params)
[Tue Oct 29 08:42:45.144747 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib/python3.9/site-packages/kojihub/kojixmlrpc.py", line 337, in _dispatch
[Tue Oct 29 08:42:45.144767 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     ret = koji.util.call_with_argcheck(func, params, opts)
[Tue Oct 29 08:42:45.144776 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib/python3.9/site-packages/koji/util.py", line 507, in call_with_argcheck
[Tue Oct 29 08:42:45.144790 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     return func(*args, **kwargs)
[Tue Oct 29 08:42:45.144799 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib/python3.9/site-packages/kojihub/kojihub.py", line 10739, in build
[Tue Oct 29 08:42:45.144806 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     return make_task('build', [src, target, opts], **taskOpts)
[Tue Oct 29 08:42:45.144820 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib/python3.9/site-packages/kojihub/kojihub.py", line 749, in make_task
[Tue Oct 29 08:42:45.144828 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     koji.plugin.run_callbacks(
[Tue Oct 29 08:42:45.144837 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib/python3.9/site-packages/koji/plugin.py", line 230, in run_callbacks
[Tue Oct 29 08:42:45.144848 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     raise koji.CallbackError('%s:\\n%s' % (msg, tb))
[Tue Oct 29 08:42:45.144853 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540] koji.CallbackError: Error running postTaskStateChange callback from _koji_plugin__epost:
[Tue Oct 29 08:42:45.144858 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540] Traceback (most recent call last):
[Tue Oct 29 08:42:45.144873 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib/python3.9/site-packages/koji/plugin.py", line 223, in run_callbacks
[Tue Oct 29 08:42:45.144878 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     func(cbtype, *cb_args, **cb_kwargs)
[Tue Oct 29 08:42:45.144893 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib/koji-hub-plugins/epost.py", line 69, in receive_event
[Tue Oct 29 08:42:45.144899 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     config = mdc.common.AddonConfig("epost")
[Tue Oct 29 08:42:45.144912 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib/python3.9/site-packages/mdc_koji_hub_plugins/common.py", line 8, in __init__
[Tue Oct 29 08:42:45.144921 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     self._read_config()
[Tue Oct 29 08:42:45.144932 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib/python3.9/site-packages/mdc_koji_hub_plugins/common.py", line 12, in _read_config
[Tue Oct 29 08:42:45.144940 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     self._config = koji.read_config_files([(self._file, True)])
[Tue Oct 29 08:42:45.144949 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib/python3.9/site-packages/koji/__init__.py", line 2342, in read_config_files
[Tue Oct 29 08:42:45.144957 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     config.read(cfgs)
[Tue Oct 29 08:42:45.144966 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib64/python3.9/configparser.py", line 697, in read
[Tue Oct 29 08:42:45.144977 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     self._read(fp, filename)
[Tue Oct 29 08:42:45.144994 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib64/python3.9/configparser.py", line 1020, in _read
[Tue Oct 29 08:42:45.145045 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     for lineno, line in enumerate(fp, start=1):
[Tue Oct 29 08:42:45.145051 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]   File "/usr/lib64/python3.9/encodings/ascii.py", line 26, in decode
[Tue Oct 29 08:42:45.145064 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540]     return codecs.ascii_decode(input, self.errors)[0]
[Tue Oct 29 08:42:45.145075 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 9: ordinal not in range(128)
[Tue Oct 29 08:42:45.145080 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540] 
[Tue Oct 29 08:42:45.145093 2024] [wsgi:error] [pid 3457742:tid 3457786] [client X.X.X.X:36540] 

This will happens, when an config file of an add-on contain an UTF-8 comment.
"crash" line:

[mail]
# ÖÄÜ?

When I set an comment like this in the global hub config it will result in:

[Tue Oct 29 08:50:01.743558 2024] [wsgi:error] [pid 3463652:tid 3463687] [client X.X.X.X:38136] 2024-10-29 08:50:01,743 [ERROR] SETUP p=3463652 koji: Traceback (most recent call last):
[Tue Oct 29 08:50:01.743661 2024] [wsgi:error] [pid 3463652:tid 3463687] [client X.X.X.X:38136]   File "/usr/lib/python3.9/site-packages/kojihub/kojixmlrpc.py", line 761, in server_setup
[Tue Oct 29 08:50:01.743670 2024] [wsgi:error] [pid 3463652:tid 3463687] [client X.X.X.X:38136]     opts = load_config(environ)
[Tue Oct 29 08:50:01.743675 2024] [wsgi:error] [pid 3463652:tid 3463687] [client X.X.X.X:38136]   File "/usr/lib/python3.9/site-packages/kojihub/kojixmlrpc.py", line 438, in load_config
[Tue Oct 29 08:50:01.743680 2024] [wsgi:error] [pid 3463652:tid 3463687] [client X.X.X.X:38136]     config = koji.read_config_files([cfdir, (cf, True)], raw=True)
[Tue Oct 29 08:50:01.743685 2024] [wsgi:error] [pid 3463652:tid 3463687] [client X.X.X.X:38136]   File "/usr/lib/python3.9/site-packages/koji/__init__.py", line 2342, in read_config_files
[Tue Oct 29 08:50:01.743699 2024] [wsgi:error] [pid 3463652:tid 3463687] [client X.X.X.X:38136]     config.read(cfgs)
[Tue Oct 29 08:50:01.743704 2024] [wsgi:error] [pid 3463652:tid 3463687] [client X.X.X.X:38136]   File "/usr/lib64/python3.9/configparser.py", line 697, in read
[Tue Oct 29 08:50:01.743708 2024] [wsgi:error] [pid 3463652:tid 3463687] [client X.X.X.X:38136]     self._read(fp, filename)
[Tue Oct 29 08:50:01.743713 2024] [wsgi:error] [pid 3463652:tid 3463687] [client X.X.X.X:38136]   File "/usr/lib64/python3.9/configparser.py", line 1020, in _read
[Tue Oct 29 08:50:01.743718 2024] [wsgi:error] [pid 3463652:tid 3463687] [client X.X.X.X:38136]     for lineno, line in enumerate(fp, start=1):
[Tue Oct 29 08:50:01.743722 2024] [wsgi:error] [pid 3463652:tid 3463687] [client X.X.X.X:38136]   File "/usr/lib64/python3.9/encodings/ascii.py", line 26, in decode
[Tue Oct 29 08:50:01.743727 2024] [wsgi:error] [pid 3463652:tid 3463687] [client X.X.X.X:38136]     return codecs.ascii_decode(input, self.errors)[0]
[Tue Oct 29 08:50:01.743734 2024] [wsgi:error] [pid 3463652:tid 3463687] [client X.X.X.X:38136] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 8: ordinal not in range(128)
[Tue Oct 29 08:50:01.743739 2024] [wsgi:error] [pid 3463652:tid 3463687] [client X.X.X.X:38136

sample /etc/koji-hub/hub.conf:

[hub]
# Äüöß

So it will not be an add-on problem.

Metadata Update from @mfilip:
- Issue tagged with: testing-done

3 months ago

Log in to comment on this ticket.

Metadata
Related Pull Requests
  • #4214 Merged 3 months ago