koji

Clone

#2582 Error: Unable to connect to server for some koji cli commands
Closed: Fixed 3 years ago by tkopecek. Opened 3 years ago by plageat.

Closed: Fixed
Reopen Issue

Hi,
I've recently setup my koji server in accordance with https://docs.pagure.org/koji/server_howto/.
After installation I've noticed, that SOME commands are broken in koji cli, for ex:

$ koji -d --debug-xmlrpc list-builds

2020-12-06 14:41:36,320 [DEBUG] koji: Opening new requests session
2020-12-06 14:41:36,320 [DEBUG] koji: url: https://kojihub.example.com/kojihub
2020-12-06 14:41:36,320 [DEBUG] koji: headers: {'User-Agent': 'koji/1', 'Content-Type': 'text/xml', 'Content-Length': '107'}
2020-12-06 14:41:36,320 [DEBUG] koji: data: '0x3c3f786d6c2076657273696f6e3d27312e30273f3e0a3c6d6574686f6443616c6c3e0a3c6d6574686f644e616d653e67657441504956657273696f6e3c2f6d6574686f644e616d653e0a3c706172616d733e0a3c2f706172616d733e0a3c2f6d6574686f6443616c6c3e0a'
2020-12-06 14:41:36,320 [DEBUG] koji: stream: True
2020-12-06 14:41:36,320 [DEBUG] koji: verify: '/home/kojiadmin/.koji/serverca.crt'
2020-12-06 14:41:36,320 [DEBUG] koji: timeout: 43200
2020-12-06 14:41:36,324 [DEBUG] koji: Opening new requests session
Error: Unable to connect to server

I've triple-checked all configs, SSL auth works fine, no problems found in connecting to DB, some commands of koji works OK (for ex. koji add-tag, koji build etc.)

After little investigation it seems that those sub-commands which do not have activate_session() call in cmd handler always return "Error: Unable to connect to server"

See also, https://bugzilla.redhat.com/show_bug.cgi?id=1904498

Interesting. Can you try running simple script (curious if we can isolate it to these calls):

import logging
import koji

logging.BasicConfig(level='DEBUG')
c = koji.ClientSession(your_https_hub_url, opts={'serverca': '/home/kojiadmin/.koji/serverca.crt'})
print(c.getAPIVersion())

Metadata Update from @tkopecek:
- Custom field Size adjusted to None
3 years ago

Well, after using
your_https_hub_url = http://kojihub.example.com/kojihub
instead of
your_https_hub_url = https://kojihub.example.com/kojihub
every broken command seems to work fine! xD

Just in case, using https:// :

$ cat test.py 

import logging
import koji



logging.basicConfig(level='DEBUG')
c = koji.ClientSession('https://kojihub.example.com/kojihub', opts={'serverca': '/home/kojiadmin/.koji/serverca.crt'})
print(c.getAPIVersion())

[kojiadmin@kojihub tmp]$ python3 test.py 
DEBUG:koji:Opening new requests session
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): kojihub.example.com:443
DEBUG:koji:Opening new requests session
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.8/http/client.py", line 1049, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.8/http/client.py", line 971, in send
    self.sock.sendall(data)
  File "/usr/lib64/python3.8/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib64/python3.8/ssl.py", line 1173, in send
    return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.8/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3.8/site-packages/urllib3/packages/six.py", line 702, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib64/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.8/http/client.py", line 1049, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.8/http/client.py", line 971, in send
    self.sock.sendall(data)
  File "/usr/lib64/python3.8/ssl.py", line 1204, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib64/python3.8/ssl.py", line 1173, in send
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 6, in <module>
    print(c.getAPIVersion())
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2346, in __call__
    return self.__func(self.__name, args, opts)
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2754, in _callMethod
    return self._sendCall(handler, headers, request)
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2672, in _sendCall
    return self._sendOneCall(handler, headers, request)
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2715, in _sendOneCall
    r = self.rsession.post(handler, **callopts)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

It looks like httpd.conf issue. Don't you've SSLVerifyClient enabled for all endpoints instead of only /kojihub/ssllogin?

You are right. I had

cat /etc/httpd/conf.d/ssl.conf | grep SSLVeri
SSLVerifyClient require
SSLVerifyDepth  10

After commenting it out, I'm also able to run cmd all commands.
But there is no way to figure it out by the output error...

Yes, there could be some better error. Not sure if we can detect it properly. It was just my guess that you're starting ssl connection but apache refuse to continue with that.

Metadata Update from @tkopecek:
- Issue tagged with: usability
3 years ago

Please, do mention at least that in docs/server_howto
Thanks

Metadata Update from @tkopecek:
- Custom field Size adjusted to small (was: None)
- Issue set to the milestone: 1.24
3 years ago

CentOS 8 has mod_ssl-1:2.4.37-30.module_el8.3.0+561+97fdbbcc.x86_64, and that package ships /etc/httpd/conf.d/ssl.conf with #SSLVerifyClient require commented out. I guess we can update the howto doc to mention that only the "sslogin" path should have SSLVerifyClient, and administrators should not set that globally (not in /etc/httpd/conf.d/ssl.conf or anywhere else).

Regarding that generic "Error: Unable to connect to server" message, I wonder if #2592 would help make the underlying server problem clearer.

I removed a bit of the hub httpd conf where we recommended setting SSLVerifyClient require outside the <Location> directive: https://pagure.io/koji/pull-request/2627

When I went to add this information to the server howto guide, I am not sure it makes sense to describe the "wrong" configuration. That guide is already really long. @plageat are you ok with resolving this without updating the guide, and landing #2592 and #2627 for this instead?

I think it's a bit better that it was before.

I agree, that the guide is quite long. Perhaps, you could mention all of this as a comment, for example, in /etc/httpd/conf.d/kojihub.conf. One line just like you wrote - "administrators should not set that globally" would be perfect.

Commit b3f548f fixes this issue

Metadata Update from @jcupova:
- Issue tagged with: no_qe
3 years ago

Login to comment on this ticket.

Metadata
None

usability no_qe

None
None
Normal
small
Related Pull Requests
  • #2627 Merged 3 years ago
Powered by Pagure 5.13.3
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.