#214 kojiweb can not run with python2-cryptography-1.5.3
Closed 7 years ago Opened 7 years ago by julian8628.

when open http://$(host)/koji, error occurs in httpd

[Thu Oct 13 09:08:58.070609 2016] [wsgi:error] [pid 21878] 2016-10-13 09:08:58,070 [WARNING] m=sslLogin u=None p=21878 r=::1:52822 koji.xmlrpc: Traceback (most recent call last):
[Thu Oct 13 09:08:58.070618 2016] [wsgi:error] [pid 21878] File "/usr/share/koji-hub/kojixmlrpc.py", line 234, in _wrap_handler
[Thu Oct 13 09:08:58.070620 2016] [wsgi:error] [pid 21878] response = handler(environ)
[Thu Oct 13 09:08:58.070621 2016] [wsgi:error] [pid 21878] File "/usr/share/koji-hub/kojixmlrpc.py", line 277, in handle_rpc
[Thu Oct 13 09:08:58.070623 2016] [wsgi:error] [pid 21878] return self._dispatch(method, params)
[Thu Oct 13 09:08:58.070625 2016] [wsgi:error] [pid 21878] File "/usr/share/koji-hub/kojixmlrpc.py", line 314, in _dispatch
[Thu Oct 13 09:08:58.070626 2016] [wsgi:error] [pid 21878] ret = koji.util.call_with_argcheck(func, params, opts)
@
[Wed Nov 16 09:43:13.882023 2016] [wsgi:error] [pid 22613] [client 192.168.33.1:60707] from OpenSSL._util import (
[Wed Nov 16 09:43:13.882204 2016] [wsgi:error] [pid 22613] [client 192.168.33.1:60707] File "/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in <module>
[Wed Nov 16 09:43:13.882365 2016] [wsgi:error] [pid 22613] [client 192.168.33.1:60707] from cryptography.hazmat.bindings.openssl.binding import Binding
[Wed Nov 16 09:43:13.882509 2016] [wsgi:error] [pid 22613] [client 192.168.33.1:60707] File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 234, in <module>
[Wed Nov 16 09:43:13.882676 2016] [wsgi:error] [pid 22613] [client 192.168.33.1:60707] Binding.init_static_locks()
[Wed Nov 16 09:43:13.882806 2016] [wsgi:error] [pid 22613] [client 192.168.33.1:60707] File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 180, in init_static_locks
[Wed Nov 16 09:43:13.882939 2016] [wsgi:error] [pid 22613] [client 192.168.33.1:60707] cls._ensure_ffi_initialized()
[Wed Nov 16 09:43:13.883077 2016] [wsgi:error] [pid 22613] [client 192.168.33.1:60707] File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 175, in _ensure_ffi_initialized
[Wed Nov 16 09:43:13.883204 2016] [wsgi:error] [pid 22613] [client 192.168.33.1:60707] cls._register_osrandom_engine()
[Wed Nov 16 09:43:13.883328 2016] [wsgi:error] [pid 22613] [client 192.168.33.1:60707] File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 140, in _register_osrandom_engine
[Wed Nov 16 09:43:13.883465 2016] [wsgi:error] [pid 22613] [client 192.168.33.1:60707] _openssl_assert(cls.lib, cls.lib.ERR_peek_error() == 0)
[Wed Nov 16 09:43:13.883575 2016] [wsgi:error] [pid 22613] [client 192.168.33.1:60707] File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 62, in _openssl_assert
[Wed Nov 16 09:43:13.883719 2016] [wsgi:error] [pid 22613] [client 192.168.33.1:60707] errors_with_text
[Wed Nov 16 09:43:13.883846 2016] [wsgi:error] [pid 22613] [client 192.168.33.1:60707] InternalError: Unknown OpenSSL error. This error is commonly encountered when another library is not cleaning up the OpenSSL error stack. If you are using cryptography with another library that uses OpenSSL try disabling it before reporting a bug. Otherwise please file an issue at https://github.com/pyca/cryptography/issues with information on how to reproduce this. ([_OpenSSLErrorWithText(code=151441516L, lib=9, func=109, reason=108, reason_text='error:0906D06C:PEM routines:PEM_read_bio:no start line'), _OpenSSLErrorWithText(code=151441516L, lib=9, func=109, reason=108, reason_text='error:0906D06C:PEM routines:PEM_read_bio:no start line')])

dependencies:
mod_ssl-2.4.23-4.fc24.x86_64
openssl-1.0.2j-1.fc24.x86_64
pyOpenSSL-16.0.0-1.fc24.noarch
python2-cryptography-1.5.3-3.fc24.x86_64

Referring to https://github.com/pyca/cryptography/issues/2699, after changing the code in "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 140
from _openssl_assert(cls.lib, cls.lib.ERR_peek_error() == 0)
to cls.lib.ERR_clear_error(), it's back to normal.

Did anyone try to deploy kojihub on Fedora 24 with latest openssl package?

Thanks


I run kojiweb on F24 routinely, so the title is incorrect

Try to fix this problem with https://pagure.io/koji/pull-request/197 and failed.

@julian8628 changed the status to Closed

7 years ago

@julian8628 changed the status to Open

7 years ago

I can't reproduce it with mentioned rpm versions. Is there any other important change in the system?

I can't reproduce it with mentioned rpm versions. Is there any other important change in the system?

My system is vagrant virtualbox F24-cloud-version vbox https://atlas.hashicorp.com/fedora/boxes/24-cloud-base

Is selinux enabled? If so, does Does 'seboolean httpd_execmem on' help, as suggested in a comment on PR#197?

Is selinux enabled? If so, does Does 'seboolean httpd_execmem on' help, as suggested in a comment on PR#197?

Disable selinux or setsebool httpd_execmem on is helpless for this problem
I upgraded pyOpenSSL to 16.2.0 and cryptography to 1.6 by pip,
this issue also occurs after httpd restart, but after several refreshing on 'https://host/koji', the error disappeared.

We're not able to reproduce it, so closing for now. If anybody else hits this issue, please reopen it.

@julian8628 changed the status to Closed

7 years ago

@julian8628 If this happens again, try adding

WSGIApplicationGroup %{GLOBAL}

in kojiweb.conf (the httpd config for kojiweb).

@mikem changed the status to Open

7 years ago

@mikem changed the status to Closed

7 years ago

When this happens, is kojiweb running on same host or different host?
I'm guessing same, but if anyone is seeing on with separate hub/web hosts I'd be interested to hear.

Is that the full exception in the initial comment? It looks like two partial tracebacks from different days with an "@" line separating them.

If I take current master branch (f8d4680), revert the workarounds
from f3a6bc8 and 20d67b5, and setsebool httpd_execmem off, then I can get a similar traceback locally.

mod_wsgi (pid=18658): Exception occurred processing WSGI script '/usr/share/koji-hub/kojixmlrpc.py'.
Traceback (most recent call last):
  File "/usr/share/koji-hub/kojixmlrpc.py", line 33, in <module>
    from koji.server import WSGIWrapper
  File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 41, in <module>
    from OpenSSL.SSL import Error as SSL_Error
  File "/usr/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import rand, crypto, SSL
  File "/usr/lib/python2.7/site-packages/OpenSSL/rand.py", line 12, in <module>
    from OpenSSL._util import (
  File "/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 234, in <module>
    Binding.init_static_locks()
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 180, in init_static_l
    cls._ensure_ffi_initialized()
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 175, in _ensure_ffi_i
    cls._register_osrandom_engine()
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 140, in _register_osr
    _openssl_assert(cls.lib, cls.lib.ERR_peek_error() == 0)
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 62, in _openssl_asser
    errors_with_text
InternalError: Unknown OpenSSL error. This error is commonly encountered when another library is not cleaning up the Op

It happens about once every four or so page loads. It does not look like it happens on the command line, even though the error is manifesting .

For me the problem seems connected to startup. I see it more right after restarting httpd. After a while I stop getting it. I figures the errors slowly exhaust the unlucky httpd instances and eventually we're left with the lucky ones that did not hit the error.

setsebool httpd_execmem on does not appear to help, at least by itself.

(This is on f24 btw)

With current head, the problem is solved for me.

If you can replicate this, please provide the full traceback from the hub logs.

after upgrading to
pyOpenSSL (16.2.0)
cryptography (1.7.2)
the errors disappear.

the bug reporting to cryptography has been fixed.

Login to comment on this ticket.

Metadata