#6972 Staging Bodhi seems to have some db connection issues
Closed: Will Not/Can Not fix a year ago Opened a year ago by bowlofeggs.

  • Describe what you need us to do:
    I'm seeing an error with staging Bodhi with communicating with the database: TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30

  • When do you need this? (YYYY/MM/DD)
    Ideally this week, so I can complete testing of Bodhi's 3.8.0 beta.

  • When is this no longer needed or useful? (YYYY/MM/DD)
    When we do not use Bodhi anymore, or if Bodhi doesn't use databases anymore :)

  • If we cannot complete your request, what is the impact?
    I won't be able to test Bodhi 3.8.0 fully.

Full traceback:

[Tue May 22 21:52:31.304414 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634] 2018-05-22 21:52:31,303 ERROR [bodhi.server][Dummy-5] Error caught.  Handling HTML response.
[Tue May 22 21:52:31.304535 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634] Traceback (most recent call last):
[Tue May 22 21:52:31.304572 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib/python2.7/site-packages/pyramid/tweens.py", line 39, in excview_tween
[Tue May 22 21:52:31.304594 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     response = handler(request)
[Tue May 22 21:52:31.304615 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib/python2.7/site-packages/pyramid/router.py", line 156, in handle_request
[Tue May 22 21:52:31.304646 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     view_name
[Tue May 22 21:52:31.304675 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib/python2.7/site-packages/pyramid/view.py", line 617, in _call_view
[Tue May 22 21:52:31.304683 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     response = view_callable(context, request)
[Tue May 22 21:52:31.304701 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib/python2.7/site-packages/pyramid/config/views.py", line 181, in __call__
[Tue May 22 21:52:31.304705 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     return view(context, request)
[Tue May 22 21:52:31.304708 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib/python2.7/site-packages/pyramid/viewderivers.py", line 389, in attr_view
[Tue May 22 21:52:31.304710 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     return view(context, request)
[Tue May 22 21:52:31.304713 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib/python2.7/site-packages/pyramid/viewderivers.py", line 367, in predicate_wrapper
[Tue May 22 21:52:31.304716 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     return view(context, request)
[Tue May 22 21:52:31.304719 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib/python2.7/site-packages/pyramid/viewderivers.py", line 300, in secured_view
[Tue May 22 21:52:31.304721 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     return view(context, request)
[Tue May 22 21:52:31.304724 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib/python2.7/site-packages/pyramid/viewderivers.py", line 438, in rendered_view
[Tue May 22 21:52:31.304727 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     result = view(context, request)
[Tue May 22 21:52:31.304729 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib/python2.7/site-packages/pyramid/viewderivers.py", line 147, in _requestonly_view
[Tue May 22 21:52:31.304732 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     response = view(request)
[Tue May 22 21:52:31.304735 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib/python2.7/site-packages/cornice/service.py", line 494, in wrapper
[Tue May 22 21:52:31.304738 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     response = view_()
[Tue May 22 21:52:31.304740 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib/python2.7/site-packages/bodhi/server/services/composes.py", line 74, in collection_get
[Tue May 22 21:52:31.304743 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     return {'composes': sorted(models.Compose.query.all())}
[Tue May 22 21:52:31.304746 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2703, in all
[Tue May 22 21:52:31.304749 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     return list(self)
[Tue May 22 21:52:31.304751 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2855, in __iter__
[Tue May 22 21:52:31.304754 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     return self._execute_and_instances(context)
[Tue May 22 21:52:31.304757 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2876, in _execute_and_instances
[Tue May 22 21:52:31.304760 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     close_with_result=True)
[Tue May 22 21:52:31.304766 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2885, in _get_bind_args
[Tue May 22 21:52:31.304769 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     **kw
[Tue May 22 21:52:31.304775 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2867, in _connection_from_session
[Tue May 22 21:52:31.304778 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     conn = self.session.connection(**kw)
[Tue May 22 21:52:31.304781 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1019, in connection
[Tue May 22 21:52:31.304783 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     execution_options=execution_options)
[Tue May 22 21:52:31.304786 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1024, in _connection_for_bind
[Tue May 22 21:52:31.304789 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     engine, execution_options)
[Tue May 22 21:52:31.304792 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind
[Tue May 22 21:52:31.304794 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     conn = bind.contextual_connect()
[Tue May 22 21:52:31.304797 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2112, in contextual_connect
[Tue May 22 21:52:31.304800 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     self._wrap_pool_connect(self.pool.connect, None),
[Tue May 22 21:52:31.304802 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect
[Tue May 22 21:52:31.304805 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     return fn()
[Tue May 22 21:52:31.304808 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 387, in connect
[Tue May 22 21:52:31.304811 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     return _ConnectionFairy._checkout(self)
[Tue May 22 21:52:31.304813 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 768, in _checkout
[Tue May 22 21:52:31.304816 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     fairy = _ConnectionRecord.checkout(pool)
[Tue May 22 21:52:31.304819 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 516, in checkout
[Tue May 22 21:52:31.304821 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     rec = pool._do_get()
[Tue May 22 21:52:31.304824 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1133, in _do_get
[Tue May 22 21:52:31.304827 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634]     (self.size(), self.overflow(), self._timeout))
[Tue May 22 21:52:31.304830 2018] [wsgi:error] [pid 15:tid 140371006846720] [client 10.131.0.1:42634] TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30

Metadata Update from @bowlofeggs:
- Issue tagged with: database, staging

a year ago

This error by itself is not an infrastructure error, this indicates that there were 5 (+10) sqlalchemy sessions open that weren't closed and a new one was attempted to be opened.
While this could be caused by an infrastrural problem like inability to connect, it is not in itself an infra issue.

Metadata Update from @puiterwijk:
- Issue close_status updated to: Will Not/Can Not fix
- Issue status updated to: Closed (was: Open)

a year ago

Login to comment on this ticket.

Metadata