I'm trying to push an update to golang-github-prometheus-alertmanager, but it fails with the following error:
golang-github-prometheus-alertmanager/rawhide> $ fedpkg push; and fedpkg build Enumerating objects: 13, done. Counting objects: 100% (13/13), done. Delta compression using up to 16 threads Compressing objects: 100% (9/9), done. Writing objects: 100% (9/9), 1.46 KiB | 1.46 MiB/s, done. Total 9 (delta 4), reused 0 (delta 0), pack-reused 0 remote: Traceback (most recent call last): remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect remote: return fn() remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 354, in connect remote: return _ConnectionFairy._checkout(self) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 751, in _checkout remote: fairy = _ConnectionRecord.checkout(pool) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 483, in checkout remote: rec = pool._do_get() remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 138, in _do_get remote: self._dec_overflow() remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__ remote: compat.reraise(exc_type, exc_value, exc_tb) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 129, in reraise remote: raise value remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 135, in _do_get remote: return self._create_connection() remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 299, in _create_connection remote: return _ConnectionRecord(self) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 428, in __init__ remote: self.__connect(first_connect_check=True) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 630, in __connect remote: connection = pool._invoke_creator(self) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect remote: return dialect.connect(*cargs, **cparams) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 453, in connect remote: return self.dbapi.connect(*cargs, **cparams) remote: File "/usr/lib64/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect remote: conn = _connect(dsn, connection_factory=connection_factory, **kwasync) remote: psycopg2.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections remote: remote: remote: The above exception was the direct cause of the following exception: remote: remote: Traceback (most recent call last): remote: File "hooks/pre-receive", line 48, in <module> remote: run_hook_file(hooktype) remote: File "/usr/lib/python3.6/site-packages/pagure/hooks/__init__.py", line 531, in run_hook_file remote: session, repo, user=username, namespace=namespace remote: File "/usr/lib/python3.6/site-packages/pagure/lib/query.py", line 2865, in _get_project remote: return query.one() remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3275, in one remote: ret = self.one_or_none() remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3244, in one_or_none remote: ret = list(self) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3317, in __iter__ remote: return self._execute_and_instances(context) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3339, in _execute_and_instances remote: querycontext, self._connection_from_session, close_with_result=True remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3354, in _get_bind_args remote: mapper=self._bind_mapper(), clause=querycontext.statement, **kw remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3332, in _connection_from_session remote: conn = self.session.connection(**kw) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1123, in connection remote: execution_options=execution_options, remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1129, in _connection_for_bind remote: engine, execution_options remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 430, in _connection_for_bind remote: conn = bind._contextual_connect() remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect remote: self._wrap_pool_connect(self.pool.connect, None), remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect remote: e, dialect, self remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1536, in _handle_dbapi_exception_noconnection remote: util.raise_from_cause(sqlalchemy_exception, exc_info) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause remote: reraise(type(exception), exception, tb=exc_tb, cause=cause) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 128, in reraise remote: raise value.with_traceback(tb) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect remote: return fn() remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 354, in connect remote: return _ConnectionFairy._checkout(self) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 751, in _checkout remote: fairy = _ConnectionRecord.checkout(pool) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 483, in checkout remote: rec = pool._do_get() remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 138, in _do_get remote: self._dec_overflow() remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__ remote: compat.reraise(exc_type, exc_value, exc_tb) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 129, in reraise remote: raise value remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/impl.py", line 135, in _do_get remote: return self._create_connection() remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 299, in _create_connection remote: return _ConnectionRecord(self) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 428, in __init__ remote: self.__connect(first_connect_check=True) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 630, in __connect remote: connection = pool._invoke_creator(self) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect remote: return dialect.connect(*cargs, **cparams) remote: File "/usr/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 453, in connect remote: return self.dbapi.connect(*cargs, **cparams) remote: File "/usr/lib64/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect remote: conn = _connect(dsn, connection_factory=connection_factory, **kwasync) remote: sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: remaining connection slots are reserved for non-replication superuser connections remote: remote: (Background on this error at: http://sqlalche.me/e/e3q8) To ssh://pkgs.fedoraproject.org/rpms/golang-github-prometheus-alertmanager ! [remote rejected] rawhide -> rawhide (pre-receive hook declined) error: failed to push some refs to 'ssh://pkgs.fedoraproject.org/rpms/golang-github-prometheus-alertmanager' Could not execute push: Failed to execute command.
Some times it only fails with:
$ fedpkg push; and fedpkg build Enumerating objects: 13, done. Counting objects: 100% (13/13), done. Delta compression using up to 16 threads Compressing objects: 100% (9/9), done. Writing objects: 100% (9/9), 1.46 KiB | 1.46 MiB/s, done. Total 9 (delta 4), reused 0 (delta 0), pack-reused 0 remote: Unspecified ref refs/heads/rawhide is blocked remote: Denied push for ref 'refs/heads/rawhide' for user 'eclipseo' remote: All changes have been rejected To ssh://pkgs.fedoraproject.org/rpms/golang-github-prometheus-alertmanager ! [remote rejected] rawhide -> rawhide (pre-receive hook declined) error: failed to push some refs to 'ssh://pkgs.fedoraproject.org/rpms/golang-github-prometheus-alertmanager' Could not execute push: Failed to execute command.
It only happens with the repo golang-github-prometheus-alertmanager.
As a workaround, I used a PR to merge my changes.
This was caused by mirrormanager crawler taking up ~1400 slots. ;(
@adrian any idea why the crawler is not freeing up db slots now? Whats changed there?
I restarted postgres and killed the old mm crawlers, so everything should be ok for a while now, but we need to fix the underlying issue.
Metadata Update from @mohanboddu: - Issue tagged with: medium-gain, medium-trouble, ops
Metadata Update from @mohanboddu: - Issue priority set to: Waiting on Assignee (was: Needs Review)
We have not seen this in the last month or so... so I guess it must have been caused by something we already fixed?
Will re-open if it happens again.
Metadata Update from @kevin: - Issue close_status updated to: Insufficient data - Issue status updated to: Closed (was: Open)
Issue status updated to: Open (was: Closed)
Metadata Update from @jbley: - Issue close_status updated to: Insufficient data - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.