fedora-infrastructure

Clone
Source Code
GIT

#12307 Staging elections kerberos issue
Closed: Fixed 2 months ago by zlopez. Opened 2 months ago by zlopez.

Closed: Fixed
Reopen Issue

Describe what you would like us to do:

After deploying the memcached update yesterday I started testing out the services. With elections I encountered the following issue:

2024-11-25 15:06:40,587 [ERROR] fedora_elections: Exception on / [GET]
Traceback (most recent call last):
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 3280, in _wrap_pool_connect
    return fn()
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 310, in connect
    return _ConnectionFairy._checkout(self)
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 868, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 476, in checkout
    rec = pool._do_get()
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get
    self._dec_overflow()
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__
    with_traceback=exc_tb,
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
    return self._create_connection()
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 256, in _create_connection
    return _ConnectionRecord(self)
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 371, in __init__
    self.__connect()
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 666, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__
    with_traceback=exc_tb,
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/engine/create.py", line 590, in connect
    return dialect.connect(*cargs, **cparams)
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 597, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/opt/app-root/lib/python3.6/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not initiate GSSAPI security context: Unspecified GSS failure.  Minor code may provide more information: Server krbtgt/FEDORAPROJECT.ORG@STG.FEDORAPROJECT.ORG not found in Kerberos database
FATAL:  password authentication failed for user "electionsuser"

When do you need this to be done by? (YYYY/MM/DD)

I noticed that the server is not correct krbtgt/FEDORAPROJECT.ORG@STG.FEDORAPROJECT.ORG instead of krbtgt/STG.FEDORAPROJECT.ORG@STG.FEDORAPROJECT.ORG, but the krb5.cfg doesn't seem to be changed for the last 4 years. And the issue doesn't seem to be related to the memcached change either.

I also noticed typo in krb5.cfg:

  .iad2.stg.fedoraproject.org = STG.FEDORAPROJECT.ORG
  iad2.stg.fedoraproject.org = STG.FEDORAPROJECT.ORG

Should be

  .stg.iad2.fedoraproject.org = STG.FEDORAPROJECT.ORG
  stg.iad2.fedoraproject.org = STG.FEDORAPROJECT.ORG

according to our server naming convention.

After fixing that (only in the project itself) the error is different, but it's still there:

Traceback (most recent call last):
  File "/opt/app-root/lib/python3.6/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/app-root/lib/python3.6/site-packages/flask/app.py", line 1518, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/app-root/lib/python3.6/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/app-root/lib/python3.6/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/opt/app-root/src/fedora_elections/__init__.py", line 234, in index
    prev_elections = models.Election.get_older_election(SESSION, now)[:5]
  File "/opt/app-root/src/fedora_elections/models.py", line 170, in get_older_election
    return query.all()
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2768, in all
    return self._iter().all()
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2906, in _iter
    execution_options={"_sa_orm_load_options": self.load_options},
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1711, in execute
    conn = self._connection_for_bind(bind)
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1553, in _connection_for_bind
    engine, execution_options
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 747, in _connection_for_bind
    conn = bind.connect()
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 3234, in connect
    return self._connection_cls(self, close_with_result=close_with_result)
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
    else engine.raw_connection()
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 3313, in raw_connection
    return self._wrap_pool_connect(self.pool.connect, _connection)
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 3284, in _wrap_pool_connect
    e, dialect, self
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2118, in _handle_dbapi_exception_noconnection
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 3280, in _wrap_pool_connect
    return fn()
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 310, in connect
    return _ConnectionFairy._checkout(self)
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 868, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 476, in checkout
    rec = pool._do_get()
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get
    self._dec_overflow()
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__
    with_traceback=exc_tb,
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
    return self._create_connection()
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 256, in _create_connection
    return _ConnectionRecord(self)
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 371, in __init__
    self.__connect()
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 666, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__
    with_traceback=exc_tb,
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/engine/create.py", line 590, in connect
    return dialect.connect(*cargs, **cparams)
  File "/opt/app-root/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 597, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/opt/app-root/lib/python3.6/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not initiate GSSAPI security context: Unspecified GSS failure.  Minor code may provide more information: Server postgres/db01.stg.iad2.fedoraproject.org@STG.FEDORAPROJECT.ORG not found in Kerberos database
FATAL:  password authentication failed for user "electionsuser"

That is definitely not a correct kerberos server. I'm not sure if this is some issue in psycopg2 library or our kerberos configuration. The elections pod was running fine till I tried to redeploy it.

@zlopez did you see any weird message / error at the domain log under /var/log/sssd/ ?

@seddik Do you think on the elections machine?

@seddik Do you think on the elections machine?

yes, just an idea if there is any refused message or something like that

The issue is that the logs are not accessible for the default user and the sudo is not allowed. So I can't really look into them. I need to find what needs to be set in OpenShift to allow you to read those files.

I was able to at least look before it crashed. The sssd folder is not even in /var/log so there isn't much to find there.

I was able to at least look before it crashed. The sssd folder is not even in /var/log so there isn't much to find there.

it really doesn't help :(

Metadata Update from @zlopez:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
2 months ago

It's nice to hear that ;)

Log in to comment on this ticket.

Metadata

low-gain Needs investigation ops

None
None
Waiting on Assignee
Boards 1
ops Status: Backlog
Powered by Pagure 5.14.1
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© Red Hat, Inc. and others.