#615 ODCS 0.5.0 rejects all requests
Opened 2 years ago by sgallagh. Modified 2 years ago

Last week, Fedora Infrastructure upgraded ODCS to 0.5.0, after which all requests to start a compose were rejected. @kevin downgraded the production systems back to 0.4.0, but we need to figure out what went wrong there.


I don't have access to the logs at this point, but by looking at the differences between 0.4.0 and 0.5.0, my hunch is that it's related to the GSSAPI changes.

I don't know much about the fedora odcs instance and have some questions. Which auth backend is used? oidcs or kerberos? What errors are returned to client after upgraded to 0.5.0?
Is https://odcs.fedoraproject.org/api/1/about/ the instance you are going to upgrade? I found that it's still running 0.3.6.

The front-end service is still on 0.3.6. The backend was updated to 0.5.0 (but indeed, we probably ought to make sure those remain in sync)

The configuration is using oidcs:

class ProdConfiguration(BaseConfiguration):
    AUTH_BACKEND = 'openidc'
    AUTH_OPENIDC_REQUIRED_SCOPES = [
        'openid',
        'https://id.fedoraproject.org/scope/groups',
        'https://pagure.io/odcs/new-compose',
        'https://pagure.io/odcs/renew-compose',
        'https://pagure.io/odcs/delete-compose',
    ]
    OIDC_BASE_NAMESPACE = 'https://pagure.io/odcs/'

    AUTH_OPENIDC_USERINFO_URI = 'https://id.fedoraproject.org/openidc/UserInfo'

I dug back into the logs and found this:

Apr 11 15:43:15 odcs-backend-releng01.iad2.fedoraproject.org systemd[1]: Starting odcs-celery-backend.service - ODCS Celery backend service...
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]: Usage: python -m celery [OPTIONS] COMMAND [ARGS]...
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]: Try 'python -m celery --help' for help.
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]: Error: Invalid value for '-A' / '--app':
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]: Unable to load celery application.
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]: While trying to load the module odcs.server.celery_tasks the following error occurred:
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]: Traceback (most recent call last):
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:   File "/usr/lib/python3.10/site-packages/celery/bin/celery.py", line 57, in convert
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:     return find_app(value)
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:   File "/usr/lib/python3.10/site-packages/celery/app/utils.py", line 384, in find_app
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:     sym = symbol_by_name(app, imp=imp)
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:   File "/usr/lib/python3.10/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:     module = imp(module_name, package=package, **kwargs)
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:   File "/usr/lib/python3.10/site-packages/celery/utils/imports.py", line 105, in import_from_cwd
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:     return imp(module, package=package)
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:   File "/usr/lib64/python3.10/importlib/__init__.py", line 126, in import_module
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:     return _bootstrap._gcd_import(name[level:], package, level)
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:   File "/usr/lib/python3.10/site-packages/odcs/server/__init__.py", line 56, in <module>
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:     from odcs.server import views  # noqa
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:   File "/usr/lib/python3.10/site-packages/odcs/server/views.py", line 67, in <module>
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]:     app.openapispec = openapi.spec if openapi else None
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000838]: NameError: name 'openapi' is not defined
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000837]: celery multi v5.2.6 (dawn-chorus)
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000837]: > Starting nodes...
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000837]:         > worker@odcs-backend-releng01.iad2.fedoraproject.org: * Child terminated with exit code 2
Apr 11 15:43:16 odcs-backend-releng01.iad2.fedoraproject.org sh[2000837]: FAILED

I suspect that https://pagure.io/odcs/c/4f92d82ef72d1758c50839e1464203cf1f0b9e30?branch=master is incomplete.

/usr/lib/python3.10/site-packages/odcs/server/views.py has been modified on odcs-backend-releng01.iad2.fedoraproject.org, so the traceback above may not be really representative. The import of apispec is commented out and app.openapispec hardcoded to None. The traceback is likely from when the change was only halfway finished.

The VM has odcs-0.4.0-2.fc36.noarch right now and seems to be working fine.

I don't see any obvious issue in the changes between 0.4 and 0.5.

  • The auth changes regarding gssapi were only on client side.
  • There are changes to how pulp composes work, which is irrelevant for Fedora.
  • The config loading is slightly changed. It is supposed to be backwards compatible, but there may be some bug there.

/usr/lib/python3.10/site-packages/odcs/server/views.py has been modified on odcs-backend-releng01.iad2.fedoraproject.org, so the traceback above may not be really representative. The import of apispec is commented out and app.openapispec hardcoded to None. The traceback is likely from when the change was only halfway finished.

I did this to get composes flowing again. This is a hack after downgrading to 0.4.0.

The VM has odcs-0.4.0-2.fc36.noarch right now and seems to be working fine.

It worked after commenting out that section to make it not try to import openapi.

I don't see any obvious issue in the changes between 0.4 and 0.5.

0.4.0 also doesn't seem to work without commenting that import in that file.

  • The auth changes regarding gssapi were only on client side.
  • There are changes to how pulp composes work, which is irrelevant for Fedora.
  • The config loading is slightly changed. It is supposed to be backwards compatible, but there may be some bug there.

We can upgrade it again for debugging, but I'd defer to @sgallagh there.

Go ahead and upgrade it for debugging.

Log in to comment on this ticket.

Metadata