Please, schedule a Bodhi upgrade to 7.0.x. The steps are:
staging
infra-stg
Before branching F38, so releng can start using the new frozen release state. I will open a ticket on releng to inform to adjust the SOPs.
frozen
Do note that this is on my list, I simply had 0 time to do it last week.
Also, I need to sort out sigul for f37 before I can upgrade bodhi-backend01 to f37.
Sure, I just thought it was nice to have a tracker ticket to monitor the process.
About sigul, I've started to review the un-retirement of python-nss [1] to fix the FTB/FTI.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=2133080
Metadata Update from @zlopez: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: high-gain, medium-trouble, ops
ok. I upgraded bodhi-backend01 to f37. (using python-nss build in infra tags).
I then merged your pr and build and deployed bodhi in openshift stg.
Unfortunately it seems to crash on start:
config: ./gunicorn.conf.py wsgi_app: None bind: ['0.0.0.0:8080'] backlog: 2048 workers: 4 worker_class: gthread threads: 2 worker_connections: 1000 max_requests: 0 max_requests_jitter: 0 timeout: 30 graceful_timeout: 30 keepalive: 2 limit_request_line: 4094 limit_request_fields: 100 limit_request_field_size: 8190 reload: False reload_engine: auto reload_extra_files: [] spew: False check_config: False print_config: False preload_app: True sendfile: None reuse_port: False chdir: / daemon: False raw_env: [] pidfile: None worker_tmp_dir: None user: 1000760000 group: 0 umask: 0 initgroups: False tmp_upload_dir: None secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'} forwarded_allow_ips: ['127.0.0.1'] accesslog: None disable_redirect_access_to_syslog: False access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s" errorlog: - loglevel: DEBUG capture_output: False logger_class: gunicorn.glogging.Logger logconfig: None logconfig_dict: {} syslog_addr: unix:///dev/log syslog: False syslog_prefix: None syslog_facility: user enable_stdio_inheritance: False statsd_host: None dogstatsd_tags: statsd_prefix: proc_name: None default_proc_name: /etc/bodhi/production.ini pythonpath: None paste: /etc/bodhi/production.ini on_starting: <function OnStarting.on_starting at 0x7f3f7cb9e480> on_reload: <function OnReload.on_reload at 0x7f3f7cb9e5c0> when_ready: <function WhenReady.when_ready at 0x7f3f7cb9e700> pre_fork: <function Prefork.pre_fork at 0x7f3f7cb9e840> post_fork: <function Postfork.post_fork at 0x7f3f7cb9e980> post_worker_init: <function PostWorkerInit.post_worker_init at 0x7f3f7cb9eac0> worker_int: <function WorkerInt.worker_int at 0x7f3f7cb9ec00> worker_abort: <function WorkerAbort.worker_abort at 0x7f3f7cb9ed40> pre_exec: <function PreExec.pre_exec at 0x7f3f7cb9ee80> pre_request: <function PreRequest.pre_request at 0x7f3f7cb9efc0> post_request: <function PostRequest.post_request at 0x7f3f7cb9f060> child_exit: <function ChildExit.child_exit at 0x7f3f7cb9f1a0> worker_exit: <function WorkerExit.worker_exit at 0x7f3f7cb9f2e0> nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7f3f7cb9f420> on_exit: <function OnExit.on_exit at 0x7f3f7cb9f560> proxy_protocol: False proxy_allow_ips: ['127.0.0.1'] keyfile: None certfile: None ssl_version: 2 cert_reqs: 0 ca_certs: None suppress_ragged_eofs: True do_handshake_on_connect: False ciphers: None raw_paste_global_conf: [] strip_header_spaces: False 2022-12-07 01:05:27,246 INFO [bodhi.server][MainThread] Warming up caches… Traceback (most recent call last): File "/usr/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context self.dialect.do_execute( File "/usr/lib64/python3.11/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute cursor.execute(statement, parameters) psycopg2.errors.UndefinedColumn: column updates.critpath_groups does not exist LINE 2: ...tes_pushed, updates.critpath AS updates_critpath, updates.cr... ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/bin/gunicorn", line 8, in <module> sys.exit(run()) ^^^^^ File "/usr/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 67, in run WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run() File "/usr/lib/python3.11/site-packages/gunicorn/app/base.py", line 231, in run super().run() File "/usr/lib/python3.11/site-packages/gunicorn/app/base.py", line 72, in run Arbiter(self).run() ^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/gunicorn/arbiter.py", line 58, in __init__ self.setup(app) File "/usr/lib/python3.11/site-packages/gunicorn/arbiter.py", line 118, in setup self.app.wsgi() File "/usr/lib/python3.11/site-packages/gunicorn/app/base.py", line 67, in wsgi self.callable = self.load() ^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 56, in load return self.load_pasteapp() ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_pasteapp return get_wsgi_app(self.app_uri, defaults=self.cfg.paste_global_conf) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/gunicorn/app/pasterapp.py", line 19, in get_wsgi_app return loadapp( ^^^^^^^^ File "/usr/lib/python3.11/site-packages/paste/deploy/loadwsgi.py", line 253, in loadapp return loadobj(APP, uri, name=name, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/paste/deploy/loadwsgi.py", line 278, in loadobj return context.create() ^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/paste/deploy/loadwsgi.py", line 715, in create return self.object_type.invoke(self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/paste/deploy/loadwsgi.py", line 235, in invoke filtered = context.next_context.create() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/paste/deploy/loadwsgi.py", line 715, in create return self.object_type.invoke(self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/paste/deploy/loadwsgi.py", line 152, in invoke return fix_call(context.object, context.global_conf, **context.local_conf) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/paste/deploy/util.py", line 55, in fix_call val = callable(*args, **kw) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/bodhi/server/__init__.py", line 370, in main generic._generate_home_page_stats() File "/usr/lib/python3.11/site-packages/decorator.py", line 232, in fun return caller(func, *(extras + args), **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func return self.get_or_create( ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/dogpile/cache/region.py", line 1042, in get_or_create with Lock( File "/usr/lib/python3.11/site-packages/dogpile/lock.py", line 185, in __enter__ return self._enter() ^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/dogpile/lock.py", line 94, in _enter generated = self._enter_create(value, createdtime) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/dogpile/lock.py", line 178, in _enter_create return self.creator() ^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/dogpile/cache/region.py", line 995, in gen_value created_value = creator( ^^^^^^^^ File "/usr/lib/python3.11/site-packages/bodhi/server/views/generic.py", line 153, in _generate_home_page_stats "critpath_testing_count": get_testing_counts(True, False), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/bodhi/server/views/generic.py", line 123, in get_testing_counts return query.count() ^^^^^^^^^^^^^ File "/usr/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 3175, in count return self._from_self(col).enable_eagerloads(False).scalar() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 2892, in scalar ret = self.one() ^^^^^^^^^^ File "/usr/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 2869, in one return self._iter().one() ^^^^^^^^^^^^ File "/usr/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 2915, in _iter result = self.session.execute( ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/site-packages/sqlalchemy/orm/session.py", line 1714, in execute result = conn._execute_20(statement, params or {}, execution_options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 return meth(self, args_10style, kwargs_10style, execution_options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection return connection._execute_clauseelement( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement ret = self._execute_context( ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context self._handle_dbapi_exception( File "/usr/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception util.raise_( File "/usr/lib64/python3.11/site-packages/sqlalchemy/util/compat.py", line 211, in raise_ raise exception File "/usr/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context self.dialect.do_execute( File "/usr/lib64/python3.11/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column updates.critpath_groups does not exist LINE 2: ...tes_pushed, updates.critpath AS updates_critpath, updates.cr... ^ [SQL: SELECT count(*) AS count_1 FROM (SELECT updates.id AS updates_id, updates.autokarma AS updates_autokarma, updates.autotime AS updates_autotime, updates.stable_karma AS updates_stable_karma, updates.stable_days AS updates_stable_days, updates.unstable_karma AS updates_unstable_karma, updates.requirements AS updates_requirements, updates.require_bugs AS updates_require_bugs, updates.require_testcases AS updates_require_testcases, updates.display_name AS updates_display_name, updates.notes AS updates_notes, updates.type AS updates_type, updates.status AS updates_status, updates.request AS updates_request, updates.severity AS updates_severity, updates.suggest AS updates_suggest, updates.locked AS updates_locked, updates.pushed AS updates_pushed, updates.critpath AS updates_critpath, updates.critpath_groups AS updates_critpath_groups, updates.close_bugs AS updates_close_bugs, updates.date_submitted AS updates_date_submitted, updates.date_modified AS updates_date_modified, updates.date_approved AS updates_date_approved, updates.date_testing AS updates_date_testing, updates.date_stable AS updates_date_stable, updates.alias AS updates_alias, updates.release_id AS updates_release_id, updates.user_id AS updates_user_id, updates.test_gating_status AS updates_test_gating_status, updates.from_tag AS updates_from_tag FROM updates WHERE updates.critpath = true AND updates.status = %(status_1)s ORDER BY updates.date_submitted DESC) AS anon_1] [parameters: {'status_1': 'testing'}] (Background on this error at: https://sqlalche.me/e/14/f405)
seems a db thing. Should I sync the db from prod?
Oh, I did a 'oc rollback bodhi-web' on it for now, so it's the older version running.
I think it fails because it needs a db upgrade. There are four playbooks to be run:
# Run the bodhi-backend playbook to ensure everything is up to date $ sudo rbac-playbook -l staging groups/bodhi-backend.yml # Synchronize the database from production to staging $ sudo rbac-playbook manual/staging-sync/bodhi.yml -l staging # Upgrade the Bodhi backend on staging $ sudo rbac-playbook manual/upgrade/bodhi.yml -l staging # Upgrade the Bodhi frontend on staging $ sudo rbac-playbook openshift-apps/bodhi.yml -l staging
(do note that Releng SOPs are a bit different from the release process described in Bodhi docs)
Yeah, I forgot to run those. ;)
Ran them now, but... still crashing. but different traceback. :)
2022-12-07 20:03:54,129 INFO [bodhi.server][MainThread] Warming up caches… Traceback (most recent call last): File "/usr/lib64/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context self.dialect.do_execute( File "/usr/lib64/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute cursor.execute(statement, parameters) psycopg2.errors.UndefinedColumn: column updates.date_pushed does not exist LINE 2: ..., updates.date_approved AS updates_date_approved, updates.da... ^ The above exception was the direct cause of the following exception: ...
That's odd... it is right that it can't find date_pushed column, as it was removed from db. But what I find even more odd is that python3.10: F37 should have python 3.11, so it seems it still using the old base image...
date_pushed
python3.10
Let me guess (I'm trying to guess how ansible works): does it fail on the third playbook (manual/upgrade/bodhi.yml -l staging)? If so, I guess it fails because the third playbook runs migrations on the database and at the end tries to reload the frontend, but the frontend is updated only in the fourth playbook...
manual/upgrade/bodhi.yml -l staging
None of them errored that I recall...
But if it's running the old migrations it would have just done nothing and blew up when the new app tried to run?
I ran though them again and this time it seems to be happy. So perhaps it just ran the right migrations after the initial failed deployment?
Great news, thanks for taking care of it.
Shall we move to prod now? Or is there further to do?
I'll prepare a final 7.0.1 release with some small bugfixes in the weekend, then I'll try to build the RPMs in Koji (I may have to disable tests in the specfile due to https://bugzilla.redhat.com/show_bug.cgi?id=2160515 ).
There has been almost no testing on staging about the new frozen release state and the modified push... I suppose we will find out if anything is broken when deployed to prod :frowning:
I have built the 7.0.1 RPMs for F37 and F36 (the latest are building now), so you can proceed updating prod at your convenience. I've disabled auto-push of the updates to avoid unexpected updates from happening in case of a base image rebuild.
And... we are live in prod. ;)
Let me know if you see any problems.
I'll watch updates pushes here in a few and confirm they go ok.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Thank you! I'll notify that on devel mailing list.
Login to comment on this ticket.