fedora-infrastructure

#7067 Staging OpenShift fails to deploy, also can't get logs
Closed: Fixed Opened by dcallagh.

Closed: Fixed
Reopen Issue
  • Describe what you need us to do:
    I am trying to deploy the latest Waiverdb in stage. Yesterday the deployment ran but was rolled back because of bad data in the db, which has now been fixed (#7061). But now when I trigger a deployment again there seems to be something wrong with OpenShift. It is not doing the deployment at all.

The deployment immediately goes to:

  - lastTransitionTime: 2018-06-29T05:19:46Z
    lastUpdateTime: 2018-06-29T05:19:46Z
    message: replication controller "waiverdb-web-19" has failed progressing
    reason: ProgressDeadlineExceeded
    status: "False"
    type: Progressing

which seems odd. And when I try to get the logs for it, which would normally indicate why it is crashing, I get this error:

$ oc -n waiverdb log dc/waiverdb-web -f
error: couldn't get deployment waiverdb-web-19: Get https://172.30.0.1:443/api/v1/namespaces/waiverdb/replicationcontrollers/waiverdb-web-19: dial tcp 172.30.0.1:443: getsockopt: no route to host

Since I don't have permission to see anything in the OpenShift console itself, it's hard to guess what might be going wrong. But it seems like some or all of the cluster nodes are not working properly?

  • When do you need this? (YYYY/MM/DD)

  • When is this no longer needed or useful? (YYYY/MM/DD)

  • If we cannot complete your request, what is the impact?
    cannot deploy Waiverdb

Fixed. Please do confirm you can deploy and see the logs, but I think it's all fixed now.

:factory:

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Thanks Kevin! Indeed I can see that waiverdb has rolled out successfully now.

But I tried doing the same for Greenwave and I think there are still some issues...

The oc log command works now at least, but the deployment still "failed to make progress":

$ oc -n greenwave log dc/greenwave-web
--> Scaling up greenwave-web-19 from 0 to 2, scaling down greenwave-web-17 from 2 to 0 (keep 2 pods available, don't exceed 3 pods)
    Scaling greenwave-web-19 up to 1
error: timed out waiting for any update progress to be made

With that specific error, that means you should check the logs for the spawned pods.
Since this means that the spawning pods never finish their Readiness probe, and openshift reverts the deployment.

For example, in this case:

$ oc logs greenwave-web-17-rvvwv
.....
Traceback (most recent call last):
  File "/usr/bin/gunicorn-3", line 11, in <module>
    load_entry_point('gunicorn==19.7.1', 'console_scripts', 'gunicorn')()
  File "/usr/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 74, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
  File "/usr/lib/python3.6/site-packages/gunicorn/app/base.py", line 203, in run
    super(Application, self).run()
  File "/usr/lib/python3.6/site-packages/gunicorn/app/base.py", line 72, in run
    Arbiter(self).run()
  File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 231, in run
    self.halt(reason=inst.reason, exit_status=inst.exit_status)
  File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 344, in halt
    self.stop()
  File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 393, in stop
    time.sleep(0.1)
  File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 244, in handle_chld
    self.reap_workers()
  File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 524, in reap_workers
    raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
[2018-07-02 00:39:53 +0000] [11] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/gunicorn/arbiter.py", line 578, in spawn_worker
    worker.init_process()
  File "/usr/lib/python3.6/site-packages/gunicorn/workers/base.py", line 126, in init_process
    self.load_wsgi()
  File "/usr/lib/python3.6/site-packages/gunicorn/workers/base.py", line 135, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/usr/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/lib/python3.6/site-packages/gunicorn/util.py", line 352, in import_app
    __import__(module)
  File "/usr/lib/python3.6/site-packages/greenwave/wsgi.py", line 7, in <module>
    app = create_app()
  File "/usr/lib/python3.6/site-packages/greenwave/app_factory.py", line 16, in create_app
    app.config.update(load_config(config_obj))
  File "/usr/lib/python3.6/site-packages/greenwave/utils.py", line 106, in load_config
    config['policies'] = load_policies(config['POLICIES_DIR'])
  File "/usr/lib/python3.6/site-packages/greenwave/utils.py", line 123, in load_policies
    greenwave.policies.validate_policies(policies)
  File "/usr/lib/python3.6/site-packages/greenwave/policies.py", line 24, in validate_policies
    'is missing attribute %s' % (policy.id, required_attribute))
RuntimeError: Policies are not configured properly as policy openqa_important_stuff_for_rawhide is missing attribute subject_type
[2018-07-02 00:39:53 +0000] [11] [INFO] Worker exiting (pid: 11)

Are you sure though? oc get pod does not show any pods from greenwave-web-19.

greenwave-web-17 is the previous deploy from 3 days ago.

Having said that, the error from deployment 17 is real and I will need to fix that.

Yes, because after it aborts the deployment (because it failed to make progress), it clears up the new generation pods.
When I just did a new deployment:

oc logs -f pod/greenwave-web-20-fwprc
....
RuntimeError: Policies are not configured properly as policy openqa_important_stuff_for_rawhide is missing attribute subject_type
....

In a few minutes, that pod will be gone, because it didn't make progress, just like 18 and 19.

Oh, I see. For some reason I thought that oc log dc/greenwave-web -f would show me the errors from the pods if they were failing to start up. But it seems not to be the case. Thanks for the help @puiterwijk .
Metadata
None
None
None
None
Lowest
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.