We are using celery in our service and we are getting this error just in communishift. Memory limits seem to be set high enough though.
Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start self.blueprint.start(self) File "/usr/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start step.start(parent) File "/usr/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start return self.obj.start() File "/usr/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 317, in start blueprint.start(self) File "/usr/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start step.start(parent) File "/usr/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 593, in start c.loop(*c.loop_args()) File "/usr/lib/python3.7/site-packages/celery/worker/loops.py", line 91, in asynloop next(loop) File "/usr/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 354, in create_loop cb(*cbargs) File "/usr/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 454, in _event_process_exit self.maintain_pool() File "/usr/lib/python3.7/site-packages/billiard/pool.py", line 1316, in maintain_pool sys.exc_info()[2]) File "/usr/lib/python3.7/site-packages/billiard/five.py", line 126, in reraise raise value.with_traceback(tb) File "/usr/lib/python3.7/site-packages/billiard/pool.py", line 1307, in maintain_pool self._maintain_pool() File "/usr/lib/python3.7/site-packages/billiard/pool.py", line 1299, in _maintain_pool self._repopulate_pool(joined) File "/usr/lib/python3.7/site-packages/billiard/pool.py", line 1284, in _repopulate_pool self._create_worker_process(self._avail_index()) File "/usr/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 449, in _create_worker_process return super(AsynPool, self)._create_worker_process(i) File "/usr/lib/python3.7/site-packages/billiard/pool.py", line 1116, in _create_worker_process w.start() File "/usr/lib/python3.7/site-packages/billiard/process.py", line 124, in start self._popen = self._Popen(self) File "/usr/lib/python3.7/site-packages/billiard/context.py", line 333, in _Popen return Popen(process_obj) File "/usr/lib/python3.7/site-packages/billiard/popen_fork.py", line 24, in __init__ self._launch(process_obj) File "/usr/lib/python3.7/site-packages/billiard/popen_fork.py", line 72, in _launch self.pid = os.fork() MemoryError: [Errno 12] Cannot allocate memory
Metadata Update from @kevin: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: communishift
Can you duplicate this with a regular celery install not in openshift? Can you tell me what build/pod this is? Mind if I try and spin some up to debug?
There's a number of python things that could cause this error... like running out of fd's, or locks... are you sure these are not the issue?
Any news here @rpitonak ?
Metadata Update from @cverna: - Issue priority set to: Waiting on Reporter (was: Waiting on Assignee)
@rpitonak If you are still seeing this, feel free to re-open and provide more information.
Thanks!
Metadata Update from @kevin: - Issue close_status updated to: Insufficient data - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.