#8211 communishift: Cannot allocate memory
Closed: Insufficient data 6 days ago by kevin. Opened a month ago by rpitonak.

We are using celery in our service and we are getting this error just in communishift. Memory limits seem to be set high enough though.

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/usr/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/usr/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 317, in start
    blueprint.start(self)
  File "/usr/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 593, in start
    c.loop(*c.loop_args())
  File "/usr/lib/python3.7/site-packages/celery/worker/loops.py", line 91, in asynloop
    next(loop)
  File "/usr/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 354, in create_loop
    cb(*cbargs)
  File "/usr/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 454, in _event_process_exit
    self.maintain_pool()
  File "/usr/lib/python3.7/site-packages/billiard/pool.py", line 1316, in maintain_pool
    sys.exc_info()[2])
  File "/usr/lib/python3.7/site-packages/billiard/five.py", line 126, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3.7/site-packages/billiard/pool.py", line 1307, in maintain_pool
    self._maintain_pool()
  File "/usr/lib/python3.7/site-packages/billiard/pool.py", line 1299, in _maintain_pool
    self._repopulate_pool(joined)
  File "/usr/lib/python3.7/site-packages/billiard/pool.py", line 1284, in _repopulate_pool
    self._create_worker_process(self._avail_index())
  File "/usr/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 449, in _create_worker_process
    return super(AsynPool, self)._create_worker_process(i)
  File "/usr/lib/python3.7/site-packages/billiard/pool.py", line 1116, in _create_worker_process
    w.start()
  File "/usr/lib/python3.7/site-packages/billiard/process.py", line 124, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.7/site-packages/billiard/context.py", line 333, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.7/site-packages/billiard/popen_fork.py", line 24, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.7/site-packages/billiard/popen_fork.py", line 72, in _launch
    self.pid = os.fork()
MemoryError: [Errno 12] Cannot allocate memory

Metadata Update from @kevin:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: communishift

a month ago

Can you duplicate this with a regular celery install not in openshift? Can you tell me what build/pod this is? Mind if I try and spin some up to debug?

There's a number of python things that could cause this error... like running out of fd's, or locks... are you sure these are not the issue?

Metadata Update from @cverna:
- Issue priority set to: Waiting on Reporter (was: Waiting on Assignee)

18 days ago

@rpitonak If you are still seeing this, feel free to re-open and provide more information.

Thanks!

Metadata Update from @kevin:
- Issue close_status updated to: Insufficient data
- Issue status updated to: Closed (was: Open)

6 days ago

Login to comment on this ticket.

Metadata