Issue #2481: koji.util.rmtree() is not multi-process/thread safe - koji

koji

#2481 koji.util.rmtree() is not multi-process/thread safe

Closed: Fixed 3 years ago by tkopecek. Opened 3 years ago by julian8628.

from https://pagure.io/koji/issue/2361#comment-677216:

more race condition problem found in rmtree()

test script:

import os
from multiprocessing import Process
import koji

_DIR = 'brew-testing-dir'


def _mkdirs():
    os.mkdir(_DIR)
    for i in range(0, 100):
        dd = os.path.join(_DIR, 'subdir_' + str(i))
        os.mkdir(dd)
        for j in range(0, 100):
            ddd = os.path.join(dd, 'subdir_' + str(j))
            os.mkdir(ddd)


def test_koji_util_rmtree_parallel():
    _mkdirs()
    tasks = [Process(target=koji.util.rmtree, args=(_DIR,))
             for i in range(0, 10)]
    for task in tasks:
        task.start()
    for task in tasks:
        task.join()

if __name__ == '__main__':
    test_koji_util_rmtree_parallel()

errors:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/yzhu/git/redhat/fork/koji/koji/util.py", line 442, in rmtree
    _rmtree(dev)
  File "/home/yzhu/git/redhat/fork/koji/koji/util.py", line 451, in _rmtree
    dirs = _stripcwd(dev)
  File "/home/yzhu/git/redhat/fork/koji/koji/util.py", line 475, in _stripcwd
    for fn in os.listdir('.'):
OSError: [Errno 116] Stale file handle: '.'

Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/yzhu/git/redhat/fork/koji/koji/util.py", line 442, in rmtree
    _rmtree(dev)
Process Process-1:
  File "/home/yzhu/git/redhat/fork/koji/koji/util.py", line 469, in _rmtree
    os.chdir(subdir)
FileNotFoundError: [Errno 2] No such file or directory: 'subdir_60'

Metadata Update from @tkopecek:
- Custom field Size adjusted to None
- Issue assigned to julian8628

3 years ago

mikem commented 3 years ago

In what situations do we legitimately have parallel rmtrees for the same path?

Do we have an actual reported problem due to this behavior, or it is just this script?

Edited 3 years ago by mikem

julian8628 commented 3 years ago

AFAIK,
- in kojira, it is possible to call rmtree() for the same path (it appeared in the testing of PR #2443)
- in issue #2361 the race condition happened in _stripcwd(), but it is still possible to meet this problem in other places in rmtree()