Issue #2698: rmtree can fail with mysterious error in race - koji

koji

#2698 rmtree can fail with mysterious error in race

Closed: Fixed 3 years ago by tkopecek. Opened 3 years ago by mikem.

I've seen some bad behavior from the updated rmtree code. Here is a trace of rmtree hitting a "phantom error" that leaves parts of the directory intact.

koji-clean-repos2(171):         while not dirs and dirstack:
koji-clean-repos2(172):             try:
koji-clean-repos2(173):                 os.chdir('..')

We fail to chdir up

koji-clean-repos2(181):             except OSError as e:
koji-clean-repos2(182):                 if e.errno in (errno.ENOENT, errno.ESTALE):
koji-clean-repos2(184):                     dirstack = []

side note: calling _stripcwd here makes no sense

koji-clean-repos2(185):                     dirs = _stripcwd(dev)
 --- modulename: koji-clean-repos2, funcname: _stripcwd
koji-clean-repos2(221):     dirs = []
koji-clean-repos2(222):     try:
koji-clean-repos2(223):         fdirs = os.listdir('.')
koji-clean-repos2(224):     except OSError as e:
koji-clean-repos2(226):         if e.errno in (errno.ENOENT, errno.ESTALE):
koji-clean-repos2(227):             return dirs

_stripcwd returns [] after it fails to listdir
we chdir to root

koji-clean-repos2(186):                     try:
koji-clean-repos2(187):                         os.chdir(root)

BUT! we've cleared dirs and dirstack, so the code thinks we're done

koji-clean-repos2(171):         while not dirs and dirstack:
koji-clean-repos2(195):         if not dirs:
koji-clean-repos2(197):             break

Then we hit the rmtree finally clause

koji-clean-repos2(155):         os.chdir(cwd)
koji-clean-repos2(156):     try:
koji-clean-repos2(157):         os.rmdir(path)
koji-clean-repos2(158):     except OSError as e:
koji-clean-repos2(159):         if e.errno == errno.ENOTEMPTY:
koji-clean-repos2(160):             logger.warning('%s path is not empty, but it may be a phantom error caused by some'
koji-clean-repos2(161):                            ' race condition', path, exc_info=True)

Note that the the code is returning without error even though it knows the directory is not empty. This is definitely an error condition and should not be ignored.