#1182 FileExistsError: [Errno 17] File exists
Closed: Invalid 4 years ago by lsedlar. Opened 4 years ago by bowlofeggs.

I hit an error composing the F29-updates-testing repo in staging:

https://kojipkgs.stg.fedoraproject.org/compose/updates/Fedora-29-updates-testing-20190506.4/logs/global/traceback.global.log

Traceback (most recent call last):
  File "/usr/bin/pungi-koji", line 482, in <module>
    main()
  File "/usr/bin/pungi-koji", line 261, in main
    run_compose(compose, create_latest_link=create_latest_link, latest_link_status=latest_link_status)
  File "/usr/bin/pungi-koji", line 381, in run_compose
    essentials_phase.stop()
  File "/usr/lib/python3.7/site-packages/pungi/phases/weaver.py", line 53, in stop
    self.pool.stop()
  File "/usr/lib/python3.7/site-packages/kobo/threads.py", line 120, in stop
    six.reraise(exc_info[0], exc_info[1], exc_info[2])
  File "/usr/lib/python3.7/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.7/site-packages/kobo/threads.py", line 67, in run
    self.process(item, num)
  File "/usr/lib/python3.7/site-packages/pungi/phases/weaver.py", line 69, in process
    phase.start()
  File "/usr/lib/python3.7/site-packages/pungi/phases/base.py", line 64, in start
    self.run()
  File "/usr/lib/python3.7/site-packages/pungi/phases/gather/__init__.py", line 111, in run
    manifest=self.manifest)
  File "/usr/lib/python3.7/site-packages/pungi/phases/gather/link.py", line 122, in link_files
    pool.stop()
  File "/usr/lib/python3.7/site-packages/kobo/threads.py", line 120, in stop
    six.reraise(exc_info[0], exc_info[1], exc_info[2])
  File "/usr/lib/python3.7/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.7/site-packages/kobo/threads.py", line 67, in run
    self.process(item, num)
  File "/usr/lib/python3.7/site-packages/pungi/linker.py", line 63, in process
    self.pool.linker.link(src, dst, link_type=self.pool.link_type)
  File "/usr/lib/python3.7/site-packages/pungi/linker.py", line 218, in link
    raise OSError(errno.EEXIST, "File exists")
FileExistsError: [Errno 17] File exists
Frame link in /usr/lib/python3.7/site-packages/pungi/linker.py at line 218
<CODE>
     211     def link(self, src, dst, link_type="hardlink-or-copy"):
     212         """Link directories recursively."""
     213         if os.path.isfile(src) or os.path.islink(src):
     214             self._link_file(src, dst, link_type)
     215             return
     216 
     217         if os.path.isfile(dst):
-->  218             raise OSError(errno.EEXIST, "File exists")
     219 
     220         if not self.test:
     221             if not os.path.exists(dst):
     222                 makedirs(dst)
     223             shutil.copystat(src, dst)
     224 
</CODE>
<LOCALS>
                 dst = '/mnt/koji/compose/updates/Fedora-29-updates-testing-20190506.4/compose/Everything/source/tree/Packages/r/rdopkg-0.49.0-1.fc29.src.rpm'
           link_type = 'hardlink-or-copy'
                self = <pungi.linker.Linker object at 0x7f1e9fe28e80>
     self._inode_map = {(48, 27531575): '/mnt/koji/compose/updates/Fedora-29-updates-testing-20190506.4/compose/Everything/source/tree/Packages/d/delve-1.2.0-1.fc29.src.rpm', (48, 38070092): '/mnt/koji/compose/updates/Fedor
        self._logger = None
    self.always_copy = []
           self.test = False
                 src = '/mnt/koji/vol/prod/packages/rdopkg/0.49.0/1.fc29/data/signed/429476b4/src/rdopkg-0.49.0-1.fc29.src.rpm'
</LOCALS>
Frame process in /usr/lib/python3.7/site-packages/pungi/linker.py at line 63
<CODE>
      56         src, dst = item
      57 
      58         if (num % 100 == 0) or (num == self.pool.queue_total):
      59             self.pool.log_debug("Linked %s out of %s packages" % (num, self.pool.queue_total))
      60 
      61         directory = os.path.dirname(dst)
      62         makedirs(directory)
-->   63         self.pool.linker.link(src, dst, link_type=self.pool.link_type)
      64 
      65 
      66 class Linker(kobo.log.LoggingBase):
      67     def __init__(self, always_copy=None, test=False, logger=None):
      68         kobo.log.LoggingBase.__init__(self, logger=logger)
      69         self.always_copy = always_copy or []
</CODE>
<LOCALS>
           directory = '/mnt/koji/compose/updates/Fedora-29-updates-testing-20190506.4/compose/Everything/source/tree/Packages/r'
                 dst = '/mnt/koji/compose/updates/Fedora-29-updates-testing-20190506.4/compose/Everything/source/tree/Packages/r/rdopkg-0.49.0-1.fc29.src.rpm'
                item = <ERROR WHILE CONVERTING VALUE TO STRING>
                 num = 818
                self = <LinkerThread(Thread-102, stopped 139769471821568)>
          self._args = <ERROR WHILE CONVERTING VALUE TO STRING>
      self._daemonic = False
         self._ident = 139769471821568
   self._initialized = True
    self._is_stopped = True
        self._kwargs = {}
          self._name = 'Thread-102'
       self._started = <threading.Event object at 0x7f1e9fe28a90>
        self._stderr = <_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>
        self._target = None
   self._tstate_lock = None
         self.daemon = False
         self.failed = True
    self.get_timeout = 1
          self.ident = 139769471821568
           self.kill = True
           self.name = 'Thread-102'
           self.pool = <pungi.linker.LinkerPool object at 0x7f1e9fe28e10>
        self.running = False
                 src = '/mnt/koji/vol/prod/packages/rdopkg/0.49.0/1.fc29/data/signed/429476b4/src/rdopkg-0.49.0-1.fc29.src.rpm'
</LOCALS>

I think this is not a bug in Pungi, but instead a rather unfortunate race condition.

The error happened when packages from Koji volume were hardlinked into the compose for Everything.x86_64. The problematic file was vol/prod/packages/rdopkg/0.49.0/1.fc29/data/signed/429476b4/src/rdopkg-0.49.0-1.fc29.src.rpm.

In order for this exception to be raised, multiple conditions have to be met: the target path must exist and not be a symlink (which was true here since the file was hardlinked for another architecture already).

Another condition is that the source file must not be a file (because the same function can work with entire directories). In this case I think the source file did not exist.

At least checking from stage composer machine I'm getting this:

$ ls /mnt/fedora_koji_prod/koji/packages/rdopkg/0.49.0/1.fc29/data/signed/429476b4/src/rdopkg-0.49.0-1.fc29.src.rpm
ls: cannot access '/mnt/fedora_koji_prod/koji/packages/rdopkg/0.49.0/1.fc29/data/signed/429476b4/src/rdopkg-0.49.0-1.fc29.src.rpm': No such file or directory

There is a sigcache for this key. I think what happened here was that the signed copy was garbage collected while the compose was running.

On Tue, 2019-05-07 at 07:20 +0000, Lubom=C3=ADr Sedl=C3=A1=C5=99 wrote:

I think what happened here was that the signed copy was garbage
collected while the compose was running.

Oh interesting. Thanks for the analysis!

Metadata Update from @lsedlar:
- Issue close_status updated to: Invalid
- Issue status updated to: Closed (was: Open)

4 years ago

Login to comment on this ticket.

Metadata