pbrezina / copr / copr

Forked from copr/copr 2 years ago
Clone

94be497 distgit: allow `import_package` function to run in parallel

6 files Authored by frostyx 3 years ago, Committed by praiskup 3 years ago,
    distgit: allow `import_package` function to run in parallel
    
    aka fix blocking the importing queue by texlive
    
    Limiting `import_package` function to run only once at the time across
    all threads is too restricting. Although it brings some parallel
    processing, speeds up going through a large queue and prevents
    copr-dist-git from getting stuck when an importing fails (for various
    reasons), it is not good enough. The `import_package` does the biggest
    portion of the importing process and can take a long time for some
    packages, e.g. texlive (because it has ~6k sources).
    
    Even though I couldn't produce any issues when removing the locking
    mechanism altogether, I don't think we should just get rid of
    it. Theoretically some nasty conflict could happen when trying to
    import multiple large packages at the same time.
    
    IMHO we should lock on the repoository level, i.e. the combination of
    `owner/project/package`. With this, I can easily submit a texlive to
    be built, and while it is importing, I can successfully run
    beaker-tests and have them finished before the texlive import is
    complete, i.e. it is not blocking the importing queue. At the same
    time, if I submit two texlive pacages to the same project, they won't
    be imported in parallel but rather one after the other, so there won't
    be any conflict.
    
        
file modified
+7 -0
file modified
+15 -15
file modified
+12 -7