#679 Do not store SRPMs for failed builds
Closed 5 years ago by praiskup. Opened 5 years ago by frostyx.
copr/ frostyx/copr prunerepo-clean-copr  into  master

@@ -162,13 +162,13 @@ 

                      continue

  

              try:

-                 cmd = ['prunerepo', '--verbose', '--days', str(self.prune_days),

-                        '--cleancopr', '--nocreaterepo', chroot_path]

+                 cmd = ['prunerepo', '--verbose', '--days', str(self.prune_days), '--nocreaterepo', chroot_path]

                  stdout = runcmd(cmd)

                  loginfo(stdout)

                  createrepo(path=chroot_path, front_url=self.opts.frontend_base_url,

                             username=username, projectname=projectname,

                             override_acr_flag=True)

+                 clean_copr(chroot_path, self.prune_days, verbose=True)

              except Exception as err:

                  logexception(err)

                  logerror("Error pruning chroot {}/{}:{}".format(username, projectdir, sub_dir_name))
@@ -178,6 +178,50 @@ 

          loginfo("Pruning finished for projectdir {}/{}".format(username, projectdir))

  

  

+ def clean_copr(path, days=DEF_DAYS, verbose=True):

+     """

+     Remove whole copr build dirs if they no longer contain a RPM file

+     """

+     loginfo("Cleaning COPR repository...")

+     for dir_name in os.listdir(path):

+         dir_path = os.path.abspath(os.path.join(path, dir_name))

+ 

+         if not os.path.isdir(dir_path):

+             continue

+         if not os.path.isfile(os.path.join(dir_path, 'build.info')):

+             continue

+         if is_rpm_in_dir(dir_path):

+             continue

+         if time.time() - os.stat(dir_path).st_mtime <= days * 24 * 3600:

+             continue

+ 

+         if verbose:

+             loginfo('Removing: ' + dir_path)

+         shutil.rmtree(dir_path)

+ 

+         # also remove the associated log in the main dir

+         build_id = os.path.basename(dir_path).split('-')[0]

+         buildlog_name = 'build-' + build_id + '.log'

+         buildlog_path = os.path.abspath(os.path.join(path, buildlog_name))

+         rm_file(os.path.join(path, buildlog_path))

+ 

+ 

+ def rm_file(path, verbose=True):

+     """

+     Remove file given its absolute path

+     """

+     if verbose:

+         loginfo("Removing: "+path)

+     if os.path.exists(path) and os.path.isfile(path):

+         os.remove(path)

+ 

+ 

+ def is_rpm_in_dir(path):

+     files = os.listdir(path)

+     srpm_ex = (".src.rpm", ".nosrc.rpm")

+     return any([f for f in files if f.endswith(".rpm") and not f.endswith(srpm_ex)])

+ 

+ 

  def main():

      args = parser.parse_args()

      config_file = os.environ.get("BACKEND_CONFIG", "/etc/copr/copr-be.conf")

Fix #619

This fix was originally proposed as https://pagure.io/prunerepo/pull-request/1
but it makes more sense to own this part of the code.

This PR additionally adds a --path parameter to make the debugging easier. If you don't like it, we can throw it away or move to separate PR.

I pretty much dislike long lines (so wrap is prefered), it e.g. makes patch-review much easier...

The non-deamon mode seems misleading .. we run it in cron.

This feels like a work-around, what about to re-specify the whole result-dir (the path without ownername and projectdir), and thus guarantee that we never debug against real data? If you intend to have something like "white-list" feature (only one project affected), I'd love to see something like --whitelist or so (and the original idea --dry-run anyways).

+1 for the first three commits. The 3rd would be nicer with description like "clean data for failed builds again", but that's nit..

I don't really love the 4th commit tbh.

The 3rd would be nicer with description like "clean data for failed builds again"

.. or "clean failed builds again". The current description is even the title of this PR: "Do not store SRPMs for failed builds" and I always have to think what is this about :-D

It is less than 120 characters, so I squashed it to one line. I can change it back.

Ah, I forgot there's 120 chars guideline.

+1 for the first three commits. The 3rd would be nicer with description like "clean data for failed builds again", but that's nit..

I will try to rewrite the description so it is not confusing

I don't really love the 4th commit tbh.

No problem. I just wasn't sure how to work with the copr_prune_results.py script, so I've added the param for convenience. I wasn't sure, whether to add --path, --project, --whitelist, etc. Let's move this one commit to another PR since it is a low priority and unrelated to #619.

PEP8 limits line length to 79 characters, but AFAIK we are agreed on 120 characters in our codebase. IDK whether it is written somewhere or whether is it even true. We may want to re-discuss the topic.

3 new commits added

  • [backend] clean data for failed builds; fix #619
  • [backend] replace runnecessary regex with str.endswith
  • [backend] move clean_copr from prunerepo to our codebase
5 years ago

The 4th commit is gone and the 3rd commit name is rewritten.

Merged as 3 commits ending by 81a400802b42d2a8f1c9a33c3de6e2145ba44141

Pull-Request has been closed by praiskup

5 years ago