#892 backend: admin opt-out createrepo after build-deleting
Merged 4 years ago by praiskup. Opened 4 years ago by praiskup.

@@ -255,6 +255,12 @@

              result_base_url = "/".join(

                  [self.results_root_url, ownername, project_dirname, chroot])

  

+             project = "{}/{}".format(ownername, project_dirname)

+             if  project in self.opts.build_deleting_without_createrepo.split():

+                 self.log.warning("createrepo takes too long in %s, skipped",

+                                  project)

+                 return

+ 

              try:

                  createrepo(

                      path=chroot_path,

@@ -278,6 +278,9 @@

  

          opts.prune_days = _get_conf(cp, "backend", "prune_days", None, mode="int")

  

+         opts.build_deleting_without_createrepo = _get_conf(

+                 cp, "backend", "build_deleting_without_createrepo", "")

+ 

          # ssh options

          opts.ssh = Munch()

          opts.ssh.builder_config = _get_conf(

@@ -85,6 +85,10 @@

  #redis_port=6379

  #redis_db=0

  

+ # Don't run createrepo after build delete action in those (space separate list)

+ # projects.  Notify users before adding them here.

+ #build_deleting_without_createrepo=user1/projectFoo @python/python3.8

+ 

  [builder]

  # default is 1800

  timeout=3600

For projects like @python/python3.8, createrepo_c run takes too
long (up to 1 minute per one chroot). Deleting large amount of
builds then creates long action queue which blocks even other
projects.

We should make actions parallel to mitigate those problems, but
this is temporary work-around that allows us to "blacklist"
expensive projects.

rebased onto 984e8fe40b7a13d96941108788823dd171943df7

4 years ago

@churchyard FYI, I noticed you probably removed some builds from the
@python/python3.8 project yesterday. This hanged our action queue..

So I blacklisted (see the patch in this PR) the @python/python3.8 project
from running createrepo_c after deleting build... Which means that
you need to re-create the repositories manually after build deleting.
Sorry for inconvenience, I believe we'll have parallel action processing
soon...

Oh, I've thought that without appdata, this works fine, sorry for clogging the queue.

Will the first successful build regenerate the repo?

Oh, I've thought that without appdata, this works fine, sorry...

No problem, shouldn't happen anymore. If you'd remove one package, it wouldn't cause much troulbes ... but there was at least 70 builds to be deleted when I checked that; so it caused some rush.

Will the first successful build regenerate the repo?

Yes, though without having the repo regenerated manually your build may fail as your build may need to install the deleted package that still appears to be in the repo (from metadata). So it is better to plan the removal.

ack. thanks for the heads up

rebased onto 078abb567110d47f2226a0ef46e67fd1cd95edf1

4 years ago

Can anyone take a look at this?

It looks good to me, +1.

It is a little unfortunate, that those settings are not bound to the disable appstream ones (see PR#742).
If we would rename .disable-appstream file to something else, that would mean both "disable appstream" and "disable createrepo when deleting", we could configure both settings at once.
When leaving the PR as is, we would IMHO set both things for every huge project anyway.

This is hopefully temporary hack, which no user basically wants to have turned-on forever. It is breaking the repo after package deletes (requires manual re-create). IOW, this is short-term work-around before we'll handle createrepo processes a better way.

the .disable-appstream OTOH won't have a reasonable solution in feasible future

rebased onto 16e68d88ad2da00c98ad749569d0ad39415ba72f

4 years ago

rebased onto 8f50517

4 years ago

Pull-Request has been merged by praiskup

4 years ago