#1589 query builds per chunks in prune-signed-builds
Merged 4 years ago by tkopecek. Opened 4 years ago by tkopecek.
tkopecek/koji issue1587  into  master

file modified
+15 -2
@@ -1587,11 +1587,24 @@ 

      if not options.build:

          if options.verbose:

              print("Getting builds...")

-         qopts = {'state' : koji.BUILD_STATES['COMPLETE']}

+         qopts = {

+             'state' : koji.BUILD_STATES['COMPLETE'],

+             'queryOpts': {

+                 'limit': 50000,

+                 'offset': 0,

+                 'order': 'build_id',

+             }

+         }

          if options.package:

              pkginfo = session.getPackage(options.package)

              qopts['packageID'] = pkginfo['id']

-         builds = [(b['nvr'], b) for b in session.listBuilds(**qopts)]

+         builds = []

+         while True:

+             chunk = [(b['nvr'], b) for b in session.listBuilds(**qopts)]

+             if not chunk:

+                 break

+             builds.extend(chunk)

+             qopts['queryOpts']['offset'] += qopts['queryOpts']['limit']

          if options.verbose:

              print("...got %i builds" % len(builds))

          builds.sort()

For bigger installations query all builds can run out of memory on hub
side. Query these in 50k chunks.

Fixes: https://pagure.io/koji/issue/1587

There's a stray debugging print call in there

At some point we probably need to consolidate the chunked builds code, since we have different versions of it different places (including in koji-tools), but I don't think we have to make that part of this PR.

Also, it would be nice to have a guaranteed stable way of doing these chunked queries. At the moment, it's technically possible to miss a build or see a duplicate if the query results change in the middle. Again, probably something to tackle separately.

I've filed #1628 to track my comments above.

Otherwise looks fine and seems to be working here. Just need to drop the stray print()

rebased onto 462ad65

4 years ago

Commit bf624f5 fixes this pull-request

Pull-Request has been merged by tkopecek

4 years ago