#1931 Speedup for listing builds via APIv3 (take 2)
Closed 2 years ago by praiskup. Opened 2 years ago by frostyx.
copr/ frostyx/copr api-list-builds-speed-2  into  main

file modified
+1 -1
@@ -617,7 +617,7 @@ 

          :param args: argparse arguments provided by the user

          """

          ownername, projectname = self.parse_name(args.project)

-         pagination = {"limit": 100}

+         pagination = {"limit": 1000}

  

          builds_list = self.client.build_proxy.get_list(ownername, projectname,

                                                         pagination=pagination)

@@ -84,6 +84,16 @@ 

      if packagename:

          query = BuildsLogic.filter_by_package_name(query, packagename)

  

+     from sqlalchemy.orm import joinedload

+     # Loading relationships straight away makes running `to_dict` somewhat

+     # faster, which adds up over time, and  brings a significant speedup for

+     # large projects

+     query = query.options(

+         joinedload(models.Build.build_chroots),

+         joinedload(models.Build.package),

+         joinedload(models.Build.copr),

+     )

+ 

      # WORKAROUND

      # We can't filter builds by status directly in the database, because we

      # use a logic in Build.status property to determine a build status.

See PR#1914

I found two bottlenecks

1) to_dict function isn't extremely slow but it can be made faster
by pre-loading the models.Build relationships. This adds up and
brings a significant speedup for large projects

2) Querying the first pages is blazing fast but it gets 10 times
slower when approaching large offsets. We can mitigate the slowdown
by querying more results per page because the time for convering
database results to JSON remains the same but we simply don't have
to calculate the offset so often.

With this changes, I was able to reduce the time for completing

time copr-cli list-builds @rubygems/rubygems --output-format=json

from ~42min to ~15min.

See also PR#1930 which is a little bit faster but much more complex.
We need to decide whether the complexity is worth it and choose either this PR or #1930.

I think there is room for another speedup, so I will try to cut up the time more.

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

Pull-Request has been closed by praiskup

2 years ago