PR#1931: Speedup for listing builds via APIv3 (take 2) - copr/copr

copr / copr

#1931 Speedup for listing builds via APIv3 (take 2)

Closed 2 years ago by praiskup. Opened 2 years ago by frostyx.

copr/ frostyx/copr api-list-builds-speed-2 into main

frontend: speedup for listing builds via APIv3

Jakub Kadlcik • 2 years ago

82195bc

cli/copr_cli/main.py

file modified

+1 -1

		`@@ -617,7 +617,7 @@`
		`:param args: argparse arguments provided by the user`
		`"""`
		`ownername, projectname = self.parse_name(args.project)`
		`- pagination = {"limit": 100}`
		`+ pagination = {"limit": 1000}`

		`builds_list = self.client.build_proxy.get_list(ownername, projectname,`
		`pagination=pagination)`

frontend/coprs_frontend/coprs/views/apiv3_ns/apiv3_builds.py

file modified

+10

		`@@ -84,6 +84,16 @@`
		`if packagename:`
		`query = BuildsLogic.filter_by_package_name(query, packagename)`

		`+ from sqlalchemy.orm import joinedload`
		+ # Loading relationships straight away makes running `to_dict` somewhat
		`+ # faster, which adds up over time, and brings a significant speedup for`
		`+ # large projects`
		`+ query = query.options(`
		`+ joinedload(models.Build.build_chroots),`
		`+ joinedload(models.Build.package),`
		`+ joinedload(models.Build.copr),`
		`+ )`
		`+`
		`# WORKAROUND`
		`# We can't filter builds by status directly in the database, because we`
		`# use a logic in Build.status property to determine a build status.`

frostyx commented 2 years ago

See PR#1914

I found two bottlenecks

1) to_dict function isn't extremely slow but it can be made faster
by pre-loading the models.Build relationships. This adds up and
brings a significant speedup for large projects

2) Querying the first pages is blazing fast but it gets 10 times
slower when approaching large offsets. We can mitigate the slowdown
by querying more results per page because the time for convering
database results to JSON remains the same but we simply don't have
to calculate the offset so often.

With this changes, I was able to reduce the time for completing

time copr-cli list-builds @rubygems/rubygems --output-format=json

from ~42min to ~15min.

frostyx commented 2 years ago

See also PR#1930 which is a little bit faster but much more complex.
We need to decide whether the complexity is worth it and choose either this PR or #1930.

I think there is room for another speedup, so I will try to cut up the time more.

Edited 2 years ago by frostyx