PR#88: Attempt to reduce the amount of mdapi queries - fedora-qa/oraculum

fedora-qa / oraculum

#88 Attempt to reduce the amount of mdapi queries

Merged 3 years ago by frantisekz. Opened 3 years ago by frantisekz.

optimize_versions into master

Don't retry requests to mdapi at all

František Zatloukal • 3 years ago

d9e0485

Attempt to reduce the amount of mdapi queries

František Zatloukal • 3 years ago

5c91e5e

oraculum/utils/dashboard_helpers.py

file modified

+8 -4

		`@@ -42,10 +42,14 @@`
		`Returns json data from provided url`
		`"""`
		`session = requests.Session()`
		`- retries = Retry(total=3,`
		`- backoff_factor=0.1,`
		`- status_forcelist=[429, 500, 502, 503, 504])`
		`- session.mount('https://', HTTPAdapter(max_retries=retries))`
		`+`
		`+ if attempt_retry:`
		`+ retries = Retry(total=3,`
		`+ backoff_factor=0.1,`
		`+ status_forcelist=[429, 500, 502, 503, 504])`
		`+ session.mount('https://', HTTPAdapter(max_retries=retries))`
		`+ else:`
		`+ session.mount('https://', HTTPAdapter())`
		`try:`
		`resp = session.get(json_url)`
		`except ConnectionError:`

oraculum/utils/pagure.py

file modified

+7 -1

		`@@ -153,7 +153,7 @@`
		`versions = {}`
		`releases = CACHE.get("fedora_releases")`
		`for release in releases["values"][:-1]: # Skip Rawhide`
		`- versions[release_from_dist(str(release))] = {"stable": None, "testing": None}`
		`+ versions["Fedora %d" % release] = {"stable": None, "testing": None}`
		`# Append Rawhide in standardized naming format`
		`versions["Fedora Rawhide"] = {"stable": None, "testing": None}`
		`for release in app.config["EPEL_RELEASES"]:`
		`@@ -182,6 +182,12 @@`
		`for release in versions:`
		`if "Fedora" in release and not versions[release]["stable"]:`
		`if "Rawhide" in release:`
		`+ # Before the mass rebuild, it'll be pretty common for packages to not have any Rawhide build`
		`+ # Let's copy out version from previous Fedora for cases when Rawhide does not have any build`
		`+ if versions["Fedora %d" % releases["values"][-2]]["stable"]:`
		`+ versions[release]["stable"] = versions["Fedora %d" % releases["values"][-2]]["stable"]`
		`+ # Such speed, much wow!`
		`+ continue`
		`mdapi_response = get_json("https://mdapi.fedoraproject.org/%s/srcpkg/%s" % ("rawhide", package), False, False)`
		`else:`
		`mdapi_response = get_json("https://mdapi.fedoraproject.org/%s/srcpkg/%s" % ("f" + release[-2:], package), False, False)`

frantisekz commented 3 years ago

I'll need to try and rethink how oraculum does handle version refreshing. This PR has already been deployed as an attempt to at least reduce the number of (reeeeaaly slow) mdapi queries.

Since we're before Rawhide/f34 rebuild, we were querying mdapi for almost every package. This exhausted all our workers, permanently.

Also, fixes querying versions for Fedoras other than Rawhide, this was broken since #93412b5 (as we were relying on "undefined" behavior of that function).

I'll add some more tweaks tomorrow, based on observations.