PR#183: bugzilla: disable "quick" sync in favor of a full sync - fedora-qa/blockerbugs

fedora-qa / blockerbugs

#183 bugzilla: disable "quick" sync in favor of a full sync

Merged 3 years ago by kparal. Opened 3 years ago by kparal.

feature/bzfixes into develop

bugzilla: disable "quick" sync in favor of a full sync

Kamil Páral • 3 years ago

baa9268

blockerbugs/cli.py

file modified

+9 -3

		`@@ -173,8 +173,14 @@`


		`def sync_bugs(args):`
		`- fullsync = args.full`
		`docheck = args.check`
		`+ fullsync = args.full`
		`+ # FIXME: as a workaround for very slow standard sync times, perform a full`
		`+ # sync always. If no downsides are found, completely remove the "--full"`
		`+ # cmdline option (and related function arguments) and make full sync the`
		`+ # only approach.`
		`+ fullsync = True`
		`+`

		`# grab the URL from app config so that it's syncing from the right place`
		`bzurl = app.config['BUGZILLA_XMLRPC']`
		`@@ -271,7 +277,7 @@`
		`help='Synchronize all')`
		`sync_parser.add_argument('-f', '--full',`
		`action='store_true', default=False,`
		`- help='Force full sync')`
		`+ help='Force full sync (ignored temporarily)')`
		`sync_parser.add_argument('-c', '--check',`
		`action='store_true', default=False,`
		`help='Force check for missing blocker bugs after sync')`
		`@@ -281,7 +287,7 @@`
		`help='Synchronize bugs only')`
		`sync_bugs_parser.add_argument('-f', '--full',`
		`action='store_true', default=False,`
		`- help='Force full sync')`
		`+ help='Force full sync (ignored temporarily)')`
		`sync_bugs_parser.add_argument('-c', '--check',`
		`action='store_true', default=False,`
		`help='Force check for missing blocker bugs after sync')`

blockerbugs/util/bz_interface.py

file modified

		`@@ -83,6 +83,8 @@`
		`# keep in mind that OP -> open parenthesis, CP -> closed parenthesis`
		`# and the OR is to make sure that a bug is returned if ANY of the`
		`# changed after conditions are met`
		`+`
		`+ # FIXME: This seems to be tremendously slowing down Bugzilla5`
		`query.update({`
		`'f2': 'OP',`
		`'j2': 'OR',`

kparal commented 3 years ago

The standard "quick" sync takes one or two orders of magnitude longer than a
full sync, and often times out. It seems that while the timestamp optimizations
were helpful with older Bugzilla versions, with Bugzilla5 they are taking
extremely long to evaluate. It seems much better to always do a "full" sync
instead.

If we don't see any negative consequences, let's remove the quick sync code
completely.

@frantisekz @lbrabec Thoughts?

zuul commented 3 years ago

Build succeeded.

blockerbugs-pytest : SUCCESS in 1m 56s

frantisekz commented 3 years ago

The standard "quick" sync takes one or two orders of magnitude longer than a
full sync, and often times out.

This sounds really weird, we're using the quick path in oraculum with a great success. The oraculum code for bugzilla is basically this function: https://pagure.io/fedora-qa/oraculum/blob/master/f/oraculum/utils/bugzilla.py#_91

For eg. kernel package, the full sync takes between 20 and 30 seconds, the quick path takes about 3 seconds. Might be worth mentioning that we're using BZ's REST api instead of CGI.

We're creating BZ object with this:

BUGZILLA = bugzilla.Bugzilla("bugzilla.redhat.com",
                            tokenfile="/tmp/python-bugzilla-token-%d" % os.getpid(),
                            cookiefile="/tmp/python-bugzilla-cookie-%d" % os.getpid(),
                            force_rest=True,
                            api_key=app.config["BZ_API_KEY"])

kparal commented 3 years ago

Time measurements from the BBA server:
sync-bugs: 45-60 sec
sync-bugs --full: 15 sec

frantisekz commented 3 years ago

PR seems okay (it doesn't matter that much if sync takes 60 seconds or 15 on the server imho, but for a better development experience, merging wouldn't hurt anything).

I can take a look next week and try if I can figure out what's the issue with the quick path here.

On the other hand, we should start making use of fedora messages for real-time sync with bugzilla and probably leave only the full sync to run from time to time just to be sure, just for the initial syncup or only when something goes wrong. That's a discussion to have somewhere else.

kparal commented 3 years ago

If I just add force_rest=True, the queries blow up. It seems the REST API is incompatible with the XMLRPC API. I'll gladly work on updating it for the REST API, but that would be a separate change in the future. It might yield some performance gain, although the performance with full sync seems to be very acceptable even currently (and much better than the current quick sync).

kparal commented 3 years ago

REST API conversion task filed as #184

kparal commented 3 years ago

Commit b01fea8 fixes this pull-request

Pull-Request has been merged by kparal

3 years ago