PR#30: If failing to process a repo, wait 30 seconds and try again - mdapi

mdapi

#30 If failing to process a repo, wait 30 seconds and try again

Merged 8 years ago by pingou. Opened 8 years ago by pingou.

fix_cron into master

Let's loop a few time to retrieve the repo before bailing out

Pierre-Yves Chibon • 8 years ago

65ed2da

Make the sleeping time configurable

Pierre-Yves Chibon • 8 years ago

1d54658

If failing to process a repo, wait 30 seconds and try again

Pierre-Yves Chibon • 8 years ago

5af54c4

mdapi-get_repo_md

file modified

+18 -4

		`@@ -41,6 +41,7 @@`
		`import os`
		`import shutil`
		`import tempfile`
		`+ import time`
		`import hashlib`
		`import xml.etree.ElementTree as ET`

		`@@ -504,11 +505,24 @@`
		`#)`

		`# In serial`
		`+ sleep_for = CONFIG.get('CRON_SLEEP', 30)`
		`for t in itertools.product(`
		`- [CONFIG.get('DB_FOLDER', '/var/tmp')],`
		`- repositories):`
		`-`
		`- process_repo(t)`
		`+ [CONFIG.get('DB_FOLDER', '/var/tmp')],`
		`+ repositories):`
		`+`
		`+ loop = True`
		`+ cnt = 0`
		`+ while loop:`
		`+ cnt += 1`
		`+ try:`
		`+ process_repo(t)`
		`+ loop = False`
		`+ except OSError:`
		`+ if cnt == 4:`
		`+ raise`
		`+ # Most often due to an invalid stream, so let's try a second time`
		`+ time.sleep(sleep_for)`
		`+ process_repo(t)`

		`return 0`

no initial comment

Pull-Request has been rebased

8 years ago

puiterwijk commented on line 24 of mdapi-get_repo_md 8 years ago

I think this might be a bit long, not?

pingou commented on line 24 of mdapi-get_repo_md 8 years ago

If the problem is that koji is currently writing/generating its sqlite DB, I don't think it's too much.

The cron runs every 20 minutes, so it should be fine anyway

ralph commented 8 years ago

Heh, can we make the sleep interval configurable?

pingou commented 8 years ago

Done :)

Pull-Request has been updated

8 years ago

ralph commented 8 years ago

:+1: here.

puiterwijk commented 8 years ago

Maybe instead of retry once, have a retry counter, sleep for sleep_for between every attempt, and only abort if it still doesn't work after retry_for times?

Pull-Request has been updated

8 years ago

pingou commented 8 years ago

Adjusted :)

puiterwijk commented 8 years ago

:+1: Looks good to me.

pingou commented 8 years ago

thanks, closing then :)

Pull-Request has been merged by pingou

8 years ago