pagure-importer

#100 Using multiple threads to import issues from fedorahosted
Closed by cverna. Opened by cverna.
cverna/pagure-importer support_parallel_requests  into  master

Download 100.patch

This PR enable the use of multiple threads to import issues. This is speeding up the sending of JSON-RPC requests to fedorahosted, by allowing more than 1 request at the time to be sent.

I always wonder what is the best approach between Thread and Multiprocessing ( http://blog.pingoured.fr/index.php?post/2014/11/27/Python-multiprocessing-and-queue )

So I just asked duckduckgo and landed on http://stackoverflow.com/questions/3044580/multiprocessing-vs-threading-python#3046201

So this is also looking good :)

Thanks for pointing out the Multiprocessing module (did not know about it). I might play around with it ;)

Aren't network calls(RPCs or get/post) non-CPU bound? (i don't know for sure but, this will be my guess). In that case, according to the links, threading should do better.

Network calls are definitely not CPU bound and much more I/O bound :)

Good catch @vivekanand1101 I missed this last bullet when checking this page.

So :thumbsup: for me

close as this will need to be refactored after #105 is merged

Pull-Request has been closed by cverna
Metadata
None
No Tags
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.