5a55d53 backend: speedup the background-process spawner

Authored and Committed by praiskup 2 years ago
    backend: speedup the background-process spawner
    
    The process handling in WorkerManager should be robust enough to avoid
    checking the BackgroundWorker:process() exit status.  We actually never
    provided any useful exit status value from there (except for potential
    daemon.DaemonContext() failures) — all the application logic info is
    propagated through Redis.
    
    So instead of waiting for the exit value in subprocess.check_call(),
    do the "startup" of the BackgroundWorker process in background too,
    using just Popen().  If the daemonizing logic ever fails, we will retry
    the task anyway in the next cycle(s) (Redis "started" field will not be
    set).
    
    This change might look like a fork-bomb will be much easier to happen,
    but we have all the WorkerLimit stuff in place and we don't ever start
    more BackgroundWorker processes than allowed.  But we'll be doing more
    concurrency during the process startup so the overall average startup
    time of process might prolong, and the code might be less reliable on a
    very, very slow backend machines (see the option worker_timeout_start
    which is still on 30s by default).
    
    The speedup measured on my (otherwise idling) i7-10850H box was from
    0.37s to 0.00065, IOW more than 500x faster.  This means that in the 20s
    sleeptime window we should be able to start several thousands of
    processes (should be really enough to feed all the allocated workers).
    
    Merges: #2139
    Relates: #2095
    
        
file modified
+2 -2