Today, a user noticed that several of their builds were just sitting in the init state. This was because MBS was processing 16 module builds, and since there's only a single thread that processes messages, the queue containing the messages waiting to be processed raised to over 3,000.

Since there are several things that MBS must do that take time like cloning dist-git repos and querying Koji, all other builds are blocked while this goes on. What makes things worse is when some of these fail, like a git clone failing, there's a 15 seconds sleep to try again, further blocking other activity from proceeding.

MBS does not scale well to the use-cases in Fedora, especially now that there are modules for several Fedora versions. We need to allow more messages to be processed at the same time to keep up with demand. There are some architectural issues in MBS that keep this from being possible. This ticket is about resolving those architectural issues to support multiple workers.