#1023 Add doc describing how mbs works internally when building modules.
Closed 9 months ago by ralph. Opened 10 months ago by jkaluza.
jkaluza/fm-orchestrator how-are-modules-built  into  master

@@ -0,0 +1,178 @@ 

+ How does MBS build modules?

+ ===========================

+ 

+ This document describes how are modules built internally in MBS. The goal of this document is

"how are modules" => "how modules are"

+ to explain code-flow of module builds. It assumes everything goes as expected and does not

+ mention any error handling or corner cases.

+ 

+ 

+ User submits module build request

+ ---------------------------------

+ 

+ There is MBS frontend providing REST API (See `views.py`). User sends POST request with JSON

"frontend providing REST API" => "frontend, which provides a REST API"

"User sends POST" => "The user sends a POST"

+ describing the module to build. There is mainly the URL to git repository (called `scmurl`)

"to git repository" => "to the git repository"

+ and branch name (called `branch`). The `scmurl` points to git repository containing the

"to git repository" => "to the git repository"

+ modulemd file defining the module.

+ 

+ This JSON data is handled by `views.SCMHandler`, which validates the JSON and calls

+ `utils.submit.submit_module_build_from_scm(...)` method. This goes down to

+ `submit_module_build(...)`.

+ 

+ 

+ Module Stream Expansion (MSE)

+ -----------------------------

+ 

+ The first thing done in `submit_module_build(...)` is Module Stream Expansion (MSE).

+ 

+ The submitted modulemd file might have buildrequires and requires pairs defined in ambigous way.
cqi commented 10 months ago

u is probably missed in ambigous.

"defined in" => "defined in an"

+ For example the module can buildrequire `platform:[28, 29]` modules, which means it should

Isn't f missing in platform:[28, 29]?

+ be built against the `f28` and `f29` streams of `platform` module.

+ 

+ The process of resolving these ambigous buildrequires and requires is called Module Stream

+ Expansion.

+ 

+ Input for this process is the submitted modulemd file with ambigous buildrequires/requires.

+ Output of this process is list of multiple modulemd files with all the ambigous

"is list of multiple modulemd files" => "is the list of modulemd files"

+ buildrequires/requires resolved.

+ 

+ This all happens in `utils.mse.generate_expanded_mmds(...)` method.

"happens in" => "happens in the"

+ 

+ At first, this method finds out all the possible buildrequires/requires for the input module.

+ This is done using `DBResolver` which simply finds out the modules in the MBS database.

+ In our case, it would list all the `platform:f28` and `platform:f29` modules.

+ 

+ It then uses `MMDResolver` class to find all the possible combinations of buildrequires/requires

"uses" => "uses the"

+ for which the input module can be built.

"can be built" => "can be built with"

+ 

+ In our case, it would generate two expanded modulemd files (one for each platform stream) which

+ would be identical to input modulemd file with only following exceptions:

"to input" => "to the input" and "with only following" => "with only the following"

+ 

+ - The buildrequires/requires pairs from input modulemd files will be replaced by the particular

"from input" => "from the input"

+   combination returned by `MMDResolver`

+ - The `xmd` section of generated modulemd files will contain `buildrequires` list which lists all

Mention this also includes the requires of the buildrequires.

+   the modules required to build this expanded modulemd file. This is used later by MBS.

+ - The context is computed and filled for each expanded modulemd file. It is based on the

+   expanded buildrequires and requires pairs. See `models.ModuleBuild.contexts_from_mmd(...)`.

+ 

+ Such expanded modulemd files are then added to database as next step in `submit_module_build(...)`

"as next step" => "as the next step"

+ and are handled as separate module builds later in MBS.

+ 

+ The `submit_module_build(...)` then moves the module builds to "init" state and sends message to

"sends message" => "sends a message"

+ fedmsg hub or UMB for each submitted expanded module build

"sends message to fedmsg hub or UMB" => "sends a message on the configured message bus"

+ 

+ 

+ Backend handles module moved to "init" state

+ --------------------------------------------

+ 

+ When module build is moved to "init" state, backend handles that in

"to "init"" => "to the "init""
"backend handles" => "the backend handles"
"that in" => "that in the"

+ `scheduler.handlers.modules.init(...)` method.

+ 

+ This method calls `utils.submit.record_component_builds` which reads the modulemd file

+ stored in database by frontend and records all the components (future RPM packages) in the

"in database by frontend" => "in the database by the frontend"

+ database.

+ 

+ The components are divided into the **batches** based on their buildorder in the modulemd file.

+ 

+ Once the components which are supposed to be built as part of this module build are recorded,

+ the module moves to "wait" state and another message it sent to the message bus.

"to "wait" state" => "to the "wait" state"

"message it sent to" => "message is sent on"

+ 

+ 

+ Backend handles module moved to "wait" state

+ --------------------------------------------

+ 

+ When module build is moved to "wait" state, backend handles that in

"When" => "When the"

"to "wait"" => "to the "wait""
"backend" => "the backend"
"that in" => "that in the"

+ `scheduler.handlers.modules.wait(...)` method.

+ 

+ At first, this method uses KojiModuleBuilder to generate Koji tag in which the components will be

"generate Koji" => "generate the Koji"

+ build. The Koji tag reflects the buildrequires of module by inheriting their Koji tags. In our

"build" => "built"
"of module" => "of the module"

+ case, the Koji tag would inherit just `platform:f28` or `platform:f29` Koji tag, because that's

"just" => "just the"

+ the only buildrequired module we have.

+ The list of modules buildrequired by currently building module is get from `buildrequires` list in

"by currently" => "by the currently"
"is get from" => "is determined by the"

+ the `xmd` section of expanded modulemd file.

"of expanded" => "of the expanded"

+ 

+ Once the Koji tag is ready, it tries to build `module-build-macros` package. This package contains

"build" => "build the"

+ special build macros which for example defines the dist-tag for built RPMs, ensures that filtered

"for example defines" => "for example, defines"

+ packages are not installed into the buildroot and so on.

"buildroot and so on" => "buildroot, and etc."

+ 

+ The module-build-macros is always built in first batch.

"built in first" => "built in the first"

+ 

+ 

+ Module-build-macros package is built

+ ------------------------------------

+ 

+ Once the module-build-macros package is built, Koji sends message to message hub which is handled

"sends message to message hub which" => "sends a message on the message bus, which"

+ in `scheduler.handlers.components.complete(...)` method.

"in" => "in the"

+ 

+ This method changes that state of component build in MBS database to "complete".

"that state of" => "the state of that"
"in MBS" => "in the MBS"

+ 

+ It then checks if there are any other unbuilt components in current batch. Because the

"in current" => "in the current"

"Because the" => "Because"

+ "module-build-macros" is the only component in batch 1, it can continue tagging it

+ into the Koji tag representing the module, so the module-build-macros can be later

It's probably important to note that this only gets tagged in the build tag.

For consistency with above: the module-build-macros => "module-build-macros"

+ installed during the build of next components and can influence them.

"of next components" => "of the next batch of components"

+ 

+ 

+ Module-build-macros package is tagged into the Koji tag

+ -------------------------------------------------------

+ 

+ Once the module-build-macros package is tagged by Koji, the `scheduler.handlers.tags.tagged(...)`

+ method is called.

+ 

+ This simply waits until all the components in a currently built batch are tagged in a Koji tag.

+ 

+ Because module-build-macros is the only component in batch 1, it can continue by regenerating

+ the Koji repository based on a tag, so the newly build packages (just module-build-macros
cqi commented 10 months ago

s/newly build/newly built/?

+ in our case), can be installed from that repository when building next components in a module.

"building next" => "building the next"

+ 

+ 

+ Koji repository is regenerated

+ ------------------------------

+ 

+ Once the Koji repository containing packages from currently built batch is regenerated,

"from currently" => "from the currently"

+ the `scheduler.handlers.repos.done(...)` method is called.

+ 

+ This verifies that all the packages from current batch (just module-build-macros for now)

"from current" => "from the current"

+ really appear in generated repository and if so, it starts building next batch by calling

"in generated" => "in the generated"
"building next" => "building the next"

+ `utils.batches.start_next_batch_build(...)`.

+ 

+ 

+ Building next batch

"Building next batch" => "Building the next batch"

+ -------------------

+ 

+ The `start_next_batch_build(...)` increases the `ModuleBuild.batch` counter to note that it

+ is going to build next batch with next component builds.

"build next" => "build the next"
"with next" => "with the next"

+ 

+ It then generates the list of unbuilt components in the batch and tries to reuse some from

+ previous module builds. This can happen for example when the component is built from the

+ same source as previously, no component builds in previous batches changed and the

+ buildrequires/requires of current module build are still the same as previously.

"of current" => "of the current"

+ 

+ For components which cannot be reused, it submits them to Koji.

+ 

+ 

+ Build all components in all batches in a module

+ -----------------------------------------------

+ 

+ The process for every component build is the same as for module-build-macros.

+ 

+ MBS builds it in Koji. Once all the components in current batch are built, MBS tags them into

"in current" => "in the current"

+ the Koji tag. Once they are tagged, it regenerates the Koji tag repository and then starts

+ building next batch.

"building next" => "building the next"

+ 

+ It all ends up when all batches are done.

Do you mean something like?
This process is repeated until all the batches are complete.

+ 

+ 

+ Last component is built

+ -----------------------

+ 

+ Once the last component is built and the repository is regenerated, the

+ `scheduler.handlers.repos.done(...)` method moves the module build to "done" state.

"to "done"" => "to the "done""

It might be worth mentioning that a message is sent on the bus to signify that the module is in the done state.

+ 

+ 

+ Importing module build to Koji

"Importing module" => "Importing the module"

+ ------------------------------

+ 

+ The "done" state message is handled in `scheduler.handlers.modules.done(...)` method.

"handled in" => "handled in the"

+ 

+ This method imports the module build into the Koji using the `KojiContentGenerator` class.

"into the" => "into"

+ The module build in Koji points to Koji tag with module components and also contains the

"to Koji tag" => "to the Koji tag"
"module components" => "the module's components"

+ final modulemd files generated for earch architecture the module is built for.

Not sure if it's good try and it probably leads to much more questions, but at least one could understand the basic code-flow....

s/newly build/newly built/?

u is probably missed in ambigous.

This is really helpful. :thumbsup: Thanks.

Isn't f missing in platform:[28, 29]?

"how are modules" => "how modules are"

"frontend providing REST API" => "frontend, which provides a REST API"

"User sends POST" => "The user sends a POST"

"to git repository" => "to the git repository"

"to git repository" => "to the git repository"

"defined in" => "defined in an"

"is list of multiple modulemd files" => "is the list of modulemd files"

"happens in" => "happens in the"

"can be built" => "can be built with"

"to input" => "to the input" and "with only following" => "with only the following"

"from input" => "from the input"

Mention this also includes the requires of the buildrequires.

"as next step" => "as the next step"

"sends message" => "sends a message"

"sends message to fedmsg hub or UMB" => "sends a message on the configured message bus"

"to "init"" => "to the "init""
"backend handles" => "the backend handles"
"that in" => "that in the"

"in database by frontend" => "in the database by the frontend"

"to "wait" state" => "to the "wait" state"

"message it sent to" => "message is sent on"

"to "wait"" => "to the "wait""
"backend" => "the backend"
"that in" => "that in the"

"generate Koji" => "generate the Koji"

"build" => "built"
"of module" => "of the module"

"by currently" => "by the currently"
"is get from" => "is determined by the"

"of expanded" => "of the expanded"

"for example defines" => "for example, defines"

"buildroot and so on" => "buildroot, and etc."

"built in first" => "built in the first"

"sends message to message hub which" => "sends a message on the message bus, which"

"that state of" => "the state of that"
"in MBS" => "in the MBS"

"in current" => "in the current"

It's probably important to note that this only gets tagged in the build tag.

For consistency with above: the module-build-macros => "module-build-macros"

"of next components" => "of the next batch of components"

"building next" => "building the next"

"from currently" => "from the currently"

"from current" => "from the current"

"in generated" => "in the generated"
"building next" => "building the next"

"Building next batch" => "Building the next batch"

"build next" => "build the next"
"with next" => "with the next"

"of current" => "of the current"

"in current" => "in the current"

"building next" => "building the next"

Do you mean something like?
This process is repeated until all the batches are complete.

"to "done"" => "to the "done""

It might be worth mentioning that a message is sent on the bus to signify that the module is in the done state.

"Importing module" => "Importing the module"

"handled in" => "handled in the"

"to Koji tag" => "to the Koji tag"
"module components" => "the module's components"

@jkaluza very nice document. Thank you for filing this PR. I left some comments in regards to some of the grammar. Please let me know if you need help addressing them.

@mprahl, I think you broke @jkaluza. :p

Maybe one of us can take his branch, add your edits into it, and submit a new PR? I don't think I can push to jkaluza's fork myself... (I'll try that first).

@ralph let me know if you want me to take it over or if you've already started.

Updated based on comments, rebased, and merged.

Pull-Request has been closed by ralph

9 months ago