A while ago, I left these comments on a libmodulemd issue asking about this...
My understanding is that it is definitely possible to have libdnf or even base libsolv sort build dependencies and create ordered groups of packages to build correctly. At least in my experience with OBS, the way it works is that the build scheduler uses the solver to identify what requires what and it creates order groups and processes them in a queue. If a dependency loop exists (package A requires package B which would would require package A after a build to build with more features) it does a second build and third build for that, and if the third build has the same digest (reproducibility-wise) as the second build, it's considered done. So given a distribution package set, you'd wind up with cluster X, cluster Y, and cluster Z. Cluster X is required by cluster Y, which is required by cluster Z. So X would be built first, internally resolving itself, then Y, then Z. This is what allows openSUSE to effectively be rebootstrapped entirely from scratch with minimal to no waste on a regular basis (as in more or less each month or something like that). mockchain lacked integration with a solver, so it never was capable of that. Instead, you have to build everything and keep retrying until everything is built (assuming it can be built).
My understanding is that it is definitely possible to have libdnf or even base libsolv sort build dependencies and create ordered groups of packages to build correctly.
At least in my experience with OBS, the way it works is that the build scheduler uses the solver to identify what requires what and it creates order groups and processes them in a queue. If a dependency loop exists (package A requires package B which would would require package A after a build to build with more features) it does a second build and third build for that, and if the third build has the same digest (reproducibility-wise) as the second build, it's considered done.
So given a distribution package set, you'd wind up with cluster X, cluster Y, and cluster Z. Cluster X is required by cluster Y, which is required by cluster Z. So X would be built first, internally resolving itself, then Y, then Z.
This is what allows openSUSE to effectively be rebootstrapped entirely from scratch with minimal to no waste on a regular basis (as in more or less each month or something like that).
mockchain lacked integration with a solver, so it never was capable of that. Instead, you have to build everything and keep retrying until everything is built (assuming it can be built).
mockchain
[...]
openSUSE actually uses the raw spec file to identify build dependencies. obs-build can evaluate macros and platform conditions against a spec file, and parses the spec file to identify the correct build dependencies to emit and resolve with. It's not too hard to implement the same thing using rpm, either. I did a variation of this for a project on GitLab.com for being able to build the package in CI. If you're using the spec file as the direct input rather than the source RPM, then you can evaluate all the platforms and determine the correct build dependencies without having to deal with the "archful SRPM" problem. If you want to attempt this cheaply, use the rpmspec(8) tool to do it. You can also use the RPM Python API to do it, as dnf builddep does. Since rpmspec accepts the global options that rpm does, you can use --target to make it pretend to be a specific arch for the purposes of determining build dependencies.
openSUSE actually uses the raw spec file to identify build dependencies. obs-build can evaluate macros and platform conditions against a spec file, and parses the spec file to identify the correct build dependencies to emit and resolve with. It's not too hard to implement the same thing using rpm, either. I did a variation of this for a project on GitLab.com for being able to build the package in CI.
obs-build
If you're using the spec file as the direct input rather than the source RPM, then you can evaluate all the platforms and determine the correct build dependencies without having to deal with the "archful SRPM" problem.
If you want to attempt this cheaply, use the rpmspec(8) tool to do it. You can also use the RPM Python API to do it, as dnf builddep does. Since rpmspec accepts the global options that rpm does, you can use --target to make it pretend to be a specific arch for the purposes of determining build dependencies.
rpmspec(8)
dnf builddep
rpmspec
rpm
--target
Since then, @ignatenkobrain has made a variant (admittedly, a simple one) of what I asked for generating Rust modulemd YAML files. Can we have something like this generalized (similarly to how I described in my comments above) and incorporated into MBS so that packagers don't have to try to figure out build groups, build ordering, or any other weird details for chain building packages for making modules?
I will leave quick comment here. When other people evaluated this some time ago, there were opinions that we don't know what the Provides are unless we build an RPM and without the knowledge of Provides, we cannot find out the dependency tree.
I'm definitely not against this feature and I would like this to see this implemented and make everyone's life easier.
The issue also is that we are just few developers working on MBS and we also have other projects to work on, so if you really want to see this done soon and you have time to help with that, you can create some Python class which takes the list of spec files and orders them. This would be easy to integrate to MBS and maybe the code could be shared across different distributions and work independently from MBS?
@jkaluza So a good chunk of this already exists in Koschei. And @mizdebsk's version of this script uses that.
I could probably make such a thing as a library module, similar to perl-BSSolv except for Python. But if I did that, I'd rather see us start looking at integrating this at the Koji level so that regular packages benefited from this.
This stuff is useful for normal packages too, and it's frustrating that MBS is the only path anyone is making packaging improvements these days...
Login to comment on this ticket.