One example I'd like to see is Node.js: I'd like to drop the ursine nodejs package entirely and rely on the default stream of it to be available to the buildroot for non-modular packages to use.

The reason for this is that nodejs is a (direct or indirect) build-time-only dependency for a lot of packages. I want to have them available as modules for purposes of update stability between releases, for applications that rely on it, but right now I have to keep both the ursine and modular versions available for people to be able to build with it.

I want to do the same for libgit2* stack.

With my third-party packager hat on, if this happens, I'm probably going to be forced to drop building Fedora packages wherever modules are in the build-dep chain. I don't use Koji, Mock, or DNF in my package build tooling, so none of this works at all for me.

And I would be shocked if I was the only one like this. I just happen to be the guy who can actually say something since I track basically everything going on here...

@ngompa How do you pull your build dependencies if not with mock or DNF? That would be useful information to have. What I'm asking for as a first-order approximation is that any module stream that is a default stream should be available as part of the standard buildroot.

Support for non-default streams is also interesting, but I'm not asking for that right now.

At my workplace, we use an Open Build Service instance to build packages.

That means, we're using obs-build as the build engine instead of Mock, and the OBS backend is actually doing the depsolving (with perl-BSSolv) and fetching (using bs_dodup).

Thus, our build system is using libsolv directly.

@ngompa Ursa-major won't stop you from using current libsolv-based dependency solvers. Ursa-major does not make Koji use modules, but module contents, which are just RPMs which dependencies can be solved with libsolv. No module-specific DNF code would be used in resolving build dependencies after enabling ursa-major.

Let me explain how ursa-major affects Koji buildroots in more detail.
Current inheritance for ursine build tag f30-build is:

f30-build (4380)
  └─f30-override (4379)
     └─f30-updates (4373)
        └─f30 (4368)

What ursa-major does is inserting module tags into appropriate place of this tag inheritance tree. For example:

f30-build (4380)
  └─f30-override (4379)
     ├─module-nodejs-10-20180920144631-a5b0195c (4770)
     ├─module-javapackages-tools-201801-20180806152918-b43b0c8f (4300)
     └─f30-updates (4373)
        └─f30 (4368)

This way contents of selected modules are included in buildroot of non-modular packages. Modules temselves are not added to buildroots. Module contents take precedence over contents of f30 and f30-updates, but can still be overriden by buildroot overrides.

f30-build repository generated by Koji from this tag tree does not contain module metadata. It is still plain-old non-modular YUM repository.

But that's not how they get exported to the public repos (which is what I consume for package builds).

Modules are in a separate repository with foreign metadata that isn't understood by any other tools.

For example, I build pagure in OBS for testing and development (producing packages with obs-build and appliance images using kiwi). pagure requires pygit2, which in turn requires libgit2. If all the git2 ecosystem is packaged exclusively as a module with this system, it works fine for pagure packages built in Fedora through Koji (and maybe COPR, with some effort), and installation will work too, since DNF is the package manager resolving at install time and can handle module repos.

However, for me building in OBS on my system, this is completely broken because pygit2 as a module is not sanely resolvable.

One way to solve this problem would be having a separate compose variant containing contents of buildroot. Repos in this variant would be non-modular.

I don't know if we want to go down the road of adding more to the compose process. The current compose already takes half a day to complete, and it's already rife with peril...

Then you can use Koji repos instead of content released on mirrors.

But because we don't sign our repodata, I can't validate the input before using it, which is a prerequisite with OBS. It's either validated through checksums with an SSL-enabled master mirror, or through GPG checking of the repomd.xml.

That's a good point. Anyhow, I'm sure this can be solved somehow and it shouldn't stop ursa-major from being implemented.

My problem is only with @sgallagh and @ignatenkobrain wanting to remove non-module packages and using modules as build and run deps for non-module packages.

The point is to remove non-modular packages from distribution and replace them with packages built as modules. Non-modular packages would still buildrequire individual packages, which may be part of module contents, but otherwise are still just packages. Packages won't start buildrequiring modules as RPM doesn't have a way to express dependency on a module. With ursa-major buildroots are still module-unaware and some release repos are still modular. Ursa-major doesn't change anything in this aspect. What it does is de-duplication of packages - buildroots use the same packages as shipped to users.

So, how about this for a solution (credit to @sgallagh for the idea, any mistakes in explaining it are mine):

As part of rawhide/branched and updates composes we create and push a new repo 'buildroot'.
This repo contains no packages, just repodata that points to Everything and Modular for all packages that are in the buildroot at the time of compose (modular and non modular). Probibly just default modular streams, but I guess we could include others.
Users of copr and osbs and mock can adjust their config to point to this repo instead of Everything and use it for building against fedora.

Unclear how we point the repodata, ideally it would download those from a metalink, so we would need mirrormanager to see it.
cc @adrian

This might be something to add to pungi, so cc @lsedlar too

+1 to the change and something to consider is how long it will add to do the buildroot compose.

And, do we need to add this new metalink to fedora-repos? I guess not, but just asking?

As part of rawhide/branched and updates composes we create and push a new repo 'buildroot'.
This repo contains no packages, just repodata that points to Everything and Modular for all packages that are in the buildroot at the time of compose (modular and non modular). Probibly just default modular streams, but I guess we could include others.

Aside from default streams, I think we should only allow non-default streams to be configured if they meet the following criteria:

• Produces no RPM artifacts that override content in Everything
• Has no default stream. (We don't want to override ourselves either)

I think that buildroot repository should contain the same packages as included in Koji build repos (possibly with exception for buildroot overrides, but I think it would still be better to somehow include these too). Modules that are not included in buildroot via ursa-major should not appear in buildroot repo.

What kind of modules (only default-streams, no conflict with Everything etc.) would be allowed in ursa-major is a question of policy. But with too strict policy it won't be possible to transition to module-only content as this is a chicken-egg problem: you can't retire base packages as they are still used in buildroot and you can't replace them with a module as it would override base packages. With strict policy like that there would still need to be some exception process to allow non-conforming modules to be conditionally allowed in buildroot.

Moreover, buildroot repostiory is not really dependant on ursa-major. It could be implemented now and it would contain contents of f30-build (and equivalent) tags, except overrides. Once ursa-major is implemented, contents of modules added to Koji tag inheritance by it would automagically appear in the buildroot repository.

Being able to depend on modules at buildtime for pure rpms is critical for keeping e.g. Eclipse + friends in Fedora now that Maven goes module only.

How would this work for updates? Typically, over the life of a Fedora release, there are many updates, and packages can depend on other packages that were shipped as updates. How would we handle this for that part too?

I filed https://pagure.io/fesco/issue/2003 on this to ask fesco to weigh in on this request.

Closing this as we are not going to use ursa-major.