#2003 Modules in buildroot enablement
Opened 10 months ago by kevin. Modified 2 months ago

Greetings FESCo.

Please see https://pagure.io/releng/issue/7840 for full details, but releng has been asked to enable/setup/run ursa-major. This is a app that manages adding modules to the buildroot, so that non modular packages can build against them. releng has been reluctant to do this because it means that end users who build packages will now possibly not be able to do so without figuring out what rpms to add to their buildroot from what modules.

We came up with a possible compromise: creating a 'buildroot' repo thats simply repodata pointing to the normal rpms/modular rpms, so people could enable that repo and pull in the same things we used in our buildroot. That would still require a change.

I'm asking fesco to look at this and decide under what conditions enabling this is acceptable to the project.


Ursa Major is definitely one of the Modularity WG goals (https://tree.taiga.io/project/modularity-wg/epic/12).

Still, I would like to have more details on that proposed buildroot repo and how it would be implemented. Keep in mind we install with DNF and if we just point to existing repos, DNF will process the modular metadata. We would need to somehow enable the modules that Ursa Major is using and disable everything else. Or provide a set of new defaults for this.

Just using the koji repo is very simple and effective but as was noted in the ticket, this is not signed.

Metadata Update from @jforbes:
- Issue tagged with: meeting

10 months ago

AGREED: an email will be sent to the devel list for further discussion and we will review it again in 1 week (+5,0,0)

This was discussed in today's meeting (2018-11-05):
AGREED: Let's wait another week for discussion on fedora-devel (+5, 0, 0) (zbyszek, 15:12:12)
ACTION: jforbes to send out an e-mail to devel mailing list (zbyszek, 15:09:46)

ACTION is ready done: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/FPRKTPKND7DZY6JYLRQFMLULVXPA7W4G/

We will discuss this in the FESCo meeting Monday at 15:00UTC in #fedora-meeting-1 on
irc.freenode.net.

We had a lengthy discussion about this today, and decided to have more discussion here in ticket:

ACTION: contyk to write up a summary in the ticket

Metadata Update from @bowlofeggs:
- Issue untagged with: meeting
- Issue assigned to psabata

9 months ago

There isn't much to summarize when it comes to the disucssion last week so I'll just try to explain what Ursa Major is, what it does and how.

UM makes modular packages available in the standard buildroot for non-modular content by altering the inheritance chain of the buildroot tag. It watches for module build messages on the bus and updates the inheritance whenever a new build of the module becomes available for inclusion (the ready state in MBS).

Not all modules are automatically included. UM has its own configuration file with modules and streams associated with each buildroot. There is some functional overlap between the "buildroot modules" and the default streams but not all default streams need to be in the buildroot and not all "buildroot modules" need to be part of the compose or have default streams.

Packages have no way to express dependencies on modules so non-modular packages cannot buildrequire content from modules not in the buildroot. If a package requires such content, it has to be modularized.

In most cases, modules that are present in the buildroot and have defaults should be using the same stream in both situations, although there might be exceptions and this should be left to the maintainers or SIGs to decide. I suggest Release Engineering owns the UM configuration repository, just like they own the defaults, and package maintainers are responsible for submitting changes.

A side note: If a module buildrequires a conflicting stream, MBS masks the "buildroot module" for the modular build to avoid parallel module installation. UM should also inject artifical conflicts into the buildroot to make filtered modular packages unavailable after each inheritance update. There are technical challenges that need to be resolved here and I might not have the most up-to-date information on this. A comment from Factory2 would be appreciated.

Would it be acceptable (in the sense of fulfilling the original technical goal), to simply hardcode that all default modules are enabled in the UM config used for koji?

Packages have no way to express dependencies on modules so non-modular packages cannot buildrequire content from modules not in the buildroot.

This bit seems like a problem to me. Right now we require our packages to clearly express their dependencies. If we make it so there's no way to do that, we won't be able to tell which packages need to be rebuilt when BR's are updated. This is particularly important for statically compiled programs (like Go).

On Mon, 2018-11-26 at 14:20 +0000, Petr =C5=A0abata wrote:

UM makes modular packages available in the standard buildroot for
non-modular content by altering the inheritance chain of the
buildroot tag.

Does UM's config file allow expressing BRs per package, or it it just
one BR that all packages share?

Would it be acceptable (in the sense of fulfilling the original technical goal), to simply hardcode that all default modules are enabled in the UM config used for koji?

Maybe but why?

Packages have no way to express dependencies on modules so non-modular packages cannot buildrequire content from modules not in the buildroot.

This bit seems like a problem to me. Right now we require our packages to clearly express their dependencies. If we make it so there's no way to do that, we won't be able to tell which packages need to be rebuilt when BR's are updated. This is particularly important for statically compiled programs (like Go).

They still have standard package level dependencies. Modular packages pulled into the standard buildroot are just that. Not having RPM-level dependencies on modules means you just can't select a set that is not available in the repo. I don't find that weird.

UM makes modular packages available in the standard buildroot for
non-modular content by altering the inheritance chain of the
buildroot tag.

Does UM's config file allow expressing BRs per package, or it it just
one BR that all packages share?

The UM config is a list of module+streams associated with koji tags. You just say "I want the packages of nodejs:8 in the Rawhide buildroot" and it adds the tag representing the latest build of the nodejs:8 modules to the rawhide build tag inheritance (and maintains that link). This just makes the packages of that module available to everyone, just as if they were non-modular. It's rather simple.

Maybe but why?

If the default streams are enabled by default, there doesn't need to be a separate list maintained by releng. This makes the koji build repos less special than other environments, and easier to explain.

On Mon, 2018-11-26 at 14:58 +0000, Petr =C5=A0abata wrote:

They still have standard package level dependencies. Modular packages
pulled into the standard buildroot are just that. Not having RPM-
level dependencies on modules means you just can't select a set that
is not available in the repo. I don't find that weird.

I think I'm not quite understanding. Are you saying that spec files
that depend on modules still have to explicitly write a BuildRequires
line for RPMs from that module? If so, I think that addresses my
concern. If not, I think I need to understand it more =E2=98=BA

On Mon, 2018-11-26 at 14:58 +0000, Petr =C5=A0abata wrote:

The UM config is a list of module+streams associated with koji tags.
You just say "I want the packages of nodejs:8 in the Rawhide
buildroot" and it adds the tag representing the latest build of the
nodejs:8 modules to the rawhide build tag inheritance (and
maintains that link). This just makes the packages of that module
available to everyone, just as if they were non-modular. It's rather
simple.

Does this mean that people who want modules in the BuildRoot will make
the BR bigger for everyone, even packages that don't use those modules?

@zbyszek, we could do that and currently it probably wouldn't hurt anything but I'm not sure if sacrificing the flexibility this offers is worth it. It would also require additional changes to UM to understand the defaults structure and how to properly load them.

@bowlofeggs, you can imagine modules are repositories. RPMs cannot depend on repositories but they still need proper RPM-level dependencies to build. MBS and UM manage what repositories are available to you -- MBS does it for module builds based on the module-level buildrequires, UM does it for everyone else based on its configuration.

Adding more "repositories" to the buildroot doesn't mean that the buildroot will grow for individual packages. It just makes the available package set larger, just like adding more packages to the distribution.

On Tue, 2018-11-27 at 10:58 +0000, Petr =C5=A0abata wrote:

@bowlofeggs, you can imagine modules are repositories. RPMs cannot
depend on repositories but they still need proper RPM-level
dependencies to build. MBS and UM manage what repositories are
available to you -- MBS does it for module builds based on the
module-level buildrequires, UM does it for everyone else based on its
configuration.
=20
Adding more "repositories" to the buildroot doesn't mean that the
buildroot will grow for individual packages. It just makes the
available package set larger, just like adding more packages to the
distribution.

I think I understand now, thanks for that explanation!

I'd like to note that epel really would like ursa-major or something like it to use with epel8.

Since rhel8 has tons of modular content, the only way epel8 could build against it is via making epel8 content modular, or having something like ursa-major so we could build against (some) of those modules. We could of course just require everything to be modular, but that seems heavy weight for just a single package for example.

In most cases, modules that are present in the buildroot and have defaults should be using the same stream in both situations, although there might be exceptions and this should be left to the maintainers or SIGs to decide. I suggest Release Engineering owns the UM configuration repository, just like they own the defaults, and package maintainers are responsible for submitting changes.

@zbyszek, we could do that and currently it probably wouldn't hurt anything but I'm not sure if sacrificing the flexibility this offers is worth it. It would also require additional changes to UM to understand the defaults structure and how to properly load them.

There is already too much manual workload for releng, therefore it would be great if we could just enable all the module repos instead of requiring a manual process for releng.

There is already too much manual workload for releng, therefore it would be great if we could just enable all the module repos instead of requiring a manual process for releng.

We can't, because modules are designed to conflict with each other. At a package level, you'd basically have broken or confused dependencies all the time.

Metadata Update from @psabata:
- Issue tagged with: meeting

8 months ago

What @ngompa says is correct, although I assume @till actually meant just the defaults, which would be fine.

@mizdebsk, as a potential user of "buildroot-only modules", would such a decision limit you significantly?

As long as buildroot-only modules are allowed to have default streams, limiting module builds available in ursine buildroots to default streams only does not affect my use of buildroot-only modules. However in general I see this as a significant limitation.

I think we should deploy UM with all default streams enabled, and also dump all those rpms into a repo for consumption by external users per @nirik's description [https://pagure.io/releng/issue/7840#comment-533905].

If it turns out we need more flexibility, we can always introduce a human-maintained list later on.

Metadata Update from @psabata:
- Issue untagged with: meeting

8 months ago

From today's meeting:

  * There two major points that need to be addressed regarding Ursa
    Major. #1 Make it painless for package maintainers and releng to
    maintain the UM config, and #2 Make it reasonably easy for non-koji
    users to consume our buildroot content  (contyk, 16:07:21)
  * We will discuss these topics in the ticket  (contyk, 16:07:37)

To address the first point:

We could have a fedora-module-defaults commit hook that would generate Ursa Major configuration for the relevant tags, including all the defaults in the buildroot. To accomodate for buildroot-only/building block modules, we could concatenate this with additional separately maintained configs. Changing the latter would require FESCo approval. I believe this should be relatively painless for everyone while retaining the flexibility separate Ursa Major configuration offers.

To address the second point:

I think the only workable option here are buildroot composes. We just have to figure out the details -- they should be cheap to produce and as similar to the environment in koji as possible (preferrably identical). This will likely require some changes to MBS local (mock) builds to prevent stream collisions.

@psabata for people who run local mirrors and use those local mirrors for their packaging purposes in order to speed up the mock chroots bootstrapping, is the koji buildroots something they can reasonably mirror or at least caching reverse proxy easily?

@maxamillion, koji repos are perfectly sufficient.

@maxamillion Koji repos are not signed, so keep that in mind for consumers of that.

Metadata Update from @zbyszek:
- Issue tagged with: meeting

8 months ago

This was discussed in today's FESCo meeting:
AGREED: We'll gather more information about the required changed to
pungi and additional resources required for this and revisit next
week (+7, 0, 0)

See https://pagure.io/pungi/issue/1095.

Just an update: Holidays and PTO make the discussions quite difficult. I think we should revisit this next year.

Metadata Update from @psabata:
- Issue untagged with: meeting

8 months ago

I guess thats ok, but it would be nice to figure a way forward for epel in the mean time.

Perhaps we could discuss that on the epel-devel list or the like.

Metadata Update from @sgallagh:
- Issue tagged with: meeting

7 months ago

There were some agreements on the meeting of 2019-01-14:

AGREED: FESCo will permit standing up an Ursa Major instance to be configured for use only in EPEL 8 this time. (+5, 0, -1)

AGREED: FESCo would like a set of detailed requirements gathered regarding packager experience for local builds. Once approved, these will need to be met before enabling Ursa Major in Fedora. (+6, 0, -0)

ACTION: sgallagh to gather requirements

(Not sure if this is supposed to be closed or not.)

Metadata Update from @churchyard:
- Issue untagged with: meeting

7 months ago

The resolutions that have to be made should be announced. @sgallagh: do you want this ticket open to track the follow-up?

Metadata Update from @zbyszek:
- Issue tagged with: pending announcement

7 months ago

Metadata Update from @churchyard:
- Issue untagged with: pending announcement

6 months ago

I'm not closing this yes, there is an action for @sgallagh and the original proposal was not accepted.

Metadata Update from @churchyard:
- Issue assigned to sgallagh (was: psabata)

6 months ago

@sgallagh In cca 6 weeks, 1525 Fedora packlages will be FTBFS if we don't get an army of volunteers for @mizdebsk's Java packages or if we don't get this sorted out. I don't particularly need LibreOffice, so I think I can live without it, but chances are others grew fond of it. JFYI this now has a high priority.

While I understand that everybody has right to orphan their packages, this certainly has the "modularity broke Fedora" bitter tase in it.

useless meme

I know there is work being done to get default module streams into the buildroot without using the unpopular ursa-major solution. I'll ping around. But I'm not sure that will be ready for f30.

Metadata Update from @psabata:
- Issue tagged with: meeting

6 months ago
  * AGREED: contyk will prepare & share a proposal solving the buildroot
    problem without Ursa Major. We will revisit next week. (+7, 0, -0)
    (contyk, 15:48:53)

Furthermore we discussed what to do with those java packages if this is not done in time. The overall opinion was to follow the standard process and let interested parties maintain the ursine versions.

Metadata Update from @psabata:
- Issue untagged with: meeting

6 months ago

I won't make it to today's meeting but I've shared the write-up with the stakeholders today, asking for sign-offs.

Public draft: https://docs.google.com/document/d/1NxBbTPVJ17Ax4k-NbtorzxYspJ_CsjQ9m3In2wqWqVU/edit?usp=sharing

For EPEL 8, we need to decompose RHEL 8 BaseOS and AppStream repositories into one purely non-modular repository and one repository for each AppStream module

Wait, there's a way to decompose modules back into individual standard repos? Why can't we just do that across the board? That would fix virtually all ecosystem compatibility issues, too...

Metadata Update from @zbyszek:
- Issue tagged with: meeting

6 months ago

I read the draft. The Motivation and Current implementation sections make a lot of sense and are written in a very detailed and clear manner. But then we get to the actual proposal, and it's a) brief, b) rather vague.

Koji needs to be updated so that build tag repositories can be created from more than one source -- the tag contents and zero or more external repositories.

Where are those repositories going to live? How many will there be (order of magnitude)? Who will decide which ones koji should include? How will this be communicated to koji?

The resulting repository must preserve potential modular metadata included in these repositories if the tag configuration says so.

Can you please very verbosely explain what this second sentence means? My understanding was that this ticket is about allowing non-modular content to be built against modules, which means that at some point the modular metadata is not supposed to propagate, and we have some "normal" repositories. But if the modular metadata is still required, how is this supposed to be consumed by consumers who don't understand modules?

Also, "if tag configuration says so" — please expand how the tag configuration does that.

Wait, there's a way to decompose modules back into individual standard repos? Why can't we just do that across the board? That would fix virtually all ecosystem compatibility issues, too...

Of course you can rearrange the RPMs into different repos. I'm not really following what you're suggesting here -- shipping modules in their own repos? This came up early on in the modularity discussions but I'd like to know whether it's what you're talking about, or whether it's something else.

Metadata Update from @psabata:
- Issue untagged with: meeting

6 months ago

I read the draft. The Motivation and Current implementation sections make a lot of sense and are written in a very detailed and clear manner. But then we get to the actual proposal, and it's a) brief, b) rather vague.

Koji needs to be updated so that build tag repositories can be created from more than one source -- the tag contents and zero or more external repositories.

Where are those repositories going to live? How many will there be (order of magnitude)? Who will decide which ones koji should include? How will this be communicated to koji?

It's all open for discussion. Personally I think the simplest solution for Fedora would be having just one external repository with the modules supposed to be included in the non-modular buildroot, and source just this one repo into the build tag.

We previously talked what modules could or should be included in this buildroot and I believe the idea was to include all default modules, plus some potential additions. In essence this means it would be driven by package maintainers (and rel-eng since they maintain the defaults repo).

In EPEL, we could include them individually as their content and number would only be changed whenever RHEL gets an update. Again, RHEL defaults would be part of the buildroot. Or we could put them all into one repo as well, but we'd still need the individual repos for MBS.

The resulting repository must preserve potential modular metadata included in these repositories if the tag configuration says so.

Can you please very verbosely explain what this second sentence means? My understanding was that this ticket is about allowing non-modular content to be built against modules, which means that at some point the modular metadata is not supposed to propagate, and we have some "normal" repositories. But if the modular metadata is still required, how is this supposed to be consumed by consumers who don't understand modules?
Also, "if tag configuration says so" — please expand how the tag configuration does that.

Default modules have their package sets visible, even if they are not enabled. Installing content from such modules implicitly enables them. Therefore you can have a module foo with a bar component labeled as default (in the modular metadata) and installing bar with a simple dnf install bar will enable the module.

You can also have modules foo:a, foo:b and baz:1, where foo:a and baz:1 are the defaults. However, baz:1 depends on foo:b, which cannot be enabled together with foo:a. Installing content from foo:a will make baz:1 unavailable. Installing baz:1 will enable foo:b and make foo:a unavailable. This is not possible without modular metadata and modularity-aware tooling (such as DNF) in the buildroot and is one of the reasons why Ursa Major isn't sufficient here.

This will work transparently for non-modular content, unless they depend on packages from conflicting streams, which I would consider a packaging error.

The proposed solution would work for Fedora and EPEL 8 but you're correct consumers who use our buildroots yet don't use DNF (or other modularity-aware tools) inside those buildroots would struggle.

Adding tag flags that control whether source repository metadata should be preserved might be useful for MBS which manages the buildroot via inheritance. This might or might not be needed. To me it makes sense to have such a flag even just for backwards compatibility -- Koji doesn't preserve these today and it could only do that if a flag is set.

Wait, there's a way to decompose modules back into individual standard repos? Why can't we just do that across the board? That would fix virtually all ecosystem compatibility issues, too...

Of course you can rearrange the RPMs into different repos. I'm not really following what you're suggesting here -- shipping modules in their own repos? This came up early on in the modularity discussions but I'd like to know whether it's what you're talking about, or whether it's something else.

This is what I'm talking about, yes. If the specialized metadata was reduced to just indicating the module repo relationships (that is, if foo module repo requires bar module repo and conflicts with baz module repo), and the rest was just standard rpm-md, that would resolve pretty much all my issues because integrating it and leveraging the modules would be easy.

What would be great is if this could be synchronous - so when a module is tagged into f30 / f30-updates / f30-override (depending on where things are in the Fedora release cycle), the buildroot is marked as needing regeneration, a compose is queued, etc. What would be less good is if composes are in a cron job, and packagers have to wait an uncertain amount of time before a build picks up an update.

Having a repository available seems to solve many of the concerns - it will be possible to build in mock or copr. We'd still lose the 'dnf builddep' workflow - which is useful both for 'fedpkg local' and for hacking on upstream projects. I don't personally consider this a blocker - it is possible to enable the modules explicitly, and tooling could be improved to automate that.

In terms of whether these modules are implicitly in the buildroot for MBS-built modules, I can see arguments both ways:

Pro including: makes it easy to move packages into modules. We can enhance fedmod to add extra module BuildRequires to 'fedmod rpm2module' and 'fedmod rpm2flatpak', but those buildrequires would still need maintainance.

Pro not including: things are more straightforward - we're not counting on dnf to do some magic to resolve conflicts between the explicit BuildRequires of the package and the module default streams in the -build repo.

I think either way is acceptable.

Could there be a practical way to get the RPMs from the default modular stream to be in the default non-modular repos instead? i.e., rather than dnf pulling the RPMs from fedora, updates, module, and module-updates repos for users who only use the default stream, could we get the RPMs from the default stream to end up in fedora and updates, and let module and module-updates only have non-default stream RPMs?

I am fairly ignorant about modularity, so apologies if this doesn't even make sense.

Before this is done, a proposal:

Default Fedora mock configs should not include modular repos until Koji mock configs do so.

See https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/FNWYOOAUV73JUT3NKCZNW6TW66NGKJAK/ for context.

Now a dumb question about the original topic:

If mock can include modular repos why don't we do the same thing in Koji? Or is this the stuff that breaks it completely for external consumers (as @ngompa and possibly others noted)?

Because modular repos are composed by pungi.

Right. So the local mock can haz modular repo from the compose, but Koji uses Koji repos. Thanks, that makes sense. I've said the question is dump :)

Disclaimer: Mock only copies what is in fedora-repos. My proposal does not specify where is this fixed, it merely specifies the desired outcome (no modular repos in default mock configs until this is solved).

@ngompa:

So the reasons why it wasn't done that way was exactly the need for additional specialized metadata (originally even considered as a service). Then we did some performance testing for a large number of modules and the results were discouraging. Then there's also a question of mirroring. Of course everything can be implemented or resolved but we decided the current approach worked better.

@otaylor:

Thanks for the input. Including the modules directly in the platform module complicates things as MBS needs to know they are there and somehow disable them. With Ursa Major we made MBS aware of UM configs and have it generate conflicts in module-build-macros. This was ugly and there was a potential for race conditions. If the modules are included with their modular repodata, MBS would still need to be aware and somehow disable them. This would probably require additional koji features to accomplish. I think the best way is to let MBS add them for you but since it adds modules via inheritance, we still run into that problem where two defaults cannot be installed (and therefore inherited from) at the same time.

@churchyard:

The last proposal actually does just that -- we compose the defaults (or more) on the side and pull that repository into the buildroot.

And as noted in the thread, I'm in favor of dropping the fedora-modular repo from our current mock config. I don't think it should be there.

This is assigned to @sgallagh. Is there something for FESCo to do? What is the state of the proposal from @psabata? What is the reasonable timeframe to make this happen?

I'm asking because of https://pagure.io/fesco/issue/2104

IMHO People are very confused about the current situation and rightfully scared.

@psabata

1) Is there an up to date version of a doc describing Ursa Major structure somewhere?

2) Can you provide some examples for "not all buildroot modules need to be part of the compose or have default streams."? What is the reasoning behind those?

@bookwar The answer to 2) is that there are some modules that exist solely for bootstrapping which are maintained in the build infrastructure but are never distributed in a composed repository. They therefore also do not need a default stream.

Also, we should probably update this ticket to change the title. We are no longer looking to implement Ursa Major (which is a specific project and implementation of the buildroot generation) and are looking at something new. @contyk has a draft proposal in the works that he's been sharing with the Koji and Infrastructure teams, but I'm not sure if it's quite ready to bring to FESCo. I'll let @contyk make that decision.

@bookwar

1) Is there an up to date version of a doc describing Ursa Major structure somewhere?

I'm not sure the meaning of word "structure" here. If you would like to see some document, its README could be helpful, please refer to https://pagure.io/ursa-major

@cqi I think the confusion is that some people are still referring to Ursa Major as the thing we mean to replace Ursa Major with. That’s what @bookwar is asking for, I think. The most up to date document is the one @contyk linked earlier in this thread.

Also, as a status update, the mergerepo work needed to support the new plan was just accepted upstream this morning, which should unblock the Koji work.

Should we discuss this ticket during this week's meeting?

@bowlofeggs Nothing for FESCo to do at the moment. I'd take it off the agenda for now.

Just to keep this ticket fresh, there are no new decisions for FESCo to make at this point. Progress is continuing on the "Ursa Prime" solution described above. I'm hopeful that we will have a solution in staging within the next two weeks, though I think I heard @psabata mention there are a couple MBS issues remaining to be sorted out.

Metadata Update from @churchyard:
- Issue tagged with: stalled

2 months ago

Login to comment on this ticket.

Metadata