As part of ELN's intent to provide a staging ground for RHEL-targeted functionality, I'd like to request that we are permitted to enable default module streams like they would appear in RHEL.
This will be restricted to the repositories we create for ELN and will not leak to regular Fedora unless a user opts in voluntarily.
This will also provide the Modularity team a place to test solutions for the known upgrade issues.
CC @jorton @bookwar @jmracek
will not leak to regular Fedora unless a user opts in voluntarily
I am :confused: , what does that mean?
Meaning that they intentionally enable the ELN repositories. This will not be done without a conscious choice.
I'm actually thinking if such things should be decided by EPSCo instead given that ELN is basically what will become RHEL/EPEL in future.
Nevertheless, I'm ±0 here.
Emphatically, no. ELN is Fedora. It's a version of Fedora that is clearly delineated for eventually becoming a new RHEL.
ELN is Fedora
Who decides what gets modularized in ELN?
IMHO Either Fedora decides and hence the need for a Fedora modular policy still stands, or Red Hat decides and hence ELN isn't Fedora.
ELN is Fedora Who decides what gets modularized in ELN?
The maintainer of a package decides whether to build it as a module, just as they do for Fedora. Any packager can ask to be added as a comaintainer to create a module stream as well.
Unless by "gets modularized" you mean "is made default", in which case I'd like to propose the ELN SIG be delegated this privilege by FESCo (once we get this SIG properly launched).
I think FESCo decides (or can delegate the decision) and yes, ELN will need to establish a policy. I don't think ELN and Fedora necessarily need to have the same policy, however.
I'd like to see the policy before we allow this.
Suggestion for an ELN policy: "Modules may set their default stream to be the same as the most-recently-released major version RHEL. If they wish to set a default to any other stream, it must be approved by FESCo (or at FESCo's discretion, delegated to the ELN SIG)."
Please also note that ELN repositories are not (and must never) be installed automatically on any Fedora system. So setting defaults in ELN will have no effect on non-ELN Fedora systems.
Ticket #2114 is still open and still active. At least in the context that we would like to see some sort of decision made on the points raised.
I would like to see that resolved and clarified first. Allowing default module streams now for ELN in effect is making the decision that modules as-is are continuing. If ELN is Fedora, then #2114 needs resolution before this change should be considered.
Exactly.
Modules may set their default stream to be the same as the most-recently-released major version RHEL.
I don't think that's useful. This currently means RHEL8, and RHEL8 is based on F28. F28 is already ancient by Fedora standards. Even if RHEL8 has various updates over F28, they are often not the same as Fedora did. After all those are two independent development branches. I don't think we should base policy for F33 or F34 or F35 on the state of F28.
Instead, I think ELN should be what the next RHEL (currently 9) will be. Right now that means simply following rawhide.
--
Looking at this from a different side, any massive enablement of modules in ELN would make ELN significantly different from rawhide, increasing chance of build problems in ELN. And sadly, having modules in ELN makes it less likely that maintainers of those packages will fix problems in rawhide. Past (and current, sic) experience with modulular and non-modular content makes me very pessimistic about this. Overall, enabling modules just in ELN is in my opinion risky for both ELN and rawhide.
Proposal: ELN default modular streams follow the Fedora's policy on default modular streams. If the policy allows default modular streams at all, ELN may have different set of default modular streams than Fedora rawhide, following the general rule of ELN: "Package maintainers who wish their package to diverge in ELN should coordinate their changes with the ELN SIG ahead of time."
FTR I'm -1 to the original proposal.
I'd vote +1 for @churchyard's proposal (two comments up). If ELN is fedora, then it should follow the same rules.
@decathorpe
If ELN is fedora, then it should follow the same rules.
ELN is an attempt to decouple Fedora Rawhide content from Fedora build process. It intentionally diverges from the standard Fedora build process to allow certain level of experiments. Thus I think the blanket statement like this doesn't make sense.
Default stream configuration is external to both packages and modules. It is the repository metadata. I can have 10 different repositories generated from the same set of packages and modules with different metadata on default streams in them. It is no different from the compose pungi config which puts packages in different groups.
And we do have a separate branch for ELN compose and which is managed by ELN SIG.
@zbyszek
Future CentOS/RHEL includes modules. We can be afraid of it and try to stay away as much as possible, losing contributions, or we can keep playing with it on the side, with no damage done to Fedora releases, and see if it evolves into something better, which can be used in Fedora.
If we add default streams to ELN we will immediately hit all the Modularity problems we were discussing. These problems will be open, and clear and right in your face. And these problems will appear early and on the side, not blocking anything in Fedora.
I believe that if we've done ELN before Modularity we would actually have it much better understood by now. So I see this as attempt to do the experimental work in the right order.
Rather than changing Fedora policies, we would mess with them on the side. And then the practical policy may come out of it. And it would be much more valuable than any document we can write now out of thin air.
Having default module streams in ELN makes it less likely that Red Hat maintainers of those packages will fix problems in rawhide. We have seen this when we had default module streams in Fedora.
@decathorpe If ELN is fedora, then it should follow the same rules. ELN is an attempt to decouple Fedora Rawhide content from Fedora build process. It intentionally diverges from the standard Fedora build process to allow certain level of experiments. Thus I think the blanket statement like this doesn't make sense.
I disagree. If it's going to diverge then why is it working to be part of Fedora? My understanding here is that we are working to amend policies to both allow ELN to build and develop but also to make it something useful for contributors to Fedora and RHEL and CentOS. Otherwise it's just a dumping ground.
Default stream configuration is external to both packages and modules. It is the repository metadata. I can have 10 different repositories generated from the same set of packages and modules with different metadata on default streams in them. It is no different from the compose pungi config which puts packages in different groups. And we do have a separate branch for ELN compose and which is managed by ELN SIG.
Just a side comment on the ELN SIG managing the separate branch... I see scaling problems there. If the intent is to have the ELN SIG bridge the ELN process to existing package maintainers, that's fine. But if the ELN SIG is intending to manage all divergent ELN branches of all packages, that means they are effectively making a completely separate distribution and we lose the coordination with Fedora which I thought was one of the major points of ELN in the first place.
@zbyszek Instead, I think ELN should be what the next RHEL (currently 9) will be. Right now that means simply following rawhide. Future CentOS/RHEL includes modules. We can be afraid of it and try to stay away as much as possible, losing contributions, or we can keep playing with it on the side, with no damage done to Fedora releases, and see if it evolves into something better, which can be used in Fedora. If we add default streams to ELN we will immediately hit all the Modularity problems we were discussing. These problems will be open, and clear and right in your face. And these problems will appear early and on the side, not blocking anything in Fedora.
Future CentOS/RHEL includes modules. We can be afraid of it and try to stay away as much as possible, losing contributions, or we can keep playing with it on the side, with no damage done to Fedora releases, and see if it evolves into something better, which can be used in Fedora. If we add default streams to ELN we will immediately hit all the Modularity problems we were discussing. These problems will be open, and clear and right in your face. And these problems will appear early and on the side, not blocking anything in Fedora.
I don't think anyone is disagreeing with that. The missing piece here is what are the solutions to the Modularity problems we know that exists? That's what I want to see answered/addressed in the other ticket. Unless we have a policy, guidance, and other decisions made then we'll just be opening the door to 50 module problems mean 50 different solutions. Everyone will have a different idea of what is "correct".
That's probably true.
We have a chance to get things right, maybe, again. If we hold the dnf to helping resolve the other ticket and form an understood scope for Modularity and policies for packaging then we can maybe avoid some of the pain we've already seen.
And it's not just technical issues...we need policy around what having a module in Fedora means. The regular package needs maintenance, as @churchyard has pointed out. We can't have the module owners abandon regular packages. Or if they do we need a process for that. Collect a new volunteer for the existing package, etc.
* AGREED: Defer to the next meeting. Meanwhile sgallagh will work on improving policy for those modules with feedback from FESCo members. (+9, 0, -0) (contyk, 16:52:36)
Metadata Update from @ignatenkobrain: - Issue tagged with: meeting
Metadata Update from @ignatenkobrain: - Issue assigned to sgallagh
The feedback should be provided in https://pagure.io/fedora-docs/modularity/pull-request/83.
One thing that I have not mentioned in the meeting: AFAIK this question was never discussed on the devel mailing list. Would it make sense to do so?
A question about the implementation: https://fedoraproject.org/wiki/Changes/ELN_Buildroot_and_Compose#How_will_ELN_handle_side-tags.3F
ELN will take no action on the creation or lifetime of a side-tag. When the side-tag is merged back to Rawhide, at this time ELN builds will be scheduled for each included package.
So let's say that we have a situation where there are 3 packages, A, B, C, and in ELN A and C are normal, but B is from a default stream. And the rawhide packager builds A, B, C in a side-tag because they need to be rebuilt tother. IIUC, according to the description above, A, C would be untagged, but not B, since it comes from a default stream which overrides normal packages. Is this what would happen? How are such rebuilds supposed to work?
One other thing that I don't think was said (and certainly is not mentioned in the ELN proposal) and I'd like to hear explicitly wrt modules in ELN. What are they? I.e. what of the following list is true (and what is missing?):
Modules in ELN consist of:
How ELN can rebuild any module, if it is not clear which modules in Fedora are maintained? There is not even defined what modular master branch in Fedora is. Fedora mass rebuild was from this POV just bad joke and I don't want to see this repeated for ELN.
master
@churchyard
We are aiming for "ELN-buildroot rebuilds of the modular streams built for Fedora rawhide".
@vondruch We are not going to rebuild every module for ELN. Module streams will be added in ELN only if maintainer of the module explicitly requests it, and if ELN SIG approves it. Which means ELN SIG can put additional requirements on the module content and support.
That'd simplify the whole discussion a lot. Since default module streams are not present in rawhide, then module streams built in rawhide must build fine without default streams, so I'd expect that they also build fine in eln. So maybe this ticket is moot since default module streams are not needed?
Note that default streams are configured separately anyway, so any such default streams would be under an ELN SIG version of releng/fedora-module-defaults anyway.
I would make it a requirement to have modular defaults managed similarly, and the tooling to expressly only apply module defaults to ELN overlay repository.
Default streams for "leaf" modules is a lighter/easier case. As Neal said we can deal with default stream definition in the metadata of the eln compose config, or even via side repository with modular overrides.
But it is not enough.
We also want to make some default streams available in the ELN Buildroot itself. We want to allow certain non-modular Fedora packages (taken from the standard Rawhide master) to be built using modularized build dependencies, provided by default streams in the ELN buildroot. Similarly to how it is done in downstream.
Adding modules on top is easy. Adding modules in the base buildroot layer, without changing non-modular packages which depend on it, is what we would like to test.
Adding modules in the base buildroot layer, without changing non-modular packages which depend on it, is what we would like to test.
The problem with this specifically is that we don't have infrastructure to do this right now. Do you have have a solution for implementing this?
Note, the last time we looked at this, we talked about Ursa Major as the solution for this. Is this your current plan for this? If so, how do you intend to ensure it is exclusively restricted to the ELN overlay?
Adding modules in the base buildroot layer, without changing non-modular packages which depend on it, is what we would like to test. The problem with this specifically is that we don't have infrastructure to do this right now. Do you have have a solution for implementing this? Note, the last time we looked at this, we talked about Ursa Major as the solution for this. Is this your current plan for this? If so, how do you intend to ensure it is exclusively restricted to the ELN overlay?
The problem with this specifically is that we don't have infrastructure to do this right now. Do you have have a solution for implementing this? Note, the last time we looked at this, we talked about Ursa Major as the solution for this. Is this your current plan for this? If so, how do you intend to ensure it is exclusively restricted to the ELN overlay?
This information is far out of date. We don't require Ursa Major any longer. Changes to Koji and Pungi are in production to ensure that when the buildroot repo is generated, it includes any contents listed as default in releng/fedora-module-defaults (and supports defaults in the buildroot only via the overrides subdirectory therein).
overrides
We are not going to rebuild every module for ELN. Module streams will be added in ELN only if maintainer of the module explicitly requests it, and if ELN SIG approves it. Which means ELN SIG can put additional requirements on the module content and support.
vs. https://pagure.io/fedora-docs/modularity/pull-request/83#comment-123642
Specifically, I'm looking to extend the module stream expansion feature of MBS such that we can tell it to build for ELN whenever we build for Rawhide (and disallow building for ELN independently). In other words, a modulemd specifying platform: [] would expand to [f31, f32, f33, eln, epel8]
Specifically, I'm looking to extend the module stream expansion feature of MBS such that we can tell it to build for ELN whenever we build for Rawhide (and disallow building for ELN independently).
In other words, a modulemd specifying platform: [] would expand to [f31, f32, f33, eln, epel8]
I'm confused.
We are not going to rebuild every module for ELN. Module streams will be added in ELN only if maintainer of the module explicitly requests it, and if ELN SIG approves it. Which means ELN SIG can put additional requirements on the module content and support. vs. https://pagure.io/fedora-docs/modularity/pull-request/83#comment-123642 Specifically, I'm looking to extend the module stream expansion feature of MBS such that we can tell it to build for ELN whenever we build for Rawhide (and disallow building for ELN independently). In other words, a modulemd specifying platform: [] would expand to [f31, f32, f33, eln, epel8] I'm confused.
Yeah, @bookwar and I appear to have a disconnect here. I'll discuss with her and come back to this part. Sorry for the confusion.
OK, it was a long day yesterday. I somehow forgot entirely about a meeting where we had agreed to do things differently. The version I wrote in the PR you referenced was an old approach that we determined is incorrect.
So @bookwar was correct, our current plan is as follows (and I'll update the policy doc proposal accordingly):
platform: eln
platform: f33
platform: []
f33
Adding modules in the base buildroot layer, without changing non-modular packages which depend on it, is what we would like to test. The problem with this specifically is that we don't have infrastructure to do this right now. Do you have have a solution for implementing this? Note, the last time we looked at this, we talked about Ursa Major as the solution for this. Is this your current plan for this? If so, how do you intend to ensure it is exclusively restricted to the ELN overlay? This information is far out of date. We don't require Ursa Major any longer. Changes to Koji and Pungi are in production to ensure that when the buildroot repo is generated, it includes any contents listed as default in releng/fedora-module-defaults (and supports defaults in the buildroot only via the overrides subdirectory therein).
And that includes the modular repository metadata so that module_hotfixes=1 isn't required? Or is that still needed for this? Basically, do module dependencies actually get evaluated in here? I want to make sure that if you're doing this, module dependency errors can happen in the build environment. We're mostly worked off the principle that build-time dependency resolution works the same way run-time dependency resolution does (barring the state of evaluating weak dependencies), and in this case, I think it's pretty critical that we don't wind up with a situation where invalid combinations are possible.
module_hotfixes=1
Yes, the whole point of this implementation (which we called "Ursa Prime", though it's not a service but a project name) was to ensure that Koji/mock buildroot creation behaved the same way as runtime installation.
So we now include the module metadata in the generated Koji buildroot and rely on DNF in mock to report conflicts if they exist.
My understanding of ursa prime (which we never finished deploying) was that we would make a ODCS compose or a cronjob that composed all the modules into a repo (possibly changing if they were default or not), then add that as an external repo to the rawhide buildroot. This would allow us to regenerate it as needed/quicker than the modular compose (daily) and allow us to update it for module A needing a updated module B to build. If we do this for eln, we would need to know what modules to compose into it and what ones are default or not.
Metadata Update from @ignatenkobrain: - Issue untagged with: meeting
Closing this we will be voting on the policy update on the Change Proposal once it is ready.
Metadata Update from @cverna: - Issue close_status updated to: Invalid - Issue status updated to: Closed (was: Open)
Login to comment on this ticket.