I don't understand what are you asking FESCo to do or decide here, sorry. Should we approach releng and ask them to prioritize the issue?

I don't understand what are you asking FESCo to do or decide here, sorry. Should we approach releng and ask them to prioritize the issue?

Sorry, I forgot to link the PRs I want merging. FESCo asked me to file this extra to explain the situation that I believe having a default stream will fix.

https://pagure.io/releng/fedora-module-defaults/pull-request/182
https://pagure.io/releng/fedora-module-defaults/pull-request/183

Based on what you said here in this ticket, it seems that currently, that would not fix it, but if https://pagure.io/releng/issue/9033 is resolved, it should, do I read that correctly?

There is a documented procedure about requesting a default stream. In particular, it says it should include this information:

• The list of SRPM components in the module
• Whether the contents of this module will obsolete and replace non-modular RPMs.
• Which releases of Fedora will this new default apply to (that's f31 and f32+ in this case)

It also says: "when requesting a change of default stream, strongly consider submitting a Fedora Change. "

Therefore I ask you to provide the list of the components and provide exact information about whether the contents of this module will obsolete and replace non-modular RPMs - here I mean whether they will be retired in rawhide (I know we cannot do it for f31). I also ask you to strongly consider filing a change proposal for Fedora 32.

The policy also mentions:

Please note that if the module stream masks part of the Traditional RPM repos (e.g it replaces an existing RPM or it introduces a non-trivial set of conflicts) it may not be made a default stream without the express permission of FESCo.

And since this is a FESCo ticket, everything is alright, but I thought we should mention in explicitly that this is indeed the case for Fedora 31 at least.

Based on what you said here in this ticket, it seems that currently, that would not fix it, but if https://pagure.io/releng/issue/9033 is resolved, it should, do I read that correctly?

No, that should fix scenario two above, but scenarios one and three are still broken: Users cannot upgrade or install Eclipse in the usual way unless the stream is made a default. The precise reason I filed this long ticket is that FESCo wanted details about the exact issue I face without the meetings turning into debugging sessions.

Unless I am missing something fundamental, dnf install eclipse or dnf groupinstall eclipse cannot work successfully on a freshly installed F31 system as the situation currently stands.

There is a documented procedure about requesting a default stream. In particular, it says it should include this information:

The list of SRPM components in the module
Whether the contents of this module will obsolete and replace non-modular RPMs.
Which releases of Fedora will this new default apply to (that's f31 and f32+ in this case)

I feel like we are going round in circles :-) it was my understanding that a default was approved by this ticket https://pagure.io/fesco/issue/2236 that was extensively discussed.

It is now a chicken-egg situation for me: Default stream can be merged when conflicts are resolved, I am saying here that conflicts will not go away until default stream is merged ;-)

It also says: "when requesting a change of default stream, strongly consider submitting a Fedora Change. "
Therefore I ask you to provide the list of the components and provide exact information about whether the contents of this module will obsolete and replace non-modular RPMs - here I mean whether they will be retired in rawhide (I know we cannot do it for f31). I also ask you to strongly consider filing a change proposal for Fedora 32.
The policy also mentions:

Please note that if the module stream masks part of the Traditional RPM repos (e.g it replaces an existing RPM or it introduces a non-trivial set of conflicts) it may not be made a default stream without the express permission of FESCo.

And since this is a FESCo ticket, everything is alright, but I thought we should mention in explicitly that this is indeed the case for Fedora 31 at least.

At the request of FESCo, it is the second such ticket, see https://pagure.io/fesco/issue/2236 for more

No, that should fix scenario two above, but scenarios one and three are still broken: Users cannot upgrade or install Eclipse in the usual way unless the stream is made a default. The precise reason I filed this long ticket is that FESCo wanted details about the exact issue I face without the meetings turning into debugging sessions.

Oh, this is misunderstanding. What I wanted to ask is:

Based on what you said here in this ticket, it seems that currently, making the stream the default would not fix the installation failure, but if https://pagure.io/releng/issue/9033 is resolved, it should, do I read that correctly?

Rephrased: "we need both things to happen to make it work"

I feel like we are going round in circles :-)

Please don't. I simply ask that you make a complete proposal that also includes the information about what is being retired/replaced precisely. Is including the list of SRPMs such a big ask?

To make it clear: I'm not trying to trick you. I am actually ready to vote in favor of this, however I consider that this ticket (even when combined with the previous one) lacks all the information that is needed to make a well informed decision. That's why the policy suggest very strongly that a change proposal should be filed. Since ti is not a hard requirement, you are free to ignore it, however I would expected that the missing information is provided in the ticket instead.

https://pagure.io/fesco/issue/2285#comment-612051 asks three questions. We have an answer to one of them, we need answers for the other too. Without that we're indeed treading water.

In rawhide, this doesn't seem to work either:

$ podman run -ti fedora:rawhide /bin/bash
[root@c7098f2a7de3 /]# dnf upgrade
...
[root@c7098f2a7de3 /]# dnf module install eclipse:latest
Last metadata expiration check: 0:02:18 ago on Tue Nov 19 09:58:39 2019.
Unable to resolve argument eclipse:latest
No match for package eclipse-egit-github
Unable to resolve argument eclipse:latest
No match for package eclipse-mpc
Unable to resolve argument eclipse:latest
No match for package eclipse-mylyn-context-java
Unable to resolve argument eclipse:latest
No match for package eclipse-mylyn-versions-git
Unable to resolve argument eclipse:latest
No match for package eclipse-mylyn-versions-subclipse
Error: Problems in request:
broken groups or modules: eclipse:latest, eclipse:latest, eclipse:latest, eclipse:latest, eclipse:latest
[root@c7098f2a7de3 /]# dnf module install eclipse:latest/java
Last metadata expiration check: 0:02:30 ago on Tue Nov 19 09:58:39 2019.
Unable to resolve argument eclipse:latest/java
No match for package eclipse-egit-github
Unable to resolve argument eclipse:latest/java
No match for package eclipse-mpc
Unable to resolve argument eclipse:latest/java
No match for package eclipse-mylyn-context-java
Unable to resolve argument eclipse:latest/java
No match for package eclipse-mylyn-versions-git
Unable to resolve argument eclipse:latest/java
No match for package eclipse-mylyn-versions-subclipse
Error: Problems in request:
broken groups or modules: eclipse:latest/java, eclipse:latest/java, eclipse:latest/java, eclipse:latest/java, eclipse:latest/java
[root@c7098f2a7de3 /]# dnf module enable eclipse:latest
Last metadata expiration check: 0:02:43 ago on Tue Nov 19 09:58:39 2019.
Dependencies resolved.
================================================================================
 Package           Architecture     Version             Repository         Size
================================================================================
Enabling module streams:
 ant                                1.10                                       
 eclipse                            latest                                     
 maven                              3.5                                        

Transaction Summary
================================================================================

Is this ok [y/N]: y
Complete!
[root@c7098f2a7de3 /]# dnf install eclipse
Last metadata expiration check: 0:03:06 ago on Tue Nov 19 09:58:39 2019.
Error: 
 Problem: package eclipse-jdt-1:4.11-3.fc31.noarch requires eclipse-platform = 1:4.11-3.fc31, but none of the providers can be installed
  - package eclipse-platform-1:4.11-3.fc31.x86_64 requires glassfish-jsp >= 2.2.5, but none of the providers can be installed
  - conflicting requests
  - package glassfish-jsp-2.3.3-0.14.b02.module_f32+6163+c0e6dcb2.noarch is excluded
(try to add '--skip-broken' to skip uninstallable packages)

Rephrased: "we need both things to happen to make it work"

Yes indeed. Both a default stream and fixing the releng bug.

I feel like we are going round in circles :-)

Please don't. I simply ask that you make a complete proposal that also includes the information about what is being retired/replaced precisely. Is including the list of SRPMs such a big ask?

It's really not, we just discussed this at length already during FESCo meetings (that is why it feels like a circular discussion ;-) ) but I can write it down again here. Yes I am going to retire all eclipse packages provided by this module in F32, but my primary concern here is F31, not F32.

IMO, fixing the current release of Fedora and work for needed for future releases are completely orthogonal matters.

For completeness, here is the list of eclipse srpms provided by the module:

            eclipse:
            eclipse-abrt:
            eclipse-cdt:
            eclipse-dtp:
            eclipse-ecf:
            eclipse-egit:
            eclipse-egit-github:
            eclipse-emf:
            eclipse-epp-logging:
            eclipse-gef:
            eclipse-jgit:
            eclipse-launchbar:
            eclipse-linuxtools:
            eclipse-m2e-apt:
            eclipse-m2e-buildhelper:
            eclipse-m2e-core:
            eclipse-m2e-egit:
            eclipse-m2e-mavenarchiver:
            eclipse-m2e-mavendev:
            eclipse-m2e-tycho:
            eclipse-m2e-workspace:
            eclipse-m2e-wtp:
            eclipse-mpc:
            eclipse-mylyn:
            eclipse-pydev:
            eclipse-remote:
            eclipse-subclipse:
            eclipse-testng:
            eclipse-tm-terminal:
            eclipse-usage:
            eclipse-webtools:

we just discussed this at length already during FESCo meetings

And I have repeatedly stated that meetings are not a good medium to provide this kind of info, whereas tickets are.

IMO, fixing the current release of Fedora and work for needed for future releases are completely orthogonal matters.

Yes, but you have chosen to request both of this in the same ticket.

Before I vote on either f31 or f32 default stream, I'd like to see how it fixes the problem. Currently, it is impossible to see, as the modular eclipse is not installable in either of those. Once that is solved, I'll vote in favor.

Before I vote on either f31 or f32 default stream, I'd like to see how it fixes the problem. Currently, it is impossible to see, as the modular eclipse is not installable in either of those. Once that is solved, I'll vote in favor.

Yes, you can see my chicken-egg situation :-)

I had this conversation with @mizdebsk after the GA, in #fedora-java:

[10:01:47] <mbooth> I'm not sure what can be done about that
[10:03:42] <mizdebsk> is glassfish-el part of eclipse module? does eclipse have default stream?
[10:03:42] <mbooth> mizdebsk: Yes it is part of eclipse module and no, it does not have default stream (FESCo wanted conflicts fixed first)
[10:04:49] <mizdebsk> so adding default stream should fix the problem, i will check that now
[10:07:36] <mizdebsk> mbooth, which stream should be default? 2019-06 or latest?
[10:07:05] <mbooth> mizdebsk: latest
[10:08:08] <mizdebsk> ok, checking...
[10:08:58] <mizdebsk> mbooth, "dnf install eclipse" works after i added default stream

I am just trying to ping him to get his methodology.

Yes, you can see my chicken-egg situation :-)

Indeed, if there is no way to verify this works before we enable it, that is indeed a chicken-egg situation :cry:

Please see this recording for a proof that making eclipse:latest default stream fixes the issue.

Thanks for the recording. I've been able to reproduce the success on Fedora 31, but not on Fedora 32:

[root@170b8ba24211 /]# wget https://pagure.io/releng/fedora-module-defaults/raw/master/f/eclipse.yaml -O /etc/dnf/modules.defaults.d/eclipse.yaml
[root@170b8ba24211 /]# vi /etc/dnf/modules.defaults.d/eclipse.yaml
[root@170b8ba24211 /]# cat /etc/dnf/modules.defaults.d/eclipse.yaml
document: modulemd-defaults
version: 1
data:
    module: eclipse
    modified: 201909251222
    stream: latest
    profiles:
        latest: [java]
        2019-06: [java]
[root@170b8ba24211 /]# dnf install eclipse
Last metadata expiration check: 0:03:23 ago on Tue Nov 19 12:27:11 2019.
Error: 
 Problem: package eclipse-jdt-1:4.11-3.fc31.noarch requires eclipse-platform = 1:4.11-3.fc31, but none of the providers can be installed
  - package eclipse-platform-1:4.11-3.fc31.x86_64 requires glassfish-jsp >= 2.2.5, but none of the providers can be installed
  - conflicting requests
  - package glassfish-jsp-2.3.3-0.14.b02.module_f32+6163+c0e6dcb2.noarch is excluded
(try to add '--skip-broken' to skip uninstallable packages)

Sigh, I suspect that https://pagure.io/releng/issue/9033 is also affecting rawhide :-/

I guess I will try to just rebuild the module and hope bodhi/pungi do the right thing this time...

I am desperately trying to rebuild the module but I am hamstrung by this critical MBS issue:

• https://pagure.io/fm-orchestrator/issue/1525
• https://pagure.io/fedora-infrastructure/issue/8410

This was discussed in today's FESCo meeting. The following proposal was made:
PROPOSAL: Allow setting the default stream for F31 now, and F32 once the rebuild finishes and it is verified to work.

The following votes were made in the meeting:
bookwar +1
contyk +1
jforbes +1
zbyszek -0
otaylor +1

Please add missing votes here.

Doing this on f31 before f32 is just wrong. And f32 should have a change proposal. Yet clearly, this is not what the maintainers want here. I'll abstain.

+1 to @churchyard , which makes me -0 on the voting.

Good news, my remaining issues have been fixed or worked around:

• https://pagure.io/releng/issue/9033 -- Although I still don't know the root cause of this bodhi/pungi issue, a full rebuild of the module seems to have successfully worked around it.
• https://pagure.io/fm-orchestrator/issue/1525 -- The "failsafe" feature of DNF was/is interfering with modularity builds, but there exists a configuration option in DNF to obviate the problem.
• https://pagure.io/fedora-infrastructure/issue/8410 -- Infra team has deployed the solution to the above problem and deployed it to both staging and prod.

Now the module can be successfully installed on F32, please see the following recording:

https://asciinema.org/a/xXQkZ0i5ySl7RerKB4PoTwOnc

Is there anything else that blocks my PRs from being merged?

Good news, my remaining issues have been fixed or worked around:

• https://pagure.io/releng/issue/9033 -- Although I still don't know the root cause of this bodhi/pungi issue, a full rebuild of the module seems to have successfully worked around it.
• https://pagure.io/fm-orchestrator/issue/1525 -- The "failsafe" feature of DNF was/is interfering with modularity builds, but there exists a configuration option in DNF to obviate the problem.
• https://pagure.io/fedora-infrastructure/issue/8410 -- Infra team has deployed the solution to the above problem and deployed it to both staging and prod.

Now the module can be successfully installed on F32, please see the following recording:

https://asciinema.org/a/xXQkZ0i5ySl7RerKB4PoTwOnc

Is there anything else that blocks my PRs from being merged?

I beg you one more time to consider filling a change proposal for Fedora 32. By having all the information only in a FESCo ticket, we are doing a disservice to our community.

That said, you have a right not to do that. +1 to enable default stream of eclipse on F32 and F31.

+1 for allowing default stream in f31 and f32

+1 for filing it as a change in f32 and use it to find and collect there all the work items which were not covered by current task (check what else can conflict with this stream, look into the impact on upgrade and so on)

AGREED: Eclipse can be set as the default stream for F31 and F32 (+6,2,-0)

Heads up: On Fedora 31, dnf just wants to enable this stream on regular upgrade, due to protobuf. protobuf was not mentioned in this ticket at all, but is listed in https://src.fedoraproject.org/modules/eclipse/blob/latest/f/eclipse.yaml

I suspect that the list in https://pagure.io/fesco/issue/2285#comment-612089 was not complete.

Proposal: Drop protobuf and all other non-filtered components from eclipse module ASAP (let's say by the 16.12.2019) or drop eclipse's default stream along with all ant/maven default streams (actually does ant have one? why does default stream depend on non-default?) and implement dnf module reset && dnf distro-sync for these components which used to be part of that module.

Proposal: Drop protobuf and all other non-filtered components from eclipse module ASAP (let's say by the 16.12.2019) or drop eclipse's default stream along with all ant/maven default streams (actually does ant have one? why does default stream depend on non-default?) and implement dnf module reset && dnf distro-sync for these components which used to be part of that module.

Given that the bundled protobuf is breaking other packages, I'm going to preemptively drop the default for now, so the next compose won't have it. Once this is fixed, we can switch it back on.

Thanks @sgallagh.

We also need to figure out what to do with users who already updated their systems.

As this default has been dropped "temporarily", let me just withdraw my +1 for the enablement in the first place. Based on the incomplete information provided, I'm no longer comfortable to approve such change without a proper change proposal. Enabling any default stream clearly breaks things.

@churchyard Well, in fairness, this broke things because it isn't in compliance with our requirements for default streams. I'm assuming that was an oversight on @mbooth 's part. We (FESCo and Modularity WG) agreed that the policy was that any package in a default stream must be maintained exactly as if they were maintaining it for the non-modular repository. The protobuf package is not meeting this requirement. I regret that we didn't notice this earlier.

That's exactly the reason why I want a change proposal. So we can gather all the information there and for better exposure. More people would know about this and possibly could figure that out.

We would also deploy this in rawhide first and only after it proves to work, we would consider a backport.

I agree with @churchyard.

I, too, was hit by this issue as this ticket was getting more posts today. I spent time today figuring out what RPMs I had installed from modular repos, how to remove them, and how to disable all module streams and repos. I suspect a lot of other people are going to ask how they can do it. The steps are needlessly complicated, and I feel this presents a problem for users.

There are a few things I want to discuss tied to this specific issue:
1) How do we help users regain packages with the functionality they expect?
2) What happened that allowed eclipse to bundle a less functional protobuf and override the non-modular package?
3) How do we prevent #2 from happening again?

The 'dnf module' command also needs some work. Igor's idea of a way to reset your system may be worth discussing.

As a user, I was content to let dnf do what it wanted to do with modules, but now I have trust issues with modules. I can't trust a module now not to override a package I am actively relying on with one by the same name that breaks on my system.

2) providing incomplete information to FESCo when enabling the default stream + no rules about modular packages from nondefault streams overriding nonmodular packages
3) no default modular streams

I was unfortunately burned by this today, and it was a nasty surprise to find out that something like this was even allowable for default modules.

Now I have to worry that some module may use one of the packages I am a maintainer of, do weird things to it, and then ship it out to people. This is not okay.

I wonder if we need to treat module enablement the same way we treat soname bumps (i.e. we need to make people announce them, contact all the maintainers of every component that module sources to ensure they don't conflict or break intent of capability on the system, etc.).

I'm leaning towards that modules can't be done in isolation, especially ones that may become default module streams. There's too much potential for breakage.

We also need to figure out what to do with users who already updated their systems.

I've opened https://bugzilla.redhat.com/show_bug.cgi?id=1780827

• AGREED: 1) eclipse module doesn't get a default stream in f31,
  creating a Common Issues entry is recommended 2) adding default
  stream for eclipse module for f32 can be put on discussion via
  Fedora Change process 3) FESCo will identify work items on the
  policy enforcement for default streams content (+8, 0, -0)
  (ignatenkobrain_, 15:32:47)
• LINK:
  https://docs.fedoraproject.org/en-US/modularity/installing-modules/
  doesn't actually answer this question, I think. (zbyszek, 15:33:52)
• ACTION: decathorpe will try to help fixing non-modular eclipse in
  F31 (ignatenkobrain_, 15:44:31)