FYI: @langdon mentioned at the Council meeting today that there might a a shorter workaround.

@psabata and I have been working this problem with the DNF folks for the last couple days. We are still designing the solution, but we are also in the process of trying to determine which use-cases we think need to be addressed for Beta and which for Final. We will try to have a better report to provide early next week.

@sgallagh can you provide an update on current status, even if things are still being figured out?

@sgallagh can you provide an update on current status, even if things are still being figured out?

After a good bit of investigation we think that it's not going to affect very many people during the Beta cycle. The most likely situation to encounter this issue would be in cases where a user did dnf update --enablerepo=updates-testing-modular and then later did dnf update before the module they had updated had ended up in the stable repo. Since the u-t repos are available by default in Beta, we probably won't have people actually hitting this.

We considered two workarounds:
Ensure that the repodata includes all modulemd for any module that has ever been present in the stream. This would address cases where users who don't update frequently-enough might end up with packages on their system from a release that was post-GA but not the latest module update, and thus DNF could lose track of it if the new module no longer included that package and the non-modular repos did. The downsides here were complexity in maintaining that information and a potential to rapidly grow the repodata size.
Include a special tag in all packages built as part of a module so that DNF could check and refuse to replace any package containing this tag with a package lacking it. This would require specialized handling in DNF, would break the planned "hotfix repo" functionality and would also require a mass-rebuild of all modules post-Freeze, so that was not an option either. (Note that we HAVE implemented that tag for F30 and it was part of the mass-rebuild I did there just after we branched, so this remains in our back pocket for future Fedora releases.)

The full solution and proper solution will be to have DNF keep a copy of the modulemd matching any installed RPMs in a local database (similar to how it retains traditional packages in the RPMDB) so it can always identify which RPMs on the system belong to a module stream and act accordingly.

So, my recommendation would be for us not to block Beta on this issue, but we should consider blocking GA on it.

How about the case I suggested in the bug, where a modular repo is misbehaving for some reason and a user figures it'd be safe to update non-modular content with dnf --disablerepo=(modularrepo) update? Or the case of doing dnf --disablerepo=* --enablerepo=somerepo something, which is something I use myself occasionally after picking it up on IRC or in the wiki or something, so it may be one of those 'cargo cult' things...

How about the case I suggested in the bug, where a modular repo is misbehaving for some reason and a user figures it'd be safe to update non-modular content with dnf --disablerepo=(modularrepo) update? Or the case of doing dnf --disablerepo=* --enablerepo=somerepo something, which is something I use myself occasionally after picking it up on IRC or in the wiki or something, so it may be one of those 'cargo cult' things...

I'm personally comfortable logging that as a Common Bugs entry for Beta. Those are power-user operations from my perspective.

The full solution and proper solution will be to have DNF keep a copy of the modulemd matching any installed RPMs in a local database (similar to how it retains traditional packages in the RPMDB) so it can always identify which RPMs on the system belong to a module stream and act accordingly.

So, my recommendation would be for us not to block Beta on this issue, but we should consider blocking GA on it.

Pfff. This sounds like a huge change and yet another thing which should not be landing in one of the most crucial packages between beta and final.

I agree that the impact of the bug itself does not seem to be too great, so it would be OK to delay the fix until final, iff the fix was clearly in sight and simple. But instead the fix looks rather complicated.

The full solution and proper solution will be to have DNF keep a copy of the modulemd matching any installed RPMs in a local database (similar to how it retains traditional packages in the RPMDB) so it can always identify which RPMs on the system belong to a module stream and act accordingly.
So, my recommendation would be for us not to block Beta on this issue, but we should consider blocking GA on it.

Pfff. This sounds like a huge change and yet another thing which should not be landing in one of the most crucial packages between beta and final.

It's likely to be a fair bit of effort, I agree. I do think it's fairly important to Modularity, but I don't think we're untestable at this point.

I agree that the impact of the bug itself does not seem to be too great, so it would be OK to delay the fix until final, iff the fix was clearly in sight and simple. But instead the fix looks rather complicated.

Yeah, there's definitely some schedule risk here. I won't deny that.

We will discuss this during Monday's meeting at 15:00UTC in #fedora-meeting-1 on
irc.freenode.net.

There seems to be debate going on in the bug about the extent to which dnf can/should do anything here, and what if anything it ought to do:

https://bugzilla.redhat.com/show_bug.cgi?id=1616167#c12
https://bugzilla.redhat.com/show_bug.cgi?id=1616167#c13

we may need FESCo to take another look at this to have clarity on the expectations for Final.

+1

I'm +1 for FESCo getting more involved in finding a reasonable solution to this, but otherwise I'm not sure what we're being asked to do here.

@jsmith: I think we're being asked to define exactly which problems must be solved in order for the blocker to be considered resolved.

I think for me, we need to handle at minimum the following case:

I want to use a modular package that for an older version than is in the standard repo. I install it from updates-testing because the testing version fixes a problem I know I would otherwise hit. I do dnf install --enablerepo=*-testing mypackage. Later, I want to update the rest of my system, but I don't want to update everything from u-t, so I just do dnf update. I do NOT want my package to be updated incorrectly to the non-modular version because DNF no longer recognizes that I have a modular RPM installed and selects the higher NVR of the non-modular version for an update.

@jsmith right, as @sgallagh said. From the QA side our last understanding here was that FESCo and DNF team broadly agreed that something akin to @sgallagh 's proposal above could/should be implemented for F29 Final, and the bug was set to block F29 Final on that basis. The recent discussion in the bug suggests that is not the case, so it's no longer clear to us what would constitute "resolution" of this bug as an F29 Final blocker. Thus we'd like FESCo to look at it again and make another call on what is expected to be achieved in this area before F29 can go out. Thanks.

After considerable discussion in the meeting today, I'm going to propose a more constrained requirement that we will make for blocking F29:

Proposal: At any time, a user must be able to run dnf distro-sync and the result must be that while RPMs (modular and non-modular) may be upgraded or downgraded based on the repositories that are enabled and accessible (taking into account skip-if-unavailable settings), they may NOT change streams. A change of stream is defined as follows: "An RPM installed as part of a particular module stream must not update to a different stream of that module, a stream of a different module or to a non-modular RPM." Additionally, non-modular RPMs are permitted to update into a default module stream.

Edit: Added note about non-modular RPMs being permitted to update to a default module stream.

+1 to @sgallagh's most recent proposal.

+1 to @sgallagh's amended proposal.

This was discussed in today's FESCo meeting.
* AGREED: We table this for a week to give modularity+dnf folks time
to discuss and respond to the proposed requirement. We will revisit
next meeting. (+6, 0, 0) (zbyszek, 15:54:31)
* proposal for requirements for DNF behaviour at GA:
https://pagure.io/fesco/issue/1974#comment-534269 (zbyszek,
15:54:34)
* given time constraints, we ask for the discussion to start now
(zbyszek, 15:54:37)

FYI, I updated the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1616167#c15

Let's discuss more on Monday.

AGREED: file fesco special final blocker on this, try and get more estimate input from dnf team (+7,0,-0) (jforbes, 15:18:50)

(In reply to Daniel Mach from comment #18)

Estimated delivery for this feature is end of January.

From the meeting today:

* #1974 Problematic blocker for F29: dnf 'offline' module tracking
  (contyk, 15:02:46)
  * LINK: https://pagure.io/fesco/issue/1974   (contyk, 15:02:52)
  * AGREED: drop blocker status on bug, document in release notes,
    common bugs and anywhere else we can (+7, 0, -0)  (contyk, 15:21:05)

Who will handle this? Any action on our side?

Done.

So this one has come up again for F30. The DNF feature is still not implemented. See recent comments on https://bugzilla.redhat.com/show_bug.cgi?id=1616167 . Once again, we (the blocker bug review teams) are kicking this to FESCo to decide what action is appropriate here for F30.

Apparently, DNF team is now not sure they can implement this feature for F30 either.

Any response from the DNF team on this?

We agreed with modularity team to delivery the feature in F31.

The postponement announced along today's meeting agenda.

Do we need to keep this ticket open?

I don't see a reason to keep it open. If we get close to F31 and it's not ready, then we can open a new ticket to review it.

From my point of view the original issue should not be marked as a Fedora blocker. Fedora 29 and 30 showed that this feature is nice to have and not a blocker. Playing with "soft blockers" is something that we should avoid.