I'm opening this ticket to trigger discussion of the roadmap for DNF5 in Fedora 39 - whether the switch still looks doable for this release, or whether it should be reverted for F39 and postponed to F40.
The DNF5 Change was approved with the condition that bits that are important to the distribution MUST work, but this does not seem to be the case yet, six months after this was initially approved - there's at least a few things that are still using dnf-3 or have been broken since the switch to dnf5.
(continuously updated list below)
We are now mere days before the Fedora 39 mass rebuild is scheduled to start, so I think it's time to start talking about the roadmap for getting missing pieces into place for Fedora 39, or if that is not possible within this timeframe, whether the contingency mechanism should be enacted.
@jmracek @vponcova
skip_if_unavailable=true
best=true
allow_vendor_change=true
Also posted to the "devel" list for discussion: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/ZRUXJKKSY6RS37ZX3CPA4LRSMMJ7GX3Z/
Please comment there, so we can use this ticket for tracking purposes.
not an exhaustive list, feel free to mention other things, I will update the list to include them
Switch happened in Rawhide but not in ELN: https://github.com/fedora-eln/eln/issues/149
rawhide mock / koji builds still default to dnf-3 (DNF 4)
@kevin replied to this during the FESCo meeting. My understanding is that this infra uses a released Fedora version, which means that this really depends on the previous Fedora release. If we delay dnf5-as-default to F40, then in principle we could keep improving dnf5 in F39 to the point where mock/koji can be switched to use it as default and make that switch on the builders. But I think it's more realistic that we'll first get dnf5 working in a specific Fedora release, release that, and then switch over koji after koji is upgraded to that release.
(not an exhaustive list, feel free to mention other things, I will update the list to include them)
These are issues I hit with our CI automation. After that I stopped trying and switched back to dnf-3 so we can share the same playbooks for rawhide and older releases. This will affect lots of automation scripts.
rawhide mock / koji builds still default to dnf-3 (DNF 4) @kevin replied to this during the FESCo meeting. My understanding is that this infra uses a released Fedora version, which means that this really depends on the previous Fedora release. If we delay dnf5-as-default to F40, then in principle we could keep improving dnf5 in F39 to the point where mock/koji can be switched to use it as default and make that switch on the builders. But I think it's more realistic that we'll first get dnf5 working in a specific Fedora release, release that, and then switch over koji after koji is upgraded to that release.
dnf5 is available in Fedora 38 just not yet the default. They're keeping it up to date, despite multiple breaking API changes, which seems like an Updates Policy violation to me [1]. I understand from @jmracek that 5.1.0 will be released as the first stable dnf5 version in June and then these breaking changes will stop, but that doesn't appear to have happened yet.
Even so, I thought mock doesn't use the host's package manager for most things?
ansible dnf5 module does not work because dnf5 changed public API, it is fixed in rc version
I was also asked by @decathorpe, so let me clarify the ansible situation a bit:
In Fedora 37 and 38, we have ansible-core 2.14. This version does not support Python 3.12 on remote systems and does not have a dnf5 module. We've backported a patch to fix one major instance of Python 3.12 incompatibility, but there may be more. I do not currently plan to backport dnf5 support to 2.14.
In Fedora 39/Rawhide, we have ansible-core 2.15 which has the new tech preview dnf5 module. @pbrezina is right that the current stable 2.15 release upstream is broken with our dnf5 version due to more breaking API changes, but the Fedora package has backported the patch to fix that. We've also had to backport quite a few patches for Python 3.12 support on the controller.
(The details about Python 3.12 are more so relevant to the Fedora CI breakages, but I thought I'd put it here. From what I've seen, many of the Fedora CI issues in Rawhide are from dnf5, but one that @churchyard found is due to missing patches in the ansible-core version from pip that Fedora CI uses)
In any case, I think that python3-dnf should stay installed in Fedora images alongside dnf5, so users can fallback to it until missing features are implemented in dnf and so older ansible versions can still work.
python3-libdnf5 is not present by default (at least on containers) so one has to install it before running ansible
Yeah, it'd be nice to maintain the status quo in this regard.
config-manager not available and plan is to drop backward compatibility for new implementation
This is tracked upstream in https://github.com/rpm-software-management/dnf5/issues/405, FTR. I consider this a major blocker.
[1] https://github.com/rpm-software-management/dnf5/issues/579
As for my opinion on pulling out dnf5 , I'm a bit torn. It can certainly fulfill the basic criteria of installing, upgrading, and removing packages, with some caveats, such as different best and skip_if_unavailable behavior. allow_vendor_change was at least changed back to false (the old default) in the previous release.
best
skip_if_unavailable
allow_vendor_change
false
However, there's quite a few commonly used portions of dnf that are completely not implemented or partially implemented. dnf config-manager is completely missing. dnf copr is there, but the chroot autodetection is not, so the classic dnf copr enable USER/COPR -y doesn't work. system-upgrade and offline-upgrade are completely missing, which is a big problem. How are users supposed to upgrade to Fedora 40? dnf changelog (it's there now) and dnf needs-restarting are also missing. Popular external plugins like the snapper plugin are also missing.
dnf config-manager
dnf copr
dnf copr enable USER/COPR -y
system-upgrade
offline-upgrade
dnf changelog
dnf needs-restarting
To me, the question seems to be whether we want Fedora to serve as a testing bed for dnf5 and potentially disrupt users' workflows and scripts or wait until more of the widely used functionality is implemented. I currently lean towards punting to Fedora 40.
rawhide mock / koji builds still default to dnf-3 (DNF 4) @kevin replied to this during the FESCo meeting. My understanding is that this infra uses a released Fedora version, which means that this really depends on the previous Fedora release. If we delay dnf5-as-default to F40, then in principle we could keep improving dnf5 in F39 to the point where mock/koji can be switched to use it as default and make that switch on the builders. But I think it's more realistic that we'll first get dnf5 working in a specific Fedora release, release that, and then switch over koji after koji is upgraded to that release. dnf5 is available in Fedora 38 just not yet the default. They're keeping it up to date, despite multiple breaking API changes, which seems like an Updates Policy violation to me [1]. I understand from @jmracek that 5.1.0 will be released as the first stable dnf5 version in June and then these breaking changes will stop, but that doesn't appear to have happened yet. Even so, I thought mock doesn't use the host's package manager for most things?
To be clear, the host tools are only used to build the bootstrap chroot. After that the bootstrap is used to build the actual buildroot with native rpm/dnf/etc. So, the f38 dnf5 may work for this, or may not. It definitely didn't a while back, but we haven't tried since.
rawhide mock / koji builds still default to dnf-3 (DNF 4) @kevin replied to this during the FESCo meeting. My understanding is that this infra uses a released Fedora version, which means that this really depends on the previous Fedora release. If we delay dnf5-as-default to F40, then in principle we could keep improving dnf5 in F39 to the point where mock/koji can be switched to use it as default and make that switch on the builders. But I think it's more realistic that we'll first get dnf5 working in a specific Fedora release, release that, and then switch over koji after koji is upgraded to that release. dnf5 is available in Fedora 38 just not yet the default. They're keeping it up to date, despite multiple breaking API changes, which seems like an Updates Policy violation to me [1]. I understand from @jmracek that 5.1.0 will be released as the first stable dnf5 version in June and then these breaking changes will stop, but that doesn't appear to have happened yet. Even so, I thought mock doesn't use the host's package manager for most things? To be clear, the host tools are only used to build the bootstrap chroot. After that the bootstrap is used to build the actual buildroot with native rpm/dnf/etc. So, the f38 dnf5 may work for this, or may not. It definitely didn't a while back, but we haven't tried since.
There are not many differences between DNF5 in Fedora 38 and 39. In Fedora 39 dnf5 provides dnf symlink. I am not sure whether other components of Fedora 38 are involved to build the bootstrap chroot (mock perhaps?).
We will delivery system-upgrade functionality during Fedora 39 lifecycle.
(not an exhaustive list, feel free to mention other things, I will update the list to include them) ansible dnf5 module does not work because dnf5 changed public API, it is fixed in rc version python3-libdnf5 is not present by default (at least on containers) so one has to install it before running ansible config-manager not available and plan is to drop backward compatibility for new implementation These are issues I hit with our CI automation. After that I stopped trying and switched back to dnf-3 so we can share the same playbooks for rawhide and older releases. This will affect lots of automation scripts.
I am really sorry but this is something that cannot by delivered by DNF5 component. The package content in containers is completely on distribution. The question is whether it is good idea to remove DNF5 advantage of absence of Python by default.
Reading the responses here and on the mailing list, I'm not comfortable with moving forward with DNF5 for Fedora 39. I appreciate that the developers are still working on closing the gaps, but the timeline for doing that for Fedora 39 is reeeeally tight, especially considering that it's apparently not even sure whether koji builders etc. would work when upgraded to a dnf5-y Fedora 39.
I'm proposing to initiate the contingency mechanism, in the following way: After Fedora 39 is branched off of rawhide, the switch from DNF 4 to DNF5 by default is reverted, but Rawhide will continue with DNF5, and the switch is re-targeted at Fedora 40.
We will delivery system-upgrade functionality during Fedora 39 lifecycle. (not an exhaustive list, feel free to mention other things, I will update the list to include them) ansible dnf5 module does not work because dnf5 changed public API, it is fixed in rc version python3-libdnf5 is not present by default (at least on containers) so one has to install it before running ansible config-manager not available and plan is to drop backward compatibility for new implementation These are issues I hit with our CI automation. After that I stopped trying and switched back to dnf-3 so we can share the same playbooks for rawhide and older releases. This will affect lots of automation scripts. I am really sorry but this is something that cannot by delivered by DNF5 component. The package content in containers is completely on distribution. The question is whether it is good idea to remove DNF5 advantage of absence of Python by default.
It is outside dnf package, but you can of course influence what package is or is not installed in any version of the distribution. If it should be present in container or not, I cannot say. The bindings certainly won't take lots of space.
Installing the python bindings is at least simple fix. Unavailability of working dnf5 ansible module outside rawhide is a bigger problem as users of other and older distributions will want to provision dnf5 enabled systems. But shipping dnf5 and tell users to use dnf-3 for missing features is a sign that it is not yet ready to be the default tool. In my opinion, you should strive for maximum backwards compatibility, given how core tool the dnf is. Each breaking change will probably affect many scripts.
The change of default configuration is only temporal. DNF5 will support drop-in directories for configuration overrides, but it require empty dnf.conf for proper functionality. The feature allows a different configuration per Fedora variant. The issue and discussion is tracked here - Bug 2216205 - dnf5 seems to default to skip_if_unavailable=False behavior. Basically dnf package cannot deliver the best default for all variants, but fedora-release package can.
Bug 2216205 - dnf5 seems to default to skip_if_unavailable=False behavior
Snapper plugin - The functionality of the plugin was replaced by more generic action plugin. We are documenting it how to enable snapper functionality- see https://github.com/rpm-software-management/dnf5/pull/618/files
Snapper plugin
action plugin
I'm opening this ticket to trigger discussion of the roadmap for DNF5 in Fedora 39 - whether the switch still looks doable for this release, or whether it should be reverted for F39 and postponed to F40. The DNF5 Change was approved with the condition that bits that are important to the distribution MUST work, but this does not seem to be the case yet, six months after this was initially approved - there's at least a few things that are still using dnf-3 or have been broken since the switch to dnf5: rawhide mock / koji builds still default to dnf-3 (DNF 4) Fedora CI has been partially broken since the switch to DNF5 (c.f. fedora-ci/general#416), making CI results for bodhi updates at least partially useless fedora-review has been broken since the switch to DNF5 (c.f. FedoraReview#482), which is really hurting the rate at which new packages are getting reviewed and added to Fedora skip_if_unavailable=true setting removed, regressing from the default behaviour in Fedora < 39 best=true enabled by default even though this was a Rejected Change for Fedora 31 (not an exhaustive list, feel free to mention other things, I will update the list to include them) We are now mere days before the Fedora 39 mass rebuild is scheduled to start, so I think it's time to start talking about the roadmap for getting missing pieces into place for Fedora 39, or if that is not possible within this timeframe, whether the contingency mechanism should be enacted.
The DNF5 Change was approved with the condition that bits that are important to the distribution MUST work, but this does not seem to be the case yet, six months after this was initially approved - there's at least a few things that are still using dnf-3 or have been broken since the switch to dnf5:
I reviewed above issues and most of them are resolved or close to resolved. I would like to thanks community and DNF team for awesome work.
dnf remove -y --noautoremove does not work as well.
dnf remove -y --noautoremove
I have a great news. The functionality is implemented. It only requires to call--setopt=clean_requirements_on_remove=False.
--setopt=clean_requirements_on_remove=False
My recommendation would be to report your requirements on DNF5 upstream (https://github.com/rpm-software-management/dnf5/issues) or Bugzilla. This is a standard way how to get a support from maintainers. It also allow us to track requirements and to prioritize tasks that block community with DNF5 adoption.
Have a nice day
Jaroslav
This will be discussed during today's FESCo meeting.
Metadata Update from @zbyszek: - Issue tagged with: meeting
This was discussed during today's FESCo meeting: - ACTION: decathorpe will update the list in the ticket - AGREED: We'll work on an updated list of acceptance criteria in #3039. We'll revisit the issue of contingency mechanism invocation at the next meeting. (+7, 0, 0)
Thank you for the list. I will provide additional information to each point.
Missing Pieces ELN still defaults to DNF 4 (c.f. fedora-eln/eln#149)
It should be fixed by sgallagher PR (merged to upstream): https://github.com/rpm-software-management/dnf5/pull/691
skip_if_unavailable=true setting removed, regressing from the default behaviour in Fedora < 39 best=true enabled by default even though this was a Rejected Change for Fedora 31
Will be fixed after implementation of drop-in overrides. The configuration will be overridden by distribution (suggesting fedora-release package). It allows Fedora distribution to ship different DNF configuration for each variant. It means end user will not see any difference. See https://github.com/rpm-software-management/dnf5/pull/741 and Bug 2216205 - dnf5 seems to default to skip_if_unavailable=False behavior
fedora-release
allow_vendor_change=true set by default, enabling "Sticky Vendors" (behaviour change compared to DNF 4)
Fixed in upstream - https://github.com/rpm-software-management/dnf5/pull/745
dnf config-manager command missing (rpm-software-management/dnf5#405)
Is it really a blocking feature? Common user does not use the plugin. There is a temporal workaround with dnf-plugins-core (parallel installable with dnf5). I think the plugin could be available in 4 to 5 weeks if it is a blocker.
dnf system-upgrade / offline-upgrade / offline-distrosync commands missing (the recommended way to upgrade Fedora on all editions / spins except Workstation; (rpm-software-management/dnf5#258)
The system-upgrade is used mainly for upgrade between stable Fedoras. The implementation already started. For users that wants to upgrade to rawhide there are several workarounds including direct usage of distrosync or using dnf-pluginns-core (parallel installable with dnf5). We will document workaround if it will be required.
dnf needs-restarting command missing
The plugin is quite simple but it has incorrect logic. We would like to deliver a replacement that will work properly and not a plugin that monitors hard-coded list of packages. If you decide that it is a blocker we will deliver the same logic therefore development time will be short (two weeks after marking as a blocker).
snapper plugin dropped and replaced by action plugin (needs work?)
Snapper plugin is not a core dnf functionality. It is as a part of dnf-plugins-extras and it is not even in RHEL, therefore I do not consider it as a blocker. The plugin is replaced and documented by another plugin - The documentation describes how to deliver snapper plugin functionality with DNF5 https://github.com/rpm-software-management/dnf5/pull/618. The documentation is merged in upstream. The question is whether you consider missing plugin as an issue or missing functionality. I consider this item as resolved with the next release (next week) because functionality is available and documented.
dnf remove -y --noautoremove does not work (--setopt workaround available)
Fixed in upstream - https://github.com/rpm-software-management/dnf5/pull/755 I consider this item as resolved with the next release (next week).
dnf suppresses stdout/stderr from RPM scriptlets (rpm-software-management/dnf5#522)
Information is logged in DNF5 log file, therefore it is not lost. We will fix it. From my point of view it has a medium priority (no blocker) and it should be not so difficult to implement.
high memory usage due to libsolv issue (in progress, devel list message)
There is a tracking bug Bug 2214520 - DNF5 microdnf crashing on machine with 512 MB of RAM. The patch is available (https://github.com/openSUSE/libsolv/commit/296f854b7a9d4b83f6ecb1fc0626511132bf53cf) and there is an easy workaround - dnf5 upgrade --setopt=optional_metadata_types=,comps. The patch could be back-ported but libsolv downstream is not owned by DNF team but by @ignatenkobrain . Igor what do you think?
dnf5 upgrade --setopt=optional_metadata_types=,comps
I've backported it already: https://src.fedoraproject.org/rpms/libsolv/c/4a4c274cc9d7943dadc8d5780b4115fe754e2b13
Thank you.
Thank you for the detailed answer. Some more questions below ;)
skip_if_unavailable=true setting removed, regressing from the default behaviour in Fedora < 39 best=true enabled by default even though this was a Rejected Change for Fedora 31 Will be fixed after implementation of drop-in overrides. The configuration will be overridden by distribution (suggesting fedora-release package).
Will be fixed after implementation of drop-in overrides. The configuration will be overridden by distribution (suggesting fedora-release package).
How will this work for new installations in a chroot? When the installation starts, there'll be no config files yet.
dnf config-manager command missing (rpm-software-management/dnf5#405) Is it really a blocking feature? Common user does not use the plugin. There is a temporal workaround with dnf-plugins-core (parallel installable with dnf5). I think the plugin could be available in 4 to 5 weeks if it is a blocker.
Agreed, I wouldn't consider this a blocking feature.
dnf system-upgrade / offline-upgrade / offline-distrosync commands missing (the recommended way to upgrade Fedora on all editions / spins except Workstation; (rpm-software-management/dnf5#258) The system-upgrade is used mainly for upgrade between stable Fedoras. The implementation already started. For users that wants to upgrade to rawhide there are several workarounds including direct usage of distrosync or using dnf-pluginns-core (parallel installable with dnf5).
The system-upgrade is used mainly for upgrade between stable Fedoras. The implementation already started. For users that wants to upgrade to rawhide there are several workarounds including direct usage of distrosync or using dnf-pluginns-core (parallel installable with dnf5).
This is a big one. I think we need this, but not at release time. At release time we just need this working in the previous release. If we gain the plugin at some later point, before people start upgrading to F40 en masse, that'd be good enough for me. If the plugin is developed later, can we add it also in F39?
dnf needs-restarting command missing The plugin is quite simple but it has incorrect logic.
The plugin is quite simple but it has incorrect logic.
snapper plugin dropped and replaced by action plugin (needs work?) Snapper plugin is not a core dnf functionality.
Snapper plugin is not a core dnf functionality.
dnf suppresses stdout/stderr from RPM scriptlets (rpm-software-management/dnf5#522) Information is logged in DNF5 log file, therefore it is not lost. We will fix it. From my point of view it has a medium priority (no blocker) and it should be not so difficult to implement.
I hope this can get resolved somehow. With the current (lack of) output, it's hard to diagnose failures. I wouldn't consider this a blocker, since some workarounds exist.
Overall, it really seems that it's possible to keep with the original plan. Some corner cases will be a bit rough, but the majority of normal workflows should be OK.
Thank you for the detailed answer. Some more questions below ;) skip_if_unavailable=true setting removed, regressing from the default behaviour in Fedora < 39 best=true enabled by default even though this was a Rejected Change for Fedora 31 Will be fixed after implementation of drop-in overrides. The configuration will be overridden by distribution (suggesting fedora-release package). How will this work for new installations in a chroot? When the installation starts, there'll be no config files yet.
The situation will be quite similar to dnf4. DNF4 takes configuration from installroot if configuration is present or from root (see https://dnf.readthedocs.io/en/latest/command_ref.html --installroot). DNF5 does not have autodetection, because we discovered that it is failing in some cases, but it has an option --use-host-config to use configuration from host system. Anyway this is very good question.
--use-host-config
It is not easy to me, but we propose to not replace DNF with DNF5 in Fedora 39 and postpone the replacement to Fedora 41. It will allow us to provide more features and stability that is required for software management tool. We suggest to trigger the contingency plan after branching of Fedora 39 and 40 to keep replaced DNF by DNF5 in rawhide.
If you want to postpone it to Fedora 41, why keeping it in Rawhide (Fedora 40 after branching)?
There are two reasons. Keep consistency for rawhide users. If we remove the change, rawhide users will loose dnf symlink because dnf5 will stop to provide it but dnf is already removed by DNF5 obsolete. The second reason is related to feedback from rawhide users. It is very difficult to stabilize dnf5 without active usage in rawhide.
Hmm, why 41? It's almost ready for F39; I understand delaying it to F40, but F41 isn't justified.
Also, there seemed to be a lot of energy and dynamic getting dnf5 ready lately. Maybe late, yes - but we don't want to slow that down, do we? What exposure will dnf5 get on rawhide? Too little, as we know too well from the current situation. It's the make it or break it which is driving things currently.
we propose to not replace DNF with DNF5 in Fedora 39 and postpone the replacement to Fedora 41.
So will dnf5 not be part of RHEL 10?
The second reason is related to feedback from rawhide users. It is very difficult to stabilize dnf5 without active usage in rawhide.
This is not what rawhide is for. It's for developing the next Fedora release, not experimenting with things that you plan to revert in the end.
we propose to not replace DNF with DNF5 in Fedora 39 and postpone the replacement to Fedora 41. So will dnf5 not be part of RHEL 10?
Yes, DNF5 will be not in RHEL10, therefore we decided to postpone it to Fedora 41 to not interfere with RHEL 10 branching.
The second reason is related to feedback from rawhide users. It is very difficult to stabilize dnf5 without active usage in rawhide. This is not what rawhide is for. It's for developing the next Fedora release, not experimenting with things that you plan to revert in the end.
It was not intended to experience with rawhide users. We try our best to deliver it according to original plan. as I mention before, complete revert will provide side effect and keeping it like it is provides benefits.
We could look to creating alternative composes with DNF5 on them and have regular test days for it?
That would be helpful especially if there are resources to do that. We definitely intend to hold more test days as well and already have another we're planning for soon (https://pagure.io/fedora-qa/issue/743).
we propose to not replace DNF with DNF5 in Fedora 39 and postpone the replacement to Fedora 41. So will dnf5 not be part of RHEL 10? Yes, DNF5 will be not in RHEL10, therefore we decided to postpone it to Fedora 41 to not interfere with RHEL 10 branching.
So, it's not "because it's not good for Fedora", but "because it interferes with RHEL"? That does not sound well, and even less so these days :(
Please, let's keep this thread on topic.
I officially withdraw the proposal I made in this comment, since it's now obsolete.
Other than that, I am also confused why it's being pushed back by two releases instead of one. I'm not sure if keeping Rawhide diverged from released Fedora for longer is a good idea - alignment with upstream development is a good enough reason for me, while better alignment with RHEL releases should not affect planning of Fedora releases and their features.
Since this is not a simple "postpone by one (two) releases" but requests for the change to remain in effect in Rawhide and only reverted from F39 and F40 after they are branched off, I think FESCo should vote on that request.
It should be possible to %rhel conditionalize the dnf provides and symbolic links in a way that will allow shipping this for Fedora 40 without making the change in RHEL 10 -- if RHEL 10 is the main/only reason to postpone this to F41 instead of F40. In fact, ELN still has dnf4 by default, see https://github.com/fedora-eln/eln/issues/149
%rhel
What @churchyard said. Based on the previous discussion in this ticket, dnf5 is in a state where the listed deficiencies will be fixed, which means we want dnf5 for F40, and we gain no benefit from waiting until F41.
Voted on in meeting:
APPROVED(+7,0,-0) The switch to DNF5 will be postponed (likely to Fedora 41, to be determined by the Change Owners) and reverted in Rawhide for the time being (i.e. dnf 4 will provide /usr/bin/dnf again)
Metadata Update from @tstellar: - Issue close_status updated to: Accepted - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.