#1321 Change filepath dependency policy to "MUST NOT"
Merged 4 months ago by tibbs. Opened 5 months ago by zbyszek.
zbyszek/packaging-committee file-permissions-round-two  into  master

@@ -845,8 +845,8 @@ 

  * `+/etc+`

  

  They also MAY depend on paths listed in an explicit `Provides:`.

- They SHOULD NOT include dependencies on other paths

- as that requires additional repository metadata to be downloaded.

+ They MUST NOT include dependencies on other paths,

+ as that would require downloading of additional repository metadata to be enabled.

  

  Please also note that it is not uncommon

  for multiple packages to provide the same directory.

+1

Though I hope this technical restriction in dnf4/5 can be lifted in the future.

-1

I would rather see DNF fixed.

Sure - but given that the upstream developers think that won't happen in the timeframe for Fedora 41 (!), I'd rather not wait for that ...

The problem is that if we do change this, they won't fix it ever, because it will no longer be important for them to fix it.

We have at least a year when the behaviour in dnf will not change. So it makes sense for us to change the guidelines in a way that matches the behaviour that we have, so that our users get the best experience. We shouldn't use our users as a lever to force dnf maintainers to do something faster.

Though I hope this technical restriction in dnf4/5 can be lifted in the future.

Based on the discussion, it seems that it is possible for dnf5, and unlikely for dnf4. But even if this happens, the guideline for Fedora packages should IMO stay the same. The reason is what Jaroslav Mracek wrote in the ticket: when filepath metadata is optional and is only downloaded on demand, and the repo metadata changes fast enough, it is likely that the current metadata on the mirrors is newer than what the user has locally, and if we try to download the filepath metadata, it won't match the other metadata that the user has. So the only solution would be to re-download all metadata and restart the transaction. That is heavyweight and not that great, but even if it is fully implemented, it would be noticeably slower. So even if we could, I don't think we'd want to make use of this feature. It seems that soon there will be 0 Fedora packages requiring filepath metadata, and I don't think we'd want to reintroduce this slowdown for one or two or three packages.

My issue isn't with Fedora repositories, it's that third-party repositories tend to rely on file dependencies to avoid going nuts with figuring out package names. Also, users use file dependencies to request packages when they don't know what package contains it.

The problem is that if we change this, they will absolutely stop working on implementing such a solution for DNF 5, because the focus is inward on Fedora rather than outward for the RPM ecosystem.

I cannot change the motivations of the DNF team and what their management accepts for things like this, but I can try to do what I can to push for the "beyond Fedora" case whenever possible.

users use file dependencies to request packages when they don't know what package contains it.

That case is handled just fine: when you specify a file path on the command line, dnf5 will enable the filepath metadata automatically.

it's that third-party repositories tend to rely on file dependencies to avoid going nuts with figuring out package names

This is why we're going through the Change process to make people aware of the change in behaviour as much as possible. Those external packages will have various choices:

  • change to normal dependencies. This is probably the best option for everyone.
  • ask the providing packages to add virtual Provides for some paths. (A surprising number of packages in Fedora do have such virtual provides. This is very much a realistic solution.)
  • tell the users to enable filepath metadata downloads.
  • enable filepath metadata downloads by themselves, for example by providing a config drop-in for dnf5. This would be relatively easy for third-party repositories which provide a package which contains the repo files.

they will absolutely stop working on implementing such a solution for DNF 5, because the focus is inward on Fedora rather than outward for the RPM ecosystem.

There is a plan to update dnf to print a hint to enable filepath metadata if resolution fails. This is not the ideal solution, but in practice, it will allow users to apply a work-around. If they use some external repo regularly, they can easily create the one-line config file and forget about the issue.

If you think that dnf developers are not putting enough priority in a different solution, please discuss this with them. But right now you're arguing against something that solves will clearly make the experience of Fedora users better, and solve a long-standing RFE.

+1 to the proposed change

Irrespective of whether DNF/DNF5 is modified to allow lazy-loading of file lists, I think it's still the right call for Fedora to disallow use of those other locations in our packages. So even if lazy-loading is added, I don't think we should allow them in Fedora (or at least not without an FPC exception with a really solid justification).

rebased onto 8ee8319

4 months ago

The proposal was accepted by FESCo. Please merge.

Pull-Request has been merged by tibbs

4 months ago

FWIW I think third party developers will wake up if we get to a point that Fedora is clean (never needs to download filelists for deps).

Which brings me to my next point:

Who is going to be in charge of monitoring this that this happens ... not to volunteer anyone, but I feel like the least resistance place to start would be a check in ELN.

I believe bugs were already filed for all relevant packages. My understanding is that in F40 any package which violates this rule will simply fail to install, and so the usual automated "fails to install" report will find any noncompliant packages and they will be handled by normal means.

If dnf5 gains lazy-loading of file lists in the future and that feature is turned on by default then I suppose problematic packages could creep back in, and at that point someone would need to decide whether to look for them. But this could be accomplished by making the "fails to install" test disable the lazy loading.

My understanding is that in F40 any package which violates this rule will simply fail to install, and so the usual automated "fails to install" report will find any noncompliant packages and they will be handled by normal means.

This would require changes to the script that creates the automated "fails to install" reports.

Why? If the package fails to install due to unsatisfied dependencies, won't it show up in the report? Or will dnf5, when encountering a file dependency that it cannot satisfy because it isn't loading the full file lists, do something different than fail to install the package?

The script that creates the automated "fails to install" reports does its own thing to resolve the dependencies. It uses libsolv. Whether or not dnf(5) considers file-dependencies does not automatically change the logic of the script.

Metadata