#549 Drop "File Dependencies" section
Closed: Fixed None Opened 8 years ago by vondruch.

As it turns out, DNF always download filelists. Therefore, please drop the "File Dependencies" section [1] from guidelines.

The original discussion why I'd like to use file dependencies can be found and packaging ML [2].

[1] https://fedoraproject.org/wiki/Packaging:Guidelines#File_Dependencies
[2] https://lists.fedoraproject.org/pipermail/packaging/2015-July/010794.html


I could imagine that it could be a problem with resolving to many file dependencies once we have a lot of them. Are there any statistics of that? But we could of course also just add that section again, once it really becomes an issue...

I see the problem on packaging level. Once everyone starts to use them, packages would require other packages directly less and less. This will lead to many broken dependencies because packages would expect a file which is no longer provided. IMO every package has the right to hide it's details. Binary files are fine as they are considered as API. I would even drop the /etc exception from the guideline. I am against - the packaging would be the mess.

From DNF developer point of view: The dependency resolution will be slower.

Replying to [comment:3 jsilhan]:

I see the problem on packaging level. Once everyone starts to use them, packages would require other packages directly less and less. This will lead to many broken dependencies because packages would expect a file which is no longer provided.

If my package expect some file and it is no longer provided, then the broken dependency is exactly what I want.

Binary files are fine as they are considered as API.

And the original case was jQuery, which is JavaScript. In your terms, I would consider it "binary" or "API".

Do we know that DNF will always continue this unfortunate behavior of always downloading the file lists? Simply not doing it unless necessary seems an obvious optimization.

In any case, I'm against the proposal of eliminating the guideline discouraging dependencies. I have not really considered whether it is reasonable to expand the set of acceptable cases, but that's not the proposal under consideration anyway.

Replying to [comment:4 vondruch]:

Replying to [comment:3 jsilhan]:

I see the problem on packaging level. Once everyone starts to use them, packages would require other packages directly less and less. This will lead to many broken dependencies because packages would expect a file which is no longer provided.

If my package expect some file and it is no longer provided, then the broken dependency is exactly what I want.

Exactly. File deps and package deps are not interchangable, they have different meanings. What is more appropriate deps on the individual use case.

My feeling tens to be that in most cases, file deps are a sign of leaky abstraction. The package concept is a distro level abstraction (when we're talking about inter-dependencies). If something truly does require a specific file in a specific location, that suggests a bad design - better is to (a) work with upstream developers (on either or both sides of the package split) to standardize on the file location and/or (b) have such things as search paths walked in the requiring application/library. Really, when there's a file level dependency I think we should be asking questions about design of the software being packaged, not solving software design flaws at the rpm packaging level. I've yet to see a file level dependency that wasn't indicative of fragile software design.

So lets say my package currently require jQuery 2.x. This file is provided by js-jquery package:

{{{

dnf repoquery -l js-jquery

Last metadata expiration check performed 0:00:12 ago on Tue Jul 7 09:00:47 2015.
/usr/share/javascript/jquery/2
/usr/share/javascript/jquery/2.1
/usr/share/javascript/jquery/2.1.3/jquery.js
/usr/share/javascript/jquery/2.1.3/jquery.min.js
/usr/share/javascript/jquery/2.1.3/jquery.min.js.map
/usr/share/javascript/jquery/latest
}}}

and my dependency is expressed as:

Requires: %{_jsdir}/jquery/2/jquery.js

Now if jQuery 3.x gets released, the js-jquery will be upgraded to jQuery 3.x and new js-jquery2 will will be introduced. In this case, my package will continue to work exactly as I need. While if I depended directly on js-jquery package, my package would get silently broken, because the symlinks I use will not be satisfied.

I can't see how dependency on js-jquery would be better then file dependency.

Replying to [comment:8 vondruch]:

Now if jQuery 3.x gets released, the js-jquery will be upgraded to jQuery 3.x and new js-jquery2 will will be introduced.

I don't get why will be introduced new package name instead of bumping the package version. I see only one reason and that's a big incompatibility between js-jquery and js-jquery2 and in that case you should really require just package name.

If there's really a good reason of doing a new package name and requires either one or the other, then the virtual provide should be made or wait for rpm rich dependencies supporting Boolean expressions.

Replying to [comment:5 tibbs]:

Do we know that DNF will always continue this unfortunate behavior of always downloading the file lists? Simply not doing it unless necessary seems an obvious optimization.

Downloading the file lists on demand is not impossible [1] but would require a huge design change in DNF and hawkey library. We don't want to invest our time into this feature now.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=968006

Replying to [comment:9 jsilhan]:

Replying to [comment:8 vondruch]:

Now if jQuery 3.x gets released, the js-jquery will be upgraded to jQuery 3.x and new js-jquery2 will will be introduced.

I don't get why will be introduced new package name instead of bumping the package version. I see only one reason and that's a big incompatibility between js-jquery and js-jquery2 and in that case you should really require just package name.

Apparently for the same reasons we have js-jquery1 and js-jquery in Fedora already. And for that same reason, rubygem-jquery-rails ships both versions of jQuery.

Not to mention, if we could use package versions for that purpose, we could stop discussing new packages foo1, foo2 etc for ever.

If there's really a good reason of doing a new package name and requires either one or the other, then the virtual provide should be made

Using virtual provides everywhere where file dependencies can be used solves nothing IMO. It will be extrawork and the dependency resolution will be slower as well, since you will need to consider more dependencies again.

or wait for rpm rich dependencies supporting Boolean expressions.

Can't see how this could help.

Replying to [comment:8 vondruch]:

While if I depended directly on js-jquery package, my package would get silently broken, because the symlinks I use will not be satisfied.

You can depend on js-query < 3, can't you?

(Personally, I don't understand why these "<" dependencies are not in every Fedora package. Maybe it's assumed that with every new Fedora, every package maintainer makes sure that their package is still compatible with the current packages? Either way, it's not specific to JavaScript. E.g. even every package that requires "dnf" today has no guarantee that it will work with dnf-2.0.)

Replying to [comment:11 rholy]:

Replying to [comment:8 vondruch]:

While if I depended directly on js-jquery package, my package would get silently broken, because the symlinks I use will not be satisfied.

You can depend on js-query < 3, can't you?

And when js-jquery2 is released, I'll go and change the dependency to js-jquery2? Or may be I'll be away and it will wait forever until somebody fixes it?

Or is it possible that js-jquery2 will keep providing js-jquery = 2.x? Dunno ...

Anyway, so far you proposed just workaround while the real solution works and is prohibited just by some policy. Otherwise you did not provide any reason why the policy should not be changed from DNF point of view (except "resolution will be slower", which is not supported by any numbers).

You are allowed to use file based requires, using the current guidlines. I personally would argue that it's poor packaging in your jquery example, but you can do it anyway.

As Jason said even if DNF doesn't implement the optimisation now ... it's still a possible optimisation for the future (DNF hasn't been in Fedora for 6 months yet!), and is only possible if people follow the best practice guidelines of not depending on files.

The DNF maintainers also seem to be against your proposal, so I'm going to just close the ticket.

My two last comments:

  1. The guideline is factually wrong: "Using file dependencies outside of those directories requires yum (and other depsolvers using the repomd format) to download and parse a large xml file looking for the dependency." where especially the "download" is not true for DNF, not mentioning that YUM is not default anymore, so it should not be mentioned in guidelines IMO.

  2. The metadata are downloaded by DNF probably from everlasting. The ticket [1] requesting to narrow the situation was rejected more then two years ago, so expecting that the situation will change and the "optimization" will be applied any time soon is rather optimistic.

Thanks.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=968006#c9

I agree with #1 and have just gone ahead and fixed it. Also, the directory example where was completely wrong anyway so I removed it. I did leave in mention of not downloading file lists, because yum still works that way and dnf might eventually find reason and stop screwing up important yum behavior.

Really I think the whole issue should be handled by simply making sure the jquery packaging is sane and updates are handled in a thoughtful manner (which is no different than any other kind of dependency we have). Along with better tooling to catch the case where the package really does have dangling symlinks which will go nowhere after installation.

Replying to [comment:15 vondruch]:

My two last comments:

  1. The guideline is factually wrong: "Using file dependencies outside of those directories requires yum (and other depsolvers using the repomd format) to download and parse a large xml file looking for the dependency." where especially the "download" is not true for DNF, not mentioning that YUM is not default anymore, so it should not be mentioned in guidelines IMO.

The significant word here is "requires", adding file deps. does change things in that extra information is required to be downloaded (it isn't possible to not do so). The fact that DNF currently downloads the extra data, even when it isn't used/required/needed/whatever, doesn't change this fact.
Jason's change keeps that wording, and adds some more to hopefully make things clearer, so doesn't seem bad to me (is that what you wanted vondruch)?

  1. The metadata are downloaded by DNF probably from everlasting. The ticket [1] requesting to narrow the situation was rejected more then two years ago, so expecting that the situation will change and the "optimization" will be applied any time soon is rather optimistic.

Well the things you need to concentrate on before you have a first release aren't the same as those you can work on when you've done a first release, so yeh I'll be optimistic for a couple more releases at least.

Thank you. That section is much better now.

Login to comment on this ticket.

Metadata