#338 %doc and %_pkgdocdir duplicate files and cause conflicts

Created 4 years ago by mschwendt
Modified a year ago

This is related to the Unversioned Docdirs feature for Fedora 20.

I'd like to raise awareness of a few packaging pitfalls related to that change.

My package conflicts checker has found a few builds for Rawhide, where the base package duplicates the documentation files, which are included within a noarch -doc subpackage. The reason is that the packages do something as in this fragment to install into an unversioned docdir:
{{{
%install
mkdir -p %{buildroot}%{_docdir}/%{name}
cp -a something %{buildroot}%{_docdir}/%{name}

%files
%doc AUTHORS LICENSE NEWS README

%files doc
%{_docdir}/%{name}
}}}
With the unversioned docdir in Fedora 20, the %doc line will include
the full %{_docdir}/%{name} tree in the base package, regardless of
which local doc files are specified in that line.

Worse, if it's a noarch -doc subpackage, that one may be built
on a different build host, and the files will conflict with what's
included in the arch-specific base package.

What can be done to avoid that? For example, a mathgl packager
has found out that one can %exclude a few files like this, but calls
it a hack:
{{{
%files
%doc AUTHORS LICENSE NEWS README
%exclude %{_docdir}/%{name}/something

%files doc
# be a little explicit about what to include
%dir %{_docdir}/%{name}
%{_docdir}/%{name}/something
}}}
In the linpack package review request, FPC member Ralf Corsepius
also insists on doing it like that, albeit with %_pkgdocdir (which
doesn't make a difference).

What's wrong here?

1) There is no "file listed twice" warning, if the %excludes are
missing. No warning about the duplicated documentation.

2) The build doesn't fail either, if the %excludes are missing.

3) The %doc line from the base package happily includes anything in
%{_docdir}/%{name}/, regardless of the few local files that are specified on the %doc line. It includes even files which are not specified in any %files section. In other words, the build will not even fail, if in %install a new directory or file gets installed into %_docdir/%name.

4) In a base package %files section, such as
{{{
%files
%doc AUTHORS LICENSE NEWS README
%exclude %{_docdir}/%{name}/.pdf
%exclude %{_docdir}/%{name}/html
}}}
ownership and full inclusion of everything in %{_docdir}/%{name} is
implicit and hidden behind the %doc macro magic. There is no line in
that %files section that specifically includes a path that would
lead to including %{_docdir}/%{name}/
.pdf and %{_docdir}/%{name}/html,
so the two %excludes look misplaced. Typically, one uses %exclude
in a %files section, only if the same %files section includes the
files, but if one wants the files to be included in a subpackage instead.

What to do with this ticket? I don't know. Just this: If it were my decision, I would ban %doc install of local files relative to %_builddir, if any subpackage includes files in %_pkgdocdir (or versioned docdir for older dist releases). That would force packagers to be explicit about what documentation files to include in which %files section.

Typo in the second to last sentence. It must read: ... files in %_pkgdocdir'''/%name''' (or versioned docdir...

In the past, was there an rpm-build limitation related to using %doc and also manually installing files into the same directory? I thought doing something like that caused some issue like -- only the files included via %doc ended up in packages but I could be misremembering.

https://fedoraproject.org/wiki/Packaging_tricks#Installing_documentation:_2_paths

Documentation directory is "%{_defaultdocdir}/%{name}-%{version}/". However, there are two ways to install in this directory, either using %doc or directly into the staged directory "$RPM_BUILD_ROOT%{_defaultdocdir}/%{name}-%{version}/". Both ways cannot be mixed since %doc erases everything installed in the staged doc directory after the %install stage.

So it looks like something in the recent changes made it so that %doc is not cleaning up anymore.

Since it was former best practice I would be for mschwendt's suggestion to tell packagers that either %doc or manual installation of documentation is fine but trying to do both is disallowed.

(comment collision)

Yes, there have been conflicts as well, just for the versioned default docdir (=> %_docdir/%name-%version) instead. For some time, rpmbuild even deleted and recreated that dir from scratch for the %doc magic, killing everything that has been %installed into it, causing damaged almost-empty -doc subpackages.

[...]

What would it take to fix rpmbuild? So that
{{{
%doc AUTHORS COPYING README
}}}
would result in implicit %files entries
{{{
%dir %{_pkgdocdir}
%{_pkgdocdir}/AUTHORS
%{_pkgdocdir}/COPYING
%{_pkgdocdir}/README
}}}
instead of currently:
{{{
%{_pkgdocdir}/
}}}

Removing the target doc dir before installing files in it has been removed from special %doc in rpm >= 4.9.1, http://rpm.org/ticket/836

Two remarks:

  • %doc with absolute paths seems harmless (but maybe redundant/unnecessary for absolute paths under %_docdir?)
  • There are legitimate cases for duplicate files.

Consider this scenario:

Main package ''foo'' installs COPYING to %_pkgdocdir and lists it:
{{{
%dir %{_pkgdocdir}
%doc %{_pkgdocdir}/COPYING
}}}

The (noarch) ''-doc'' subpackage copies some stuff (examples, READMEs, etc) also there, and lists it:
{{{
%dir %{_pkgdocdir}
%doc %{_pkgdocdir}/*
}}}

As ''-doc'' does not depend on the main package (it can be installed on its own), it needs to include COPYING, but it seems silly to have COPYING in /usr/share/foo-doc/COPYING.

Interestingly, there's also no warning about COPYING being listed twice in this case.

Imho, the cleanest solution would be to completely remove the magic of %doc-with relative paths, and force spec files to (a) explicitly install and (b) declare documentation files, like any other file.

Yes, files below some paths are marked as documentation automatically. No explicit %doc is needed for them:
{{{
$ rpm -E %__docdir_path
/usr/share/doc:/usr/share/man:/usr/share/info:/usr/share/gtk-doc/html::/usr/share/man:/usr/share/info:/usr/share/javadoc:/usr/doc:/usr/man:/usr/info:/usr/X11R6/man
}}}

The "file listed twice" warning is per subpackage only, which makes it irrelevant for this topic. The warning doesn't help with files, which are duplicated in multiple subpackages.

The scenario you mention is ordinary placement of files in (sub-)packages. It is obvious that if you do
{{{
%{_somepath}/COPYING
}}}
in one package and
{{{
%{_somepath}/*
}}}
in a subpackage, the latter includes also the COPYING file. That is expected, not specific to doc files and not related to what this ticket points out.

And an update: I've filed approx. 50 tickets for packages, where the -doc package contents are included also in the base package due to this issue.

Replying to [comment:4 mschwendt]:

What would it take to fix rpmbuild? So that
{{{
%doc AUTHORS COPYING README
}}}
would result in implicit %files entries
{{{
%dir %{_pkgdocdir}
%{_pkgdocdir}/AUTHORS
%{_pkgdocdir}/COPYING
%{_pkgdocdir}/README
}}}
instead of currently:
{{{
%{_pkgdocdir}/
}}}

rpmbuild would need to implement the exact semantics 'cp -pr ... <destdir>' internally for %doc or have %doc install to temporary directory for discovery and then move them to final location for packaging. Nothing impossible but annoying enough that nobody has bothered so far.

Oh and for the future: please do CC me if/when you (meaning FPC in general) have questions on technical side of rpm issues. I do try and keep an eye on FPC meeting agendas and logs but...

As a followup to the meeting log (I wish there would be more activity on a mailing-list, since not even this ticket seems to have been read) and comment 6:

  • '''Before''' RPM 4.9.1, packages, which tried to be smart by sharing a single %{_docdir}/%{name}-%{version} for all doc files in any subpackages, would end up with the %doc magic deleting contents of %{_docdir}/%{name}-%{version} prior to copying the specified files there.

  • Since RPM 4.9.1, %doc in the base package includes anything found in %{_docdir}/%{name}-%{version}.

  • %doc in subpackages (and really only to include '''local''' files) is less of a risk, because it inserts the subpackage name into the docdir, resulting in a unique dir per subpackage. So only someone who would install manually into that specially named docdir and reuse the dir in other (sub)packages would be affected.

It is not known how many subpackages install into a shared %{_docdir}/%{name}-%{version} dir, but especially those subpackages, which installed into shared unversioned %{_docdir}/%{name} already before Fedora 20, are affected by this problem.

And no, it's not thousands of packages and not only a handful either. As mentioned in comment 8, I've opened tickets for approx. 50.

The %_pkgdocdir macro doesn't change much. It only makes it easier to keep versioned docdirs in Fedora < 20.

In today's meeting, we made a proposal: Add "Packages must not use both relative %doc in the %files section and manual installation of documentation files into %{_docdir} in a single spec file" somewhere in the guidelines (I'll probably find where we added the unversioned docdir change and put it there)

This did not pass (+1:4, -1:1). After some discussion (and reading mschwendt's latest comment) I'll look into restricting the places where %doc is not allowed even more but we'll also probably need to have more FPC members vote on the new draft in order to pass it (racor expressed reservations even with the changes to the proposal).

Side note: scop, I looked for Guidelines on using %{_pkgdocdir } and couldn't find any -- have I confused an FPC guideline for using that with the FESCo Change Ticket? If so, would it be beneficial to add usage of pkgdocdir to the Guidelines?

Replying to [comment:10 toshio]:

Side note: scop, I looked for Guidelines on using %{_pkgdocdir } and couldn't find any -- have I confused an FPC guideline for using that with the FESCo Change Ticket?

Dunno, but I'm not aware of any FPC guidelines related to the docdir change.

If so, would it be beneficial to add usage of pkgdocdir to the Guidelines?

IMO yes. Also, reviewing all Wiki content which contains _defaultdocdir and/or _docdir would be a good idea.

Any final resolution/suggestions for how to handle these situations?

Hmm... I've dropped the ball on rewriting the draft in a way that might be more acceptable to get another vote. I'm deep in SCL-draft writing at the moment. If someone would like to pick up where I left off, the former proposal was to add "Packages must not use both relative %doc in the %files section and manual installation of documentation files into %{_docdir} in a single spec file" where we added the unversioned docdir change.

This covered more situations than is necessary to fix the issue. So to move forward, the guideline could be more limited (as mschwendt says: "only someone who would install manually into that specially named docdir and reuse the dir in other (sub)packages would be affected." So we could limit it to only spec files that manually install documentation into %_docdir cannot use %doc in the main package).

If I remember correctly, I had opened this ticket as a "Clarification" request not as "Guideline Draft". Hence the "I'd like to raise awareness" at the beginning. I do not know what the FPC thinks about the issue.

I've been reading the meeting logs, and feedback from the FPC has been sparse and also negative from a few members. The comments from those members are demotivating. I have not submitted a draft, because nobody has asked me to do that and because there's no documentation of the process and what a draft shall look like, so it would not just receive half-hearted votes from the few FPC members who are present in meetings. With some FPC members only making silly comments in the meeting about this ticket, the opportunity to work on this together has not been taken.

Meanwhile it's still not known what the FPC (as a whole, and not just Toshio) thinks about this issue and whether it shall be covered by the guidelines somewhere/somehow.

The key issue here simply is: Packagers, who are aware of the problem and understand it in detail, will likely remember avoiding it like a plague. For other packagers, who are not aware of it [yet], it's a pitfall.

I was bitten by this a few times recently...

The solution described by mschwendt in #comment:4 seems like the right thing to me, with the additional clarification that the magic which makes "%doc in the base package include anything found in %{_docdir}/%{name}-%{version}" should be removed. This magic leads to the need to add %exclude to the main package's %files, if there's a separate docs package. This is ugly, unintuitive, and annoying.

Packages must not use both relative %doc in the %files section and manual installation
of documentation files into %{_docdir} in a single spec file
Why? This is a nice and useful shortcut (see example below). Can't we fix rpm and %doc macros to do the right thing, instead of forcing packagers to work around the issue?

Example for an autotoolized package under review currently (libsbml):

{{{
%install
make install DESTDIR=...

%files
%doc README

%files doc

%{_pkgdocdir}/c-api
%{_pkgdocdir}/python-api
}}}

With the rule from #comment:4 whis would do the right thing.

Replying to [comment:17 zbyszek]:

Packages must not use both relative %doc in the %files section and manual installation
of documentation files into %{_docdir} in a single spec file
Why?
Because mixing %doc and copying files to %_pkgdocdir did not harmonize well throughout different releases of rpm.

Example for an autotoolized package under review currently (libsbml):

{{{
%install
make install DESTDIR=...

%files
%doc README

%files doc

%{_pkgdocdir}/c-api
%{_pkgdocdir}/python-api
}}}

With the rule from #comment:4 whis would do the right thing.
Almost.

Clean, modern autotool based packages (IIRC, autoconf does so for >> 10 years), acknowledge "--docdir=...", which defaults to a directory matching the new _pkgdocdir and installs docs to $docdir similar as they install other files to other dirs.

Therefore to add files to a package's doc which upstream doesn't install (such as "README") the "correct" (safe) approach would be to manually copy them (Like you would do with other files and other directories). The fact that %doc README + %{_pkgdocdir} might work is an "accident".

We discussed this at today's meeting (​http://meetbot.fedoraproject.org/fedora-meeting-1/2015-01-22/fpc.2015-01-22-17.01.txt), but nothing new was decided:

We talked about this at today's meeting (​​​​​http://meetbot.fedoraproject.org/fedora-meeting-2/2015-02-12/fpc.2015-02-12-17.05.txt):

  • 338 %doc and %_pkgdocdir duplicate files and cause conflicts


    (geppetto, 17:32:45)
  • LINK: https://fedorahosted.org/fpc/ticket/338 (geppetto, 17:32:49)
  • ACTION: tibbs Restate the proposal as a policy change, so we can
    vote on it (geppetto, 17:38:36)

Draft at: https://fedoraproject.org/wiki/User:Tibbs/DocdirDraft#Documentation

Changes from current guidelines are: https://fedoraproject.org/w/index.php?title=User%3ATibbs%2FDocdirDraft&diff=cur&oldid=404105

(I think that will always give a proper diff even if more changes are made.)

As always, I did some minor cleanups to the section I was editing to fix grammar and reference _pkgdocdir everywhere. The change relevant to this ticket is the addition of the second paragraph. I can provide it in isolation if people don't like my cleanups.

Also, I hope it's not necessary to define "relative".

We discussed this at today's meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2015-02-26/fpc.2015-02-26-17.01.txt):

  • 338 %doc and %_pkgdocdir duplicate files and cause conflicts


    (geppetto, 17:39:21)
  • LINK: https://fedorahosted.org/fpc/ticket/338 (geppetto, 17:39:26)
  • ACTION: %doc and %_pkgdocdir duplicate files and cause conflicts
    (+1:6, 0:0, -1:1) (geppetto, 17:58:15)

Announcement text:

The documentation section of the guidelines has been updated to include a prohibition on using both %doc and direct installation of files into %_pkgdocdir.
* http://fedoraproject.org/wiki/Packaging:Guidelines#Documentation
* https://fedoraproject.org/w/index.php?title=Packaging%3AGuidelines&diff=405928&oldid=405492
* https://fedorahosted.org/fpc/ticket/338

a year ago

Metadata Update from @tibbs:
- Issue assigned to tibbs

Login to comment on this ticket.

cancel