#411 proposal: migrate license files to %license instead of %doc
Closed: Fixed None Opened 5 years ago by mattdm.

Background:

Right now, license files are required to be marked as %doc files.

There has long been a "nodocs" parameter to RPM which skips all doc files.

In addition to the desired space-savings, this installs packages without their possibly-mandatory license files

This interaction hasn't been problematic before, because generally using nodocs is an endpoint choice with no distribution after that. But now, we are looking at building some official cloud and container images with nodocs, so it suddenly becomes important.

As a bonus, it's my understanding that this tag can automatically handle hardlinking identical license files.

Specifically, I propose:

  1. We change the guidelines
  2. We start doing it for new packages
  3. We file a F21 system-wide change for a proven packager to change all the packages that land in the cloud image for F21 (roughly, @core + dependencies plus a few extras)
  4. We file a system-wide change for F22 to update all other packages which are part of the base design
  5. Other packages updated on a as-time-permits/best-effort basis

I filed the original rpm feature ticket on this:
http://www.rpm.org/ticket/116

I don't believe that %license does automatic hardlinking, although, I asked for that.

That said, we should definitely start using %license in new packages and I'm happy to help with the proven packager changes for cloud in rawhide (a source package list would be nice).

Cool. I'll get you a preliminary list of src packages RSN. List may change by F21 release but I don't think significantly.

I'm also +1 to a guideline to use %license for license files.

Probably should file a FESCo change request for the updating of packages to include that tag -- the reason being that some endusers might be using --nodocs already to save space and %license would mean that they are likely to want to add an rm -rf /usr/share/doc to their scripts to build their images.

Replying to [comment:3 toshio]:

Probably should file a FESCo change request for the updating of packages to include that tag

Yes, I will file the change -- see point 3 above :)

I wanted to put the general plan out here first just to make sure it generally seemed good from this perspective. Spot, do you want to be a change co-owner or should I just note that you'll help?

List of changes that are needed to implement this:

https://fedoraproject.org/wiki/Packaging:Guidelines#Documentation

Add a paragraph:

"""Although license files are documentation, they are treated specially (including using a different tag. Please see [[Packaging:LicensingGuidelines]] for how to handle them"""

Change every instance of %doc on this page to %license:

https://fedoraproject.org/wiki/Packaging:LicensingGuidelines

Change item #4 in the ReviewChecklist to %license:
https://fedoraproject.org/wiki/Packaging:ReviewGuidelines

%doc to %license in spec templates:
http://fedoraproject.org/wiki/Packaging:Haskell
http://fedoraproject.org/wiki/Packaging:D
https://fedoraproject.org/wiki/Packaging:Python
(Probably other spec templates as well... This was just a quick google search)

Google didn't find any place under wiki/Legal: that mentioned the %doc tag but someone might want to review those pages (When we set it up, we tried to keep "how to package" in Packaging: and "what is legal" under Legal: so it makes sense that there wouldn't be anything to change there but it doesn't hurt to double check).

Finally a question that should come up in fesco -- is it '''critical''' that everything on the cloud product has their license tags changed from %doc to %license (because the use of --nodocs means that some packages could be shipped without their accompanying license if they are not marked) or is "best effort" and fixing any packages we've missed if the missing text is reported going to be okay?

Yes, this is definitely something we need to do, and I'll toss in my +1 now.

However, which RPM versions actually support this? We should be careful adding "MUST" guidelines in a way that breaks the "one spec across Fedora and EPEL" thing that some people insist on.

Hey, Spot, I took a look at https://git.fedorahosted.org/cgit/spin-kickstarts.git/tree/fedora-cloud-base.ks to figure out what packages are installed in the Fedora Base Cloud image. I don’t know if this is the complete list, maybe someone can add more, but this is what I found so far. This is for the F20 image, currently.

I separated the list into packages available in just Fedora or packages available in Fedora and EPEL, in case there are any problems with what tibbs was saying of not a recent enough RPM version for RHEL.

Cloud images packages

Core

Fedora only

http://pkgs.fedoraproject.org/cgit/grubby.git/

http://pkgs.fedoraproject.org/cgit/tar.git/

http://pkgs.fedoraproject.org/cgit/rsync.git/

EPEL

http://pkgs.fedoraproject.org/cgit/cloud-init.git/

http://pkgs.fedoraproject.org/cgit/heat-cfntools.git/

Subpackages:

https://apps.fedoraproject.org/packages/cloud-utils-growpart

https://apps.fedoraproject.org/packages/syslinux-extlinux

To clarify some aspects from rpm POV:

The "special" %license that behaves similarly to special %doc is supported in rpm >= 4.11. In other words, its fairly recent but all current Fedora versions support it. Current EPEL does not support it, but 7 will.

Files packaged as %license do not go into the same directory as %doc, but a similar per-package directory in /usr/share/licenses. The reason for that is mostly technical, %doc is a hack that does not like other hacks around so this was painful enough as it is :)

No automatic hardlinking is done currently. Considering how small license files generally are, the space savings would be neglible... except perhaps for (cloud etc) images where every byte might count, but image builders can just do 'hardlink /usr/share/licenses'.

Wrt EPEL compatibility concerns in the FPC meeting - specs can be kept compatible by "mapping" license to %doc on rhel < 7. Something like:
{{{
%files
%if 0%{?rhel} && 0%{?rhel} < 7
%global license %%doc
%endif

%defattr(-,root,root,-)
%license COPYING
}}}

Just note this "mapping" cannot go to spec preamble because License-tags in the spec will override it with the tag content. It doesn't have to be in %files, but it needs to be after the last License: tag.

I was hoping for a single line "if old, %doc, if else, %license", but I can't see how to do that. Guess this is the best option.

Ask and thou shalt receive :P

Just realized that inside a spec it's possible to test support for the special %license directive by testing whether %_licensedir is defined. So this should do the right thing everywhere, regardless of the distro and all:

{{{
%files
%{!?_licensedir:%global license %%doc}
%license COPYING
}}}

My hat is off to you sir. This is exactly the sort of thing I wanted.

What if the license text does not have a dedicated file, but it's a part of another file. E.g. ''README''. I guess that file should be marked as ''%license'' and thus it would disappear from ''/usr/share/doc''. Is this what users expect?

Since I'm not around for meetings for a few weeks, I am +1 to this change.

We'll want to be sure to say that %{!?_licensedir:%global license %%doc} needs to go after the preamble in both the licensing guidelines and review checklist

-1

I consider this proposal to be absurd.

Why?
- This all (cloud images) addresses a small group of user who are using a crippled distribution.
Their demands are such kind of specialized to justify the churn these people are causing.

  • The sizes of %doc'ed files are almost negligent in comparison to the overall size of the rest of the distro. The effort to implement this change is not justified.

  • %license is redundant to Licence: => I do not see any need to add %license but am expecting it to cause confusions, bugs and conflicts.

  • The effort to implement this HUGE. If I was hostile, I'd insist on letting the proponents of this proposal to implement act of bureaucracy themselves.

Replying to [comment:15 corsepiu]:

  • This all (cloud images) addresses a small group of user who are using a crippled distribution.
    Their demands are such kind of specialized to justify the churn these people are causing.

"Demands" sounds so harsh, and I don't know what you mean by "crippled". But in any case, this is a strategically important area for Fedora.

  • The sizes of %doc'ed files are almost negligent in comparison to the overall size of the rest of the distro. The effort to implement this change is not justified.

The first part of this isn't true; it's a full 5% of the base cloud image (on disk), and a significantly higher fraction for some Docker containers. It's true that it'd be nice to get more significant reduction than that, but there are few other single changes that will have a bigger impact (and we're working on those, too). Together, they add up.

The second part is irrelevant, since Tom has volunteered to help with the effort, and I'm sure others are willing to pitch in as needed, too.

  • %license is redundant to Licence: => I do not see any need to add %license but am expecting it to cause confusions, bugs and conflicts.

A different name would be fine. I'm not picky. This seems fine to me -- and it is how the RPM maintainer decided to do it so I don't see much need to bikeshed.

  • The effort to implement this HUGE. If I was hostile, I'd insist on letting the proponents of this proposal to implement act of bureaucracy themselves.

See above.

info Migrating licenses to use %license passed (+1:5, 0:1, -1:1)

Still waiting for a working rawhide cloud image in order to give the list of cloud packages to Spot. I'll update as soon as available.

Update: we have a working cloud image now, but due to dependency bloat in cloud-init (of all things), there are about twice as many packages as there should be. So, I'm hoping to get that fixed first before providing the full list.

Here's the list of source RPMs which land on the F21 cloud base image. I would like to get this number down, but we get into dependency tangles, so here we are.

{{{
acl-2.2.52-5.fc21.src.rpm
attr-2.4.47-7.fc21.src.rpm
audit-2.3.7-2.fc21.src.rpm
basesystem-10.0-10.fc21.src.rpm
bash-4.3.18-2.fc21.src.rpm
bind-9.9.5-6.P1.fc21.src.rpm
bzip2-1.0.6-12.fc21.src.rpm
ca-certificates-2013.1.97-3.fc21.src.rpm
checkpolicy-2.3-2.fc21.src.rpm
chkconfig-1.3.61-2.fc21.src.rpm
cloud-init-0.7.5-6.fc21.src.rpm
cloud-utils-0.27-11.fc21.src.rpm
coreutils-8.22-16.fc21.src.rpm
cpio-2.11-28.fc21.src.rpm
cracklib-2.9.1-3.fc21.src.rpm
cronie-1.4.11-7.fc21.src.rpm
crontabs-1.11-8.20130830git.fc21.src.rpm
crypto-policies-20140708-1.git3a7ae3f.fc21.src.rpm
cryptsetup-1.6.5-1.fc21.src.rpm
curl-7.37.0-3.fc21.src.rpm
cyrus-sasl-2.1.26-17.fc21.src.rpm
dbus-1.8.6-1.fc21.src.rpm
device-mapper-multipath-0.4.9-65.fc21.src.rpm
dhcp-4.3.0-15.fc21.src.rpm
diffutils-3.3-7.fc21.src.rpm
dmidecode-2.12-5.fc21.src.rpm
dracut-038-1.fc21.src.rpm
e2fsprogs-1.42.10-5.fc21.src.rpm
elfutils-0.159-5.fc21.src.rpm
expat-2.1.0-8.fc21.src.rpm
fedora-release-22-0.1.src.rpm
fedora-repos-22-0.1.src.rpm
file-5.19-1.fc21.src.rpm
filesystem-3.2-26.fc21.src.rpm
findutils-4.5.12-5.fc21.src.rpm
fipscheck-1.4.1-5.fc21.src.rpm
gawk-4.1.1-3.fc21.src.rpm
gcc-4.9.0-14.fc21.src.rpm
gdbm-1.11-2.fc21.src.rpm
glib2-2.41.1-1.fc21.src.rpm
glibc-2.19.90-26.fc21.src.rpm
gmp-6.0.0-5.fc22.src.rpm
gnupg2-2.0.24-1.fc21.src.rpm
gpgme-1.4.3-2.fc21.src.rpm
grep-2.20-2.fc21.src.rpm
groff-1.22.2-9.fc21.src.rpm
grubby-8.35-2.fc21.src.rpm
gzip-1.6-3.fc21.src.rpm
hardlink-1.0-19.fc21.src.rpm
heat-cfntools-1.2.7-1.fc21.src.rpm
hostname-3.15-2.fc21.src.rpm
initscripts-9.54-4.fc21.src.rpm
iproute-3.15.0-1.fc21.src.rpm
iptables-1.4.21-9.fc21.src.rpm
iputils-20140519-2.fc21.src.rpm
json-c-0.11-7.fc21.src.rpm
kernel-3.16.0-0.rc4.git2.1.fc22.src.rpm
keyutils-1.5.9-2.fc21.src.rpm
kmod-18-1.fc21.src.rpm
krb5-1.12.1-10.fc21.src.rpm
less-458-11.fc21.src.rpm
libarchive-3.1.2-8.fc21.src.rpm
libassuan-2.1.0-3.fc21.src.rpm
libcap-2.24-5.fc21.src.rpm
libcap-ng-0.7.4-5.fc21.src.rpm
libcgroup-0.41-3.fc21.src.rpm
libdb-5.3.28-6.fc21.src.rpm
libedit-3.1-6.20140213cvs.fc21.src.rpm
libestr-0.1.9-2.fc21.src.rpm
libffi-3.1-4.fc21.src.rpm
libgcrypt-1.6.1-5.fc21.src.rpm
libgpg-error-1.13-1.fc21.src.rpm
libidn-1.28-3.fc21.src.rpm
liblogging-1.0.4-2.fc21.src.rpm
libmetalink-0.1.2-5.fc21.src.rpm
libmnl-1.0.3-7.fc21.src.rpm
libnetfilter_conntrack-1.0.4-2.fc21.src.rpm
libnfnetlink-1.0.1-4.fc21.src.rpm
libpipeline-1.3.0-2.fc21.src.rpm
libpwquality-1.2.3-3.fc21.src.rpm
libseccomp-2.1.1-2.fc21.src.rpm
libselinux-2.3-3.fc21.src.rpm
libsemanage-2.3-3.fc21.src.rpm
libsepol-2.3-2.fc21.src.rpm
libssh2-1.4.3-13.fc21.src.rpm
libtasn1-4.0-1.fc21.src.rpm
libuser-0.60-4.fc21.src.rpm
libutempter-1.1.6-4.fc21.src.rpm
libverto-0.2.6-2.fc21.src.rpm
libxml2-2.9.1-3.fc21.src.rpm
libyaml-0.1.6-3.fc21.src.rpm
linux-atm-2.5.1-9.fc21.src.rpm
logrotate-3.8.7-2.fc21.src.rpm
lua-5.2.2-6.fc21.src.rpm
lvm2-2.02.107-1.fc21.src.rpm
lzo-2.06-6.fc21.src.rpm
man-db-2.6.7.1-4.fc21.src.rpm
mtools-4.0.18-5.fc21.src.rpm
ncurses-5.9-14.20140323.fc21.src.rpm
net-tools-2.0-0.25.20140707git.fc21.src.rpm
nspr-4.10.6-1.fc21.src.rpm
nss-3.16.2-1.fc21.src.rpm
nss-softokn-3.16.2-1.fc21.src.rpm
nss-util-3.16.2-1.fc21.src.rpm
openldap-2.4.39-8.fc21.src.rpm
openssh-6.6.1p1-1.fc21.1.src.rpm
openssl-1.0.1h-5.fc21.src.rpm
p11-kit-0.20.3-1.fc21.src.rpm
pam-1.1.8-12.fc21.src.rpm
parted-3.1-28.fc21.src.rpm
passwd-0.79-3.fc21.src.rpm
pcre-8.35-3.fc21.src.rpm
pinentry-0.8.3-2.fc21.src.rpm
pkgconfig-0.28-4.fc21.src.rpm
policycoreutils-2.3-6.fc21.src.rpm
popt-1.16-4.fc21.src.rpm
procps-ng-3.3.9-10.fc21.src.rpm
pth-2.0.7-23.fc21.src.rpm
pygpgme-0.3-10.fc21.src.rpm
pyliblzma-0.5.3-11.fc21.src.rpm
python-2.7.7-2.fc21.src.rpm
python-backports-1.0-4.fc21.src.rpm
python-backports-ssl_match_hostname-3.4.0.2-2.fc21.src.rpm
python-boto-2.27.0-2.fc21.src.rpm
python-chardet-2.2.1-1.fc21.src.rpm
python-cheetah-2.4.4-7.fc21.src.rpm
python-configobj-5.0.5-1.fc21.src.rpm
python-iniparse-0.4-14.fc21.src.rpm
python-IPy-0.81-11.fc21.src.rpm
python-jsonpatch-1.2-3.fc21.src.rpm
python-jsonpointer-1.0-5.fc21.src.rpm
python-prettytable-0.7.2-3.fc21.src.rpm
python-psutil-1.2.1-3.fc21.src.rpm
python-pycurl-7.19.3.1-3.fc21.src.rpm
python-requests-2.3.0-2.fc21.src.rpm
python-six-1.6.1-3.fc21.src.rpm
python-urlgrabber-3.10.1-1.fc21.src.rpm
python-urllib3-1.8.2-3.fc21.src.rpm
pyxattr-0.5.3-1.fc21.src.rpm
PyYAML-3.11-3.fc21.src.rpm
qrencode-3.4.2-2.fc21.src.rpm
readline-6.3-2.fc21.src.rpm
rootfiles-8.1-17.fc21.src.rpm
rpm-4.11.90-0.git12844.5.fc21.src.rpm
rsync-3.1.1-1.fc21.src.rpm
rsyslog-7.4.10-2.fc21.src.rpm
sed-4.2.2-7.fc21.src.rpm
selinux-policy-3.13.1-63.fc21.src.rpm
setools-3.3.8-3.fc21.src.rpm
setup-2.9.0-2.fc21.src.rpm
shadow-utils-4.1.5.1-14.fc22.src.rpm
shared-mime-info-1.3-9.fc21.src.rpm
sqlite-3.8.5-2.fc21.src.rpm
sudo-1.8.8-5.fc21.src.rpm
syslinux-6.02-5.fc21.src.rpm
systemd-215-3.fc21.src.rpm
tar-1.27.1-5.fc21.src.rpm
tcp_wrappers-7.6-77.fc21.src.rpm
texinfo-5.2-3.fc21.src.rpm
tzdata-2014e-4.fc21.src.rpm
ustr-1.0.4-16.fc21.src.rpm
util-linux-2.25-0.3.fc21.src.rpm
vim-7.4.307-2.fc21.src.rpm
xz-5.1.2-12alpha.fc21.src.rpm
yum-3.4.3-152.fc21.src.rpm
yum-metadata-parser-1.1.4-12.fc21.src.rpm
zlib-1.2.8-5.fc21.src.rpm
}}}

Thanks, Tom!

The %license change doesn't seem to be reflected in the packaging guidelines yet

I want to second what Rahul said above. This should be reflected in packaging guidelines and using %license instead of %doc for licenses should be recommended (probably not required?) for all packages.

Packaging Guidelines still say:

License Text

If the source package includes the text of the license(s) in its own file, then that file, containing the text of the license(s) for the package must be included in %doc

And fedora-review show this notes:

[ ]: Package requires other packages for directories it uses.
Note: No known owner of /usr/share/licenses
[ ]: Package must own all directories that it creates.
Note: Directories without known owners: /usr/share/licenses

Can I ignore these messages in the revision of the package where it NOT will be by default in Fedora Cloud? Or it is better do not use the %license macro yet?

I'm not sure why this is still set to meeting. It looks like it's ready to be written up, so I'll set that now.

I wrote this up, but I wasn't entirely sure about the
%{!?_licensedir:%global license %%doc}
bit. It seems this is only needed for EPEL 5 and 6 at this point, so I'm guessing that should just go in the EPEL guidelines.

Announcement text (important, should be at the top of the announcement):

%license must be used in place of %doc to designate any file containing the license information for a package. See https://fedoraproject.org/wiki/Packaging:Guidelines#Documentation and https://fedoraproject.org/wiki/Packaging:LicensingGuidelines

Replying to [comment:26 tibbs]:

I wrote this up, but I wasn't entirely sure about the
%{!?_licensedir:%global license %%doc}
bit. It seems this is only needed for EPEL 5 and 6 at this point, so I'm guessing that should just go in the EPEL guidelines.

Correct, only EPEL < 7 needs the compat hack.

Am Wed, 28 Jan 2015 12:02:27 +0000
schrieb devel-announce-request@lists.fedoraproject.org:

Date: Tue, 27 Jan 2015 21:03:05 -0600
From: "Jason L Tibbitts III" tibbs@math.uh.edu
To: devel-announce@lists.fedoraproject.org
Subject: [Guielines Change] Changes to the packaging guidelines
Message-ID: ufaoapjzjx2.fsf@epithumia.math.uh.edu
Content-Type: text/plain; charset=utf-8

%license must be used in place of %doc to designate any file
containing the license information for a package. See
https://fedoraproject.org/wiki/Packaging:Guidelines#Documentation
and https://fedoraproject.org/wiki/Packaging:LicensingGuidelines

IMHO this is a very good improvement. But is fedora-review (or especially rpmlint) able to
check for the existance of %license ? And what happens to all the existing packages, not only cloud ones? When do they have to confirm the new MUST guidelines and who/what is responsible to check that?

Well, I could do a script over all Fedora packages and in SCM, but it would produce a lot of bugzilla content, so well …

None of the issues you raise are FPC-related. We do not control the fedora-review or rpmlint scripts. We do not retroactively enforce packaging guidelines (or, indeed, enforce them at all.) If you would like to request clarification of the packaging guidelines, or to submit a draft, please open a new ticket with your issue.

note for completeness -- patch for fedora-review to support %license submitted:

https://lists.fedorahosted.org/pipermail/fedorareview/2015-February/000211.html

Note that:

{{{
License: BSD

%{!?_licensedir:%global license %%doc}
%doc README PATENTS NEWS AUTHORS
%license COPYING
}}}

does not appear to work on EPEL6. I get:

{{{
+ /bin/mkdir -p /builddir/build/BUILDROOT/libwebp-0.4.3-1.el6.x86_64/usr/share/doc/libwebp-0.4.3
+ cp -pr README PATENTS NEWS AUTHORS /builddir/build/BUILDROOT/libwebp-0.4.3-1.el6.x86_64/usr/share/doc/libwebp-0.4.3
+ exit 0
RPM build errors:
File must begin with "/": BSD
File must begin with "/": COPYING

}}}

In case anyone is stuck on this, here's a solution which worked for me:

{{{
%if 0%{?rhel} || 0%{?fedora} < 21
%doc LICENSE
%else
%license LICENSE
%endif
}}}

JFTR: https://fedoraproject.org/wiki/EPEL:Packaging#The_.25license_tag have the correct form :

%{!?_licensedir:%global license %doc}

Works with %doc not %%doc.

Metadata Update from @orion:
- Issue assigned to tibbs

2 years ago

Login to comment on this ticket.

Metadata