packaging-committee

Clone
Source Code
GIT

#813 Default rpm macro packaging layout and conventions
Opened a year ago by nim. Modified a year ago

Open
Close issue as:
accepted rejected nothingtodo invalid fixed wontfix duplicate temporaryexception permanentexception exceptionexpired

I'd like Fedora and FPC in general to agree to a general layout for rpm macros and their associated material like templates, so people can focus on writing actual macros and their documentation, and not on how the result is shipped to users.

The reason being I'm currently wrapping up three sets of rpm macros for Fedora⁰. The coding and first-pass testing for each macro set is mostly done, the next step is packaging, documenting, and guideline-ing.

I'd like to do the documenting and guideline-ing for each macro set via standard templates. Wiki (or the new docs site) is a huge time sink to write and update, and most packagers do not read the result, so it's mostly a waste of time. OTOH templates are operational and do work. Thus with hindsight, a short paragraph in guidelines that points to a package of templates, works loads better than a wall of wiki text

Here is how I understand today's Fedora best practices. To my knowledge, they are mostly unwritten packager lore. (obviously stuff in redhat-rpm-config is a special case)

%<--

A. A single project hosts macros, templates, documentation, and other helpers in packaging <foo> material, and keeps everything in sync.

  • the project SHOULD be named <foo>-rpm-macros
  • it SHOULD be hosted on Fedora infra, or a cross-distribution hosting site
    (ie pagure.io, even if I find it severely limiting from a forge POW).
  • Fedora infra makes it easy to assign/reassign ownership to SIG members

B. The project SHOULD be packaged in a <foo>-rpm-macros source package

  • the source package MAY buildrequire anything
    • however, it SHOULD be solely composed of macros, docs, scripts, that do not need much building
    • anything requiring a binary build SHOULD be packaged separately then required by the appropriate <foo>-rpm-macros subpackage

C. The <foo>-rpm-macros spec file SHOULD generate at least some of the following subpackages

  • using -n, to avoid human-unfriendly package names

C.1 A package named <foo>-srpm-macros

  • this package MUST exist if the macro set defines anything needed by rpmbuild -bs to create source packages that include <foo> code or content
  • it MUST be in the default build root (buildsys group)
  • it MUST NOT require anything and be small
    • we want builds to be fast and efficient, and only pull ecosystem-specific buildrequires in the build roots that actually need them
  • it SHOULD define a pivot macro expected to be present in every <foo> spec file
  • this macro SHOULD generate a buildrequires on <foo>-rpm-macros

C.2 A package named <foo>-rpm-macros

  • this package MUST provide any remaining rpm macro or constant needed by rpmbuild -ba to create deployment (sub)packages that include <foo> material
    • typically at least the constants defining standard file system locations for <foo> material
  • it MUST NOT assume it will exist in the default build root. It will be pulled in:
    • by the pivot macro if in <foo> spec files if <foo>-srpm-macros exists,
    • by direct buildrequires otherwise
  • it SHOULD require any other package typically needed to build <foo> (sub)packages.
    • require restrictions are limited to <foo>-srpm-macros, not <foo>-rpm-macros
    • for example: the Go compiler for Go packages, fontconfig for fonts packages,
  • it MUST require <foo>-filesystem if this package exists
  • it SHOULD NOT be necessary to <foo> packages once built, nor be required by those packages

C.3 A package named <foo>-filesystem

  • this package MUST materialize and own the file system locations defined in <foo>-rpm-macros
  • it MUST NOT require anything except other filesystem-style packages
  • it SHOULD be depended on by <foo> packages whenever they make use of those locations

C.4 A package named <foo>-rpm-templates

  • this package SHOULD depend on rpmdevtools
  • it SHOULD provide at least one template in %{_spectemplatedir} showing how <foo> macros are supposed to be used
  • it SHOULD contain any other documentation file, necessary to understand those templates
  • it SHOULD NOT be necessary to <foo> packages once built, nor be required by those packages

C.5 A package named <foo>-rpm-tools

  • it SHOULD provide anything not strictly necessary to execute the macros or to understand the templates, but can be useful to some packagers, typically convenience scripts
  • it SHOULD NOT be necessary to <foo> packages once built, nor be required by those packages

%<--

Did I get this right or did I forget something important?

Regards,

⁰

  1. "forge macros" v2 (v1 was merged in redhat-rpm-config 9 months ago)

macros to map <url,version,tag,commit,branch> metadata to classical rpm Source, URL, %setup, %dist, etc verbs, when a project is hosted on Gitlab, GitHub, etc (not Pagure because Pagure is missing needed functionality). There are hundreds if not thousands of packages in the distro that can make use of this.

v2 adds the ability to process multiple archives in a single source rpm.¹

2 "go macros" v2 (v1 was merged in go-compilers and go-srpm macros about the same time as the forge macros)

automation for packaging software written in the Go language (Go language). Go is getting huge nowadays, it's the language in which kubernetes, docker, and a lot of cloud infra software is being written today. This is really why I'm doing all this, the forge macros are just a spinoff of generic non-Go-specific functionality that Go macros need.

v2 add the ability to process multiple archives in a single source rpm, and Go BuildRequires automation (one of the huge missing bits in the v1)²

3 font macros v2 (v1 + associated guidelines were written by myself around a decade ago)

Another spinoff of 1. and 2. Mostly because golang-x-image is the upstream of the Go fonts, which has caused me to revisit the subject.

With the framework code written for 1., ten years of hindsight, experience, and rpm improvements, I can and did write macros that remove known problems in the current set, add missing functionalities like appstream handling, and are generally more convenient to use (took me about a week-end of hacking)³

¹ A first version has already been completed and merged in redhat-rpm-config 9 months ago. Thanks to everyone involved.

https://src.fedoraproject.org/rpms/redhat-rpm-config/c/7c4cd330850ee228ca727b80516e187cf87f30c7

It worked and still works fine, you can use it today in your packages, but it has a huge limitation: it assumes you only need to process a single source archive in the specfile. People rightly pointed out in the review that, while it's the general and preferred case, some packages do need to mix source archives (esp. on EL given the hard constrains EL has on not touching existing package layout).

Therefore I've prepared a new version that can process multiple sources. It's backwards-compatible, multi-source is achieved via new flags, in the absence of those flags it will assume a single source like the current merged code does.

https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/35

It adds a huge level of complexity macro-side, rpm does not really provide the framework to do this kind of multiplexing, the framework had to be coded from scratch in lua macro code. However, the framework has been coded now, it's generic, and rightly belongs in a generic package like redhat-rpm-config

² Currently in the final stages of technical testing before I document it and send a PR
https://github.com/nim-nim/go-macros/commits/dev

³ Currently past the technical testing, need to document it, send myself a PR, send the result to FPC and the Fonts SIG to switch existing packaging guidelines

IMO, this is something that Go-SIG should formalize and after agreeing on it, it should be proposed to the FPC/FESCO, but it is my opinion, if @nim thinks that this is a way lets be it

I totally agree with jcajka that having rules to package macros, does not obviate the need to review those macros with the concerned SIG before shipping;)

The way macro packages are named and split, however, should be generic and shared by all sigs.

I totally agree with jcajka that having rules to package macros, does not obviate the need to review those macros with the concerned SIG before shipping;)
The way macro packages are named and split, however, should be generic and shared by all sigs.

IMO should be discussed with them first.
Edited a year ago by jcajka

I can tell you how we have done it in Rust, and I believe this is right way:

  • rust-srpm-macros is separate package built on all architectures, contains %{rust_arches} with list of arches rust supports
  • rust-packaging contains rust-packaging and rust2rpm subpackages, first one contains dependency generators and contain Requires: rust cargo (so that rust crates have to do only BuildRequires: rust-packaging). I probably do not need to describe what second package does.

@ignatenkobrain Yes the <foo>-srpm-macros separate package is what pretty much everyone seems to be converging on. Not necessarily a separate srpm BTW, as soon as you start doing complex things in the srpm phase, you really need it synced with the rest of the packaging infra.

rust-packaging seems to be the equivalent of the <foo>-rpm-macro name I proposed (named fontpackages-devel for the current generation of fonts macros for example, and go-compilers for go even though most of it has stopped being about compilers for quite a long time)

So, another point for setting a common naming target, so everyone does not keep reinventing custom naming for the same things. It's loads friendlier for FPC and packagers alike to have common naming conventions, instead of having to guess ecosytem by ecosystem how the same things are named.

Mind you, I'm not proposing to force SIGSs to rename existing packages. They can just add a Provides: common name to their packages once FPC agrees on this common name.
Edited a year ago by nim

Not necessarily a separate srpm BTW

It has to, because rust-packaging has to have ExclusiveArch and it is affecting whole package. So you simply can't have it in one SRPM =(

It has to, because rust-packaging has to have ExclusiveArch and it is affecting whole package. So you simply can't have it in one SRPM =(

You can if you apply the tricks fedora-release uses
https://copr-be.cloud.fedoraproject.org/results/nim/go-macros-3/fedora-rawhide-x86_64/00812419-go-macros/go-macros.spec

Not as pretty as I'd like but far less dangerous than having your macros split into two srpms with "interesting" build order and synchronized push to repo requirements.

Edited a year ago by nim

I'll toss in some opinions:

  • Using foo-packaging has the potential for conflicts. For example, there is a "packaging" module for python. For actual macros, I prefer foo-srpm-macros and foo-rpm-macros; those have less chance of conflicting but should probably only actually include macros. Things like specfile generators will have upstream names under which they should be packaged.

  • Often these macro packages would be built as part of some other package. But if not, I prefer "foo-rpm-macros" for the source package name.

  • I don't see the point of putting some requirement for how the sources are hosted in the guidelines. The src.fedoraproject.org repository should be sufficient in any case.

  • I don't see the point in a guideline about "*-filesystem" packages. We do mention these and file/directory ownership in general in the main guidelines. Plus directories mentioned in macros might be dynamic (depend on something like %name, for example).

  • I think adding stricture for naming for macro templates and "convenience scripts" is probably getting out of the scope of what we'd put in guidelines. If anything, since templates would depend on rpmdevtools they could reasonably be named like "rpmdevtools-foo".

Thanks for the input.

I think adding stricture for naming for macro templates and "convenience scripts" is probably getting out of the scope of what we'd put in guidelines.

You only think that way because you've not seen many of them. As soon as there are more than a couple in distribution you'll regret they all use different conventions.

If anything, since templates would depend on rpmdevtools they could reasonably be named like "rpmdevtools-foo".

Ok, I'll use that then.
Not sure it's the best naming though, some guidelines require packagers to create other things than spec files (appstream, fontconfig, etc) so packaging templates are not limited to the spec part rpmdevtools handles
Edited a year ago by nim

Often these macro packages would be built as part of some other package. But if not, I prefer "foo-rpm-macros" for the source package name.

I can live with it. I'll do it. Thanks for the feedback.

The main drawback I see is that due to rpm’s stupid srpm = rpm description design, the srpm description will only talk about the rpm subpackage, unless you want to completely confuse packagers.

If anything, since templates would depend on rpmdevtools they could reasonably be named like "rpmdevtools-foo".

Ok, I'll use that then.
Not sure it's the best naming though, some guidelines require packagers to create other things than spec files (appstream, fontconfig, etc) so packaging templates are not limited to the spec part rpmdevtools handles

After playing with it, rpmdevtools-foo does not work too well (rpmdevtools is just here for the directory, all the rest does not depend on rpmdevtools mecanisms, and the rest includes templates for things rpmdevtools has no idea on how to manage)

So I’ll try foo-rpm-templates which is more generic and less rpmdevtools-centric
Edited a year ago by nim

So I’ll try foo-rpm-templates which is more generic and less rpmdevtools-centric

and it even works in redhat-rpm-config without feeling a bolted-on afterthought!

Login to comment on this ticket.

Metadata
None
None
None
None
Needs Review
None
None
Powered by Pagure 5.8.1
Documentation • File an Issue • About • SSH Hostkey/Fingerprint
© 2014-2019 Red Hat, Inc. and others.