This is definitely no-go for me since it makes builds non-reproducible.

So let’s talk about what being reproducible means for Fedora. Because a misleading one liner dismissal with no context is not respectful of me and of other readers.

This request is the result of one year of hard hard Fedora work to make Fedora Go package builds reproducible. And there will still be more work ahead, it’s just a necessary stepping stone.

Reproducible actually means:

• I can give you a srpm (spec + patches…)
• you can point it to the same Fedora repo state that existed when I built it
• and you will get the same result in local builds, mock, copr, koji, whatever standard rpm toolchain works best for you
• you can understand, check, review, audit, fix, update every part of the process

But it also means:

• the elements in the srpm are either sources or simple packager crafted files
• the human crafted files are actually human crafted and do not rely on an out-of-band build or generation toolchain

The last point is something people often forget. It’s the difference between building from sources, and packaging things which have already been processed with a toolchain not tracked in the spec nor in build logs.

Not being reproducible is basically what was killing Go in Fedora:

• people were processing Go sources out of band
• it generated spec file elements without any form of tracking of what was generated when and with what tool (Sure the prefered tool was known and documented. If you ignored the fact it has versions like any bit of software. And versions can change the behaviour, hopefully for the best, but still, untracked changes)
• or what was generated then manually tweaked without any tracking

After a few years of out of band generation everyone stopped updating the corresponding Go packages. Other packagers were afraid to touch spec files that included generated parts that were generated in a build process they had no track of. Worse, the original packagers had long forgotten how exactly they did those, had no wish to relearn the eventual tweaks they had applied had the time, so they stopped updating their spec files too.

So, Fedora package ecosystem death and extinction, by core packages too old to be useful that no one updates.

What I’ve been doing for a year is ripping out all the pre-generated bits and moving all the generation logic in Fedora macros, deployed in Fedora packages, using other Fedora packages as dependencies. So:

• there is zero room or temptation to generate things outside the spec
• there is zero room to use generator elements not approved by Fedora
• there is zero room to use a generator version, not matching the version in the Fedora repository at the time of the build
• the generator is actually packaged and build in Fedora using Fedora technical (and legal) guidelines
• packagers do not have to expend efforts to massage files to workaround toolchain deficiences. They just won’t do it efficiently or reliably, it does not scale, it is not durable.
• everything happens in rpmbuild using Fedora repositories. No sneak sideband internet calls. Everything sourced in our repositories. Everything tracked in Koji logs. And one should probably implement better tracking Koji-side, but you know? With this setup you can do more, because everything happens in rpmbuild using Fedora repositories. The sky’s the limit. You will never be able to state the same if any part of the process happens out of band outside rpmbuild.

Since Go massively reuses code, with a huge number of intertwinned dependencies that change quickly, the original spec generator spent quite a lot of energy computing those. And making Go builds reproducible means pushing BuildRequire generation in the Fedora rpm build process. Otherwise packagers will continue to generate out of band with the results everyone saw before.

The only technical solution that exists and works to do those within Fedora right now is pm request. That’s why I ask to enable it. Even if it’s not ideal. It’s needed so we can continue to fix the Fedora packageset for a major language, that pretty much every single container tech out there depends on, without packagers burning out mid process because they’re asked to do things out of band, without infra support, to avoid the shore of getting Fedora infra fixed.

Short term koji could be enhanced to track what pm request installs, pm request already tracks what it’s asked to install by the spec, that’s just a matter of relaying the info to the Koji build database.

Mid term mock could be enhanced to do the same thing as pm request, just better without annoying socket communication. That’s requested and tracked in
https://github.com/rpm-software-management/mock/issues/160

Even longer term rpm could be enhanced to track this all natively in srpm headers so rpm -qp works with this new info. That’s requested and tracked in
https://github.com/rpm-software-management/rpm/issues/104

But in the meanwhile we have to work with the state of the tools as they exist now, because Fedora packages need it now, and we are losing packager engagement now.

And the proposal:

• is 100% forward-thinking and compatible with the mock enhancements discussed in https://github.com/rpm-software-management/mock/issues/160
• is 100% forward-thinking and compatible with the rpm enhancements discussed in https://github.com/rpm-software-management/rpm/issues/104,
• works today
• is rpm template-friendly
• requires no future change in Fedora spec files, just a switch macro-side from pm request to the better methods when they have been implemented.

I can even anticipate the changes discussed in those two tickets, by making the current macro log BuildRequires to a file before passing them to pm request, instead of using a shell env var like I do today. And even package this file in the corresponding rpm package produced by the build process so the state is tracked without relying on Koji logs. That’s trivial to do. One day work max.

And ignatenkobrain has no alternative solution to propose.

He will tiptoe around the subject, hiding the pregenerated bits in a source file, stating the generation logic he uses for Rust is better than the one the Go SIG used, and does not need tracking, or that the out-of-band generator is hidden in fedpkg, so surely that makes it kosher, but the end result is the same.

What he wants to do is not reproducible or tracked by Fedora infra, it’s awfully Fedora-specific so it can’t be shared with other distros or even EL. He just found a thicker carpet to hide the pre-generation problem under. It’s basically the state I was at months ago, before I finished fixing things and integrating the BuildRequires logic in normal classical packager-friendly rpm macros.

I’m not asking to hide this problem I’m asking to start solving it because Fedora needs it solved.

pm request was written by the Java SIG. It’s not a Go specific solution. Many Fedora packaging SIGs need something like this today. They are losing packagers now because our tooling does not handle this technical need, and we’re discussing what-ifs doing nothing to fix this situation.

What is pm_requests?

pm_request is a Mock plugin module that lets you request mock to request the package manager to install things while you're in the middle of an operation.

It was conceived by the Java SIG developers so that they could discover dependencies and cycle through builds until they got a list recorded to put into a spec file for an official build.

Hi Till,

pm request is basically a way for the spec to ask mock to install new packages in the buildroot. Technically, the spec writes install x y z over a unix socket, mock asks dnf to install those, and replies ok/nok over the socket.

So right now it’s used by the Go macro rework I’m finishing up to ask mock to install BuildRequires at the end of %prep (after the packager used %prep to clean up sources, so BuildRequires computation starts from a controlled state).

You can see it in action at line 2508 of
https://paste.fedoraproject.org/paste/hglkYSWJAHd2YuGBX2XnsA

At this point of the build %prep setup is finished, the macro has computed %build requirements, and the spec can ask mock to instal those (this is a small easy Go package, real-world packages can have many more deps than that).

The mock-install command is just a small helper to write install x y z over the socket and read what mock answers.

It’s not ideal because pm request:

1. needs a socket, so not easy to use by a packager in a spec (I had to write mock-install to workaround this)
2. is probably more flexible that we actually need. I'd rather have the buildsys look for additional BuildRequires at specific point of the build process, than have specs poke the build system whenever they like
3. it all happens without rpmbuild involvement, so the result is not traced in rpm headers

Points 1. and 2. require mock changes (https://github.com/rpm-software-management/mock/issues/160)
Point 3. requires rpm changes (https://github.com/rpm-software-management/rpm/issues/104)

But nothing has been happening for years, everyone waits for the other to do the first step. And in the meanwhile the packaging situation in Fedora degrades (too much impedance mismatch between upstreams that posit what amounts to dynamic BuildRequires, and the lack of any solution at the distro level).

So I'm asking to deploy the existing solution, even if it is imperfect, because it is sufficient to start producing working Fedora packages, and it will get all everyone to focus on making it better, rather than waiting for the ideal solution to materialize from thin air.

Why do you need to install packages after %prep instead of specifying them as BuildRequires?

Short answer

• Specifying BuildRequires in the spec manually does not scale to the needs of current software.
• For many years, several Fedora SIGs have quietly used unofficial shadow preprocessing outside Koji's help, control and overview to maintain their packages.
• And today we are past the point where reinjecting things in specs from a shadow preprocessing phase does not scale itself (at least for Go, probably also for other languages)

(Very) long answer

git has been so successful in making software projects cheap and easy, you have a forest of inter-component build dependencies, that change quickly (blame Linus for this if you want to blame someone. I rather like the result myself).

For a long time already, and for many software languages, it has been impossible to identify BuildRequires manually by reading a static human-oriented INSTALL document and translating it manually to BuildRequire lines.

At best you have some form of computer-oriented software manifest inside each version of the source archive that tells you what the current BuildRequires are (maven-style). At worst you need to run code inspection tools on the codebase to analyse the needs of this codebase (go-style). Both require you have finished preparing sources so you can execute the bit of software that reads the manifest or performs the software analysis.

It's not an autoconf-style world anymore. Software does not adapt to the state of the system it builds on. Internet is cheap. git is cheap. vms are cheap. Open source software is plentiful. Take my BuildRequires list and make it available, or I won't build. Don't pretend to me you're one of that old legacy unixes where installing anything is impossible, I don't believe you and I don't care.

Preparing the codebase for this check is basically what a packager does in %prep. That's why the Java SIG wrote pm_request in the first place. To reuse the work done in prep and avoid managing a separate "compute buildrequires" packaging process.

Golang, as a post-git computing language, pushes this to the next level. Not only you need to run tools on the codebase, but you need to run them for pretty much every package version bump, because the dependency set changes quickly between versions.

So you can no longer afford to run things semi manually in a separate workflow, then copy the result to a static list, then feed it to a spec, then rebuild the again the resulting with a static BuildRequires list in the spec (multiply by a gazillon of interdependent software components)

It eats your packager precious time. It demotivates them. Some start replaying old BuildRequires lists to avoid the recomputing. Your mass rebuilds start failing because of the nest of not-needed-anymore-but-still-present BuildRequires lines in spec files. And you have no way to detect those before mass rebuild post mortems, because the whole BuildRequire computation happens outside the normal build process.

And, even when you find an ultra-motivated packager that will do things right every time without QA controls, every minute he spends computing BuildRequires out-of-band, then injecting the result in the spec file, is one minute less he has to check build logs, test the resulting package, engage with upstream to get problems fixed.

What pm request permits is first to pull back the whole dependency computation process within the scope and control of Fedora tooling and second to automate it away to liberate precious packager time.

And thus make it possible to package more and better, without requiring a mass of new packagers. And make it possible to package things, that would be too expensive in manpower to package otherwise.

pm request is not dangerous, because everything it sources comes from Fedora repositories, so assuming we did our reviews and QA right, is safe to use. And if a BuildRequired package is unsafe wasting someone's time to write the BuildRequire line manually, won't make it any safer.

If Koji had a way of recording pm_request information so that a build environment can be replayed when a "task" is rebuilt (for reproducible builds, for example), then I'd have less objection to this.

However, this does not currently exist, so I'm wary of allowing this right now without that enhancement.

I filed https://pagure.io/fesco/issue/2004 to ask fesco for their thoughts on this.

Since Fesco rejected the request, closing can't fix.