#303 Consider reverting decision to ban %{?_isa} in BuildRequires
Closed: Fixed None Opened 5 years ago by ellert.

The packaging guidelines were recently amended to ban the use of %{?_isa} in BuildRequires. I consider this to be a unfortunate decision that causes a lot of problems and disruptions and I would like to urge you to reconsider this decision.

The new guideline was introduced in order to work around a tiny glitch in the way rpmbuild works that causes the BuildRequires to appear as Requires of the buils source RPM, which is a minor problem and is not proportional to the huge potential problems this change introduces by mandating insufficient accuracy in the BuildRequirements.

A BuildRequirement without %{?_isa}, like .e.g zlib-devel can be satisfied by either zlib-devel.x86_64 or zlib-devel.i686, but only one of these packages actually provides the resources needed to build the package. If you have only one of them installed and it is the wrong one, your package build will either fail or build a package with missing features.

By adding proper %{?_isa} to the BuildRequirements this problem is avoided.

I consider getting this right is for more important than avoiding having a few bogus Requires in the built source rpm. Using the Requires of the source RPM to evaluate the build requirements of the package is not reliable even if there are no %{?_isa} tags, only parsing the specfile included in the source rpm can achieve this reliably.


A BuildRequirement without %{?_isa}, like .e.g zlib-devel can be satisfied by either zlib-devel.x86_64 or zlib-devel.i686, but only one of these packages actually provides the resources needed to build the package.

By adding proper %{?_isa} to the BuildRequirements this problem is avoided.

I think you are confused about why this was rejected.

Using _isa on .src.rpm packages is much like using it on .noarch.rpm packages, it looks like it solves a problem when you do testing as long as you don't test correctly with a good number of arches (the most common being to test on just x86_64, and see that it filters out .i386 packages which is what you want). But x86_64 is not the only arch in Fedora, so this can/will happen:

  1. You submit new package with _isa in the build requires.

  2. Random ppc or arm machine happens to get assigned the job of turning the .spec+sources into a .src.rpm.

  3. x86_64 machine tries to turn the generated .src.rpm into a .x86_64.rpm, sees requirements for "ppc-32" binaries and fails.

Replying to [comment:1 james]:

  1. x86_64 machine tries to turn the generated .src.rpm into a .x86_64.rpm, sees requirements for "ppc-32" binaries and fails.

It does not work like that. An x86_64 machine is perfectly capable of rebuilding a source rpm generated on a ppc maching with proper _isa source dependencies in the spec file in order to create x86_64 binary packages.

That this argument is repeated again and again shows that the decision to ban _isa in BuildRequires was not taken on the proper grounds.

The argument stated above is the only one of all the arguments given against _isa in BuildRequires that actually would have some merit if it was true - which it is not. The other arguments I have seen in this thread are all about glitches due to some tool using metadata from the source rpm header for things it was not intended to be used. Those arguments are reasons for fixing the tools to use metadata from proper sources, not a reason to force Fedora packagers to write deliberately broken spec files without proper _isa in BuildRequires.

With all due respect, please don't hide during a public discussion about this on "packaging" list and close this ticket with a false explanation. If the FPC votes on this (and they have done that before), all members must understand the topic. I wish the FPC would be more explicit about their point of view. The response in this ticket raises questions, and I agree with Mattias that it raises doubts about the FPC's decision.

So, to rehash:

It has been pointed out before (and acknowledged) that build tools, such as Mock, '''rebuild''' (!) a src.rpm for the build target prior to building the binaries. You should not rely on a downloaded src.rpm's Requires being the full !BuildRequires for the package. The binary rpmbuild will verify the build requirements when parsing the spec file.

I won't repeat my few posts from the linked mailing-list thread, but just that if %_isa in BR were to be permitted, we should fix the -source repo metadata, tools like yum-builddep, and perhaps even get rid of src.rpm Requires.

Banning %{_isa} in build-requires is just closing your eyes (or shooting the messenger if you like that better) from the fundamental issue of arch-dependent build-requires and defiencies in our tooling/repodata to handle them.

There are loads and loads of packages with arch-conditional build-requires due to eg language interpreters/compilers only being available on certain archs, different hw capabilities etc, such as
{{{
%ifarch %{ocaml_arches}
BuildRequires: ocaml
%endif
}}}

Also some packages must be able to refer to both primary and secondary arch packages on multilib arches, for example gcc.spec:
{{{
%ifarch %{multilib_64_archs} sparcv9 ppc

Ensure glibc{,-devel} is installed for both multilib arches

BuildRequires: /lib/libc.so.6 /usr/lib/libc.so /lib64/libc.so.6 /usr/lib64/libc.so
%endif
}}}

For all those cases the source-repository requires get randomly broken for different architectures on build-by-build basis, depending on which arch the src.rpm happened to be generated on, regardless of whether %{_isa} is used or not.

There are various things that could be done to address the tooling/repodata issue instead of arguing about banning one aspect of an absolutely vital feature (arch-dependent build-requires, that is), including but not limited to:
"yum-builddep foo|foo-1.0-1.src.rpm" could be made to just download & unpack the src.rpm and figure the actual build-requires by parsing the spec. This is probably the lowest hanging fruit, but it wont make repoquery for build-requires any better
It should be possible to arrange per-arch source repositories with accurate data with using src.rpm headers only (mock creates src.rpm's separately for each arch anyway, these are just thrown away currently) without exploding the space requirements for source repositories - everybody agrees that storing full src.rpm for all archs just for the metadata-difference would be lunacy. This would obviously require quite some changes to several tools & scripts though...

Panu, thanks for jumping in.

We'd like to know specifically:

  • In the case where a SPEC has a BuildRequires: foo%{?_isa}, and is generated into a SRPM where %{?_isa} is evaluated and thus, the SRPM headers say: BuildRequires:foo.sparc64

If that SRPM is downloaded by a user onto a system of a different architecture, then run through: rpmbuild --rebuild foo.src.rpm

Will rpmbuild operate off the SRPM headers or will it install the SRPM components first, and evaluate things locally from the spec?

rpmbuild evaluates the spec file contents, not the src.rpm headers.

Testing shows that it appears to reparse the spec file:

{{{
$ uname -p
x86_64
$ grep _isa python-kitchen.spec
BuildRequires: python-sqlalchemy%{?_isa}
$ rpmbuild -bs python-kitchen.spec
Wrote: /srv/git/python-kitchen/python-kitchen-1.1.1-4.fc17.src.rpm
$ rpm -qp --requires /srv/git/python-kitchen/python-kitchen-1.1.1-4.fc17.src.rpm |grep sqlalchemy
python-sqlalchemy(x86-64)
$ scp python-kitchen-1.1.1-4.fc17.src.rpm x86.lan:
$ ssh x86.lan

x86~% uname -p
i686
x86~% rpm -qp --requires python-kitchen-1.1.1-4.fc17.src.rpm|grep sqlalchemy
python-sqlalchemy(x86-64)
x86~% rpm -q --whatprovides 'python-sqlalchemy(x86-64)'
no package provides python-sqlalchemy(x86-64)
x86~% rpmbuild --rebuild python-kitchen-1.1.1-4.fc17.src.rpm
Installing python-kitchen-1.1.1-4.fc17.src.rpm
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.78Bznr
+ umask 022
+ cd /home/badger/rpmbuild/BUILD
[...]
Wrote: /home/badger/rpmbuild/RPMS/noarch/python-kitchen-1.1.1-4.fc17.noarch.rpm
Wrote: /home/badger/rpmbuild/RPMS/noarch/python-kitchen-doc-1.1.1-4.fc17.noarch.rpm
[...]
+ cd /home/badger/rpmbuild/BUILD
+ rm -rf kitchen-1.1.1
+ exit 0
x86~% echo $?
0
x86~% ls -al ~/rpmbuild/RPMS/noarch/*rpm
-rw-rw-r--. 1 badger badger 136270 Aug 29 11:49 python-kitchen-1.1.1-4.fc17.noarch.rpm
-rw-rw-r--. 1 badger badger 249705 Aug 29 11:49 python-kitchen-doc-1.1.1-4.fc17.noarch.rpm
}}}

Good that this has been verified once more. I've mentioned it in my 5 weeks old comment in this ticket ;-), where I tried to give a summary.
{{{
The binary rpmbuild will verify the build requirements when parsing the spec file.
}}}

What I find not pretty about it is that if rpmbuild complains about missing build requirements, they would not match actual package names: foo-devel.i686 != foo-devel('x86-32'). Without %_isa, the mapping from foo-devel to foo-devel.i686 is more
obvious (cf. "yum list foo-devel"), albeit a minor issue compared with forbidding src.rpm queries.

%_isa-equipped BR would be more strict when dealing with direct rpmbuilds from spec files or src.rpms. So much ought to be clear. In the meeting there has been some confusion. Imagine "!BuildRequires: foo-devel". If on x86_64 you want to build for i686, rpmbuild would not complain if foo-devel.i686 is missing, and it would be up to %configure (or similar) to detect whether the installed foo-devel.x86_64 would work or not (headers might work, lib linking test would fail). Build might fail in weird ways. With %_isa BR it would fail early.

Leaving aside the repo metadata and src.rpm query problems, %_isa could not be added to all BR, unfortunately. Not even to all -devel package BR, because there are noarch -devel packages (fun for packagers: they either would not add %_isa to BR at all or have to examine all their BR for noarch or multilib capability). Not to all development tools either, because some are noarch (e.g. autoconf/automake), others are arch-specific (e.g. libtool).

Adding to the headache after reading the meeting log (does anyone really like killing the fedora-source repoquery possibilities?):

Assume a spec file with "!BuildRequires: some-devel-tool" ('''no''' %_isa!), and that BR is available in the repo for multiple archs, e.g. x86_64 and i686, because of how it's packaged and pulled in by the multilib repo composer. So far the spec file could be built (when not using Mock!) with some-devel-tool for either arch. If a packager added %_isa to the BR, that would force users to install tools for the compat-arch while only either one can live in /usr/bin anyway.

Who will verify that %_isa-equipped BR will also work for all compat-archs of a multi-arch installation? Example: "!BuildRequires: foo%{?_isa}" in the spec works when building for native x86_64. It works when building for native i686. Will it also work when building for i686 on x86_64? Is foo.i686 guaranteed to be available in the x86_64 repo? I believe it isn't (because of the multilib compose method, and even the blacklist).

Anyway, whether to ban %_isa-equipped BR boils down the question what to do with source repo metadata, src.rpm queries, and tools like yum-builddep and repoquery?

Apologies for apparently not being clear, and also I simply did not realize there's this much confusion over how this all works (having read through the meeting log), otherwise I would've tried to be present in the meeting.

As others have already pointed out, rpmbuild never uses the src.rpm headers for build-requires.
'rpmbuild --rebuild foo.src.rpm' first unpacks the src.rpm and then evaluates the spec, ie the very equivalent of manual 'rpm -i foo.src.rpm && rpmbuild -ba ~/rpmbuild/SPECS/foo.spec'. The only thing rpmbuild ever uses the src.rpm headers for is to determine whether it can actually unpack it, meaning specifically looking at rpmlib() dependencies of the src.rpm (which are always arch-independent). The actual build-requires are however ALWAYS parsed from the spec directly.

It's the tools that do look at the src.rpm headers and assume they're arch-independent that run into trouble. For example the deal with yum-builddep is that it has three different "modes":
1. 'yum-builddep foo' which looks up the requires purely from the repository metadata. This works properly only if the source repositories are arch-specific. As this is not the case in Fedora (or anywhere else for that matter), the results are plain lottery. To add to the insult, repodata hides the actual architecture of src.rpm's (that src.rpm headers reflect the arch they were built on was "discovered" long, long time ago but swept under the carpet by renaming the arch to 'src') so this cannot be even detected.
2. 'yum-builddep foo.src.rpm' which looks up the requires from the src.rpm headers. This works properly only if the src.rpm was created on the same arch as yum-builddep is being run. With src.rpm's downloaded from Fedora source repositories, this is lottery for the same reason as above, with locally (re)built src.rpm it is realiable. In this case, yum-builddep could actually verify whether src.rpm header matches current arch and at least error out (but optimally just handle the issue because its entirely possible)
3. 'yum-builddep foo.spec' which parses the spec to evaluate the build-requires. This is the most recent addition (about three years ago) and uses librpmbuild to parse the spec to evaluate build-requires and thus always gets the dependencies right.

That our buildsys and related tools (koji/fedpkg/mock) do the equivalent of 2) largely comes from the fact that they predate the required rpm-python bindings to do it in any other way. And in fact there's absolutely no problem with doing that as long as the src.rpm is locally (re)built, which is what the buildsys does.

For getting build-requires installed, the most common problem case is an individual developer running 'yum-builddep foo' or 'yumdownloader --source foo && yum-builddep foo*.src.rpm' and expecting it to work (which is, of course, not an entirely unreasonable expectation :) Both cases could be fixed without changing the source repositories by making yum-builddep do a bit more work behind the scenes (ie just download the src.rpm if necessary, and extract the spec / rebuild the src.rpm locally and never mind the dependencies in the repodata)

The other problem case is repoquery for things like "which packages build-require foo-devel and could break / will need a rebuild when foo is updated?" Such results are not accurate currently either because of arch-conditional build-requires but it does give a reasonable clue, but yes %{_isa} in buildrequires would complicate these queries somewhat as they are on arch-dependent virtual provides instead of simple package names. OTOH various build-requires on virtual provides exist already, %{_isa} in BR (if commonly used) would just make them more wide-spread (much like they pronounce the generic issue of arch-conditional build-requires), and fixing the related issues would improve the accuracy of such queries overall. Unlike the yum-builddep issues, properly solving this does require changes to the way source repository metadata is generated though.

Not sure why this was never considered. Bumping to the meeting agenda.

We discussed this at today's meeting (http://meetbot.fedoraproject.org/fedora-meeting-1/2015-03-12/fpc.2015-03-12-17.00.txt):

  • 303 Consider reverting decision to ban %{?_isa} in BuildRequires


    (geppetto, 17:55:16)
  • LINK: https://fedorahosted.org/fpc/ticket/303 (geppetto, 17:55:17)
  • ACTION: We have no problem changing this if the tools work.
    yum-builddep doesn't, the dnf builddep rewrite also doesn't work.
    koschei which is a new Fedora tool also relies on not having to
    download/extract .src.rpm files. (geppetto, 18:15:00)
  • ACTION: So if you want to change policy here someone will have to
    speak to and work with all the tool authors to make sure the tools
    still work, if they do then changing policy should be trivial.
    (geppetto, 18:15:59)

  • Open Floor (geppetto, 18:24:03)

  • Lots more talking about ticket 303, arched build requires and using
    _isa in buildrequires (geppetto, 19:05:15)

...you probably need to look at the full meeting log, but to try to sum it up here:

  1. There are a lot of hacks here, for a bunch of reasons, and everything mostly works.

  2. Making it work 100% either requires working within known limitations, or fixing them. Altering the limitations can be dangerous as you can break more than you fix.

  3. yum-builddep and dnf builddep both still use source-repodata/source-rpm-headers, even when given local .src.rpm files as input. The later case might be possible to fix with 2015 APIs, but there are other cases.

  4. koschei currently relies on using source-repodata, AIUI.

  5. Dennis has a proposal that you should/must use _isa whenever the source package uses ifarch/ifnarch to alter the buildrequires. koschei maintainer would prefer that to be just when ifarch/ifnarch refer to primary Fedora arches. Without doing actual votes, I'd guess either proposal as an ok chance of passing.

  6. AFAIK the only 100% fixes here are: i) Ban anything that would make source headers arch dependent. ii) Produce arch'd source repos. (we don't necessarily need N source packages to be mirror'd). iii) Fix all the tools to ignore requires/etc. data on source packages (including createrepo).

  7. AFAIK nobody wants to do any of the options for 6, so hacks it is.

Some thoughts:

  1. "dnf builddep foo.src.rpm" case should be fixed to extract and parse spec from payload instead of just looking at header. (It should be fixed independently of this ticket.)

  2. "dnf builddep foo" could theoretically be made to download SRPM (or part of it, if signature verification is not performed) from repos, but I guess that this is a rare use case and it can be either left as-is (broken) or deprecated.

  3. YUM can be left as-is (it will be replaced with DNF soon enough)

  4. Koschei can be quite easily fixed to resolve build-requires from appropriate arch repository, under one condition: Koji stops building SRPMs on architectures it doesn't generate repositories for (currently just PPC). Otherwise some SRPMs in official Fedora repos could have requires with PPC %_isa which wouldn't be resolvable against any primary-arch repository. (Rebuilding SRPM by Koschei on one of primary arches would require a lot of changes and it is impractical. Using repos from secondary-arches is out of question.)

If all tools are fixed to work with %_isa in BuildRequires (there may be others besides the ones mentioned above) then it should be OK to accept modified proposal (with additional %ifarch/%ifnarch requirement).

Replying to [comment:15 mizdebsk]:

Some thoughts:

  1. "dnf builddep foo.src.rpm" case should be fixed to extract and parse spec from payload instead of just looking at header. (It should be fixed independently of this ticket.)

  2. "dnf builddep foo" could theoretically be made to download SRPM (or part of it, if signature verification is not performed) from repos, but I guess that this is a rare use case and it can be either left as-is (broken) or deprecated.

  3. YUM can be left as-is (it will be replaced with DNF soon enough)

  4. Koschei can be quite easily fixed to resolve build-requires from appropriate arch repository, under one condition: Koji stops building SRPMs on architectures it doesn't generate repositories for (currently just PPC). Otherwise some SRPMs in official Fedora repos could have requires with PPC %_isa which wouldn't be resolvable against any primary-arch repository. (Rebuilding SRPM by Koschei on one of primary arches would require a lot of changes and it is impractical. Using repos from secondary-arches is out of question.)

koji does not work like you describe. It builds the srpm in a target OS chroot for the
buildSRPMFromSCM task, it then inits a minimal buildroot on each target arch and rebuilds the srpm for each arch to make sure build deps are correct, the srpm that ends up in the tree is one from the buildArch tasks. There is no way that a SRPM can be built and shipped on an arch that the rpms are not built for.

If all tools are fixed to work with %_isa in BuildRequires (there may be others besides the ones mentioned above) then it should be OK to accept modified proposal (with additional %ifarch/%ifnarch requirement).

Replying to [comment:16 ausil]:

koji does not work like you describe. It builds the srpm in a target OS chroot for the
buildSRPMFromSCM task, it then inits a minimal buildroot on each target arch and rebuilds the srpm for each arch to make sure build deps are correct, the srpm that ends up in the tree is one from the buildArch tasks. There is no way that a SRPM can be built and shipped on an arch that the rpms are not built for.

That's great - no Koji changes should be required to fix Koschei to work with %_isa in BR.

So this is sort of floating around at the bottom of our schedule and honestly I'm not sure what FPC can do at this point. What remains to be discussed? Does anyone have a proposal upon which we can vote?

I guess I'm going to go ahead and close this. If someone has something else to add or wants to move this forward, please feel free to reopen.

Metadata Update from @mschwendt:
- Issue assigned to tibbs

2 years ago

Login to comment on this ticket.

Metadata