#2096 F31 System-Wide Change: BuildRequires generators
Closed: Accepted 7 months ago by bowlofeggs. Opened 9 months ago by bcotton.

Add possibility to generate build-time dependencies within RPM spec file and teach RPM and mock how to handle this.


There needs to be an upstream rpm release with this feature included, we
do not do feature backports in Fedora rpm. As of now, this is only being
discussed at upstream.

This is something to consider, however I'm +1 for allowing this to happen. I guess there is always a chance that that a change will not happen in time.

I am +1. I think it's OK to approve it even if the upstream doesn't have it yet. If it doesn't get in in time, we can always postpone to a later Fedora release.

+1 also, we do need to keep track of where upstream is here (like the other rpm change)

Metadata Update from @zbyszek:
- Issue tagged with: meeting

8 months ago

+1

[ Random note: if this is enabled for Python, I'm guessing we're going to have quite a few circular build dependencies, because build/runtime dependencies aren't distinguished. And while circular build dependencies don't typically matter, they do for bootstrapping, and for automatically determining build ordering. But if the facility is available it could be enabled for other languages and not Python if that turned out to be a problem. ]

For now, my ultimate goal is to have it just for Rust ecosystem. In future we might want to use it for other languages / ecosystems, but that is definitely not in scope for this change. Even Rust generators are not in this scope.

So basically Python projects usually only have very few build dependencies (setuptools, numpy, cython). It's test dependdencies and we could have a higher level macro that controls whether you want them or not (depending on %with check) etc. as we already need to bootstrap such deps manually.

This was discussed in today's meeting. We didn't reach a formal conclusion, because there are still questions about the details of the proposal. In particular contyk said he'll post some questions.

Metadata Update from @zbyszek:
- Issue untagged with: meeting

8 months ago

A generator code is already written and used out of Fedora for Go, it is only waiting for the rpm part to be used within Fedora. (a second-gen Go BR generator is being written to handle Go modules, including a form of BR version constrains

After 10 days, I count the vote as (+5,0,-0). Are @psabata's questions answered, or should I delay processing this proposal as approved?

@contyk: please post your questions now, or remain forever silent.

My main worry is we won't have the generated dependencies included in the SRPMs, which might make analysis and generating proper build order rather complicated. The change proposal includes a bunch of unknowns.

To me, voting on this proposal at this stage feels rather rushed. If the proposal owners don't think it's worth working on the code unless we approve it right now, it probably isn't.

Metadata Update from @psabata:
- Issue tagged with: meeting

8 months ago

I'd just like to not end up in the situation where we've implement something in upstream and then have FESCo said that we don't like this and that. I would prefer to hear all concerns and requirements in advance and make implementation to account for this.

I know that it should have been sent to devel ML for such discussion, but unfortunately such topics do not lead to getting such answers.


Now to your point about dependencies in SRPMs. I think it should be possible to add some flag to dependencies which would mean that those are autogenerated and then flag in rpmbuild/mock which would regenerate (or not) those dependencies. This way we can still inspect what dependencies have been used. Definitely this raises question about arch-dependent things..

Even today, you don't have correct dependencies in SRPMs if one is using %ifarch construction for BuildRequires. Because we ship just one SRPM instead of one per architecture.

And it is not only about ifarch, but dependencies can be injected depending on other RPM variables and even depending on file/time/whatsoever.


So what I would like to do from FESCo is list of concerns, I can propose ways to resolve them.

Have you considered building this tooling to be a part of packager toolbox (like rpmlint, rpmbuild new-spec or smth like this). You can build the autogenerator for packagers to run locally. Its output then is going to be a template for a spec file or recommended list of requirements.

This will create the initial userbase which is going to provide feedback, verify corner cases and so on. This userbase will then provide the source of concerns and requirements, which will give us the better ground to decide whether it is worth it to built it in the internals of the rpmbuild process.

And i don't really get your reason not to have a mailing list discussion for it.

Have you considered building this tooling to be a part of packager toolbox (like rpmlint, rpmbuild new-spec or smth like this). You can build the autogenerator for packagers to run locally. Its output then is going to be a template for a spec file or recommended list of requirements.

Well, we do that for long long time. But if you need to change something in generator, you have to re-do all the spec files instead of just changing code in generator and rebuild all affected packages (without changing any single line in them).

See rust2rpm, pyp2rpm, gem2rpm and more tooling which generates spec files out of upstream metadata.

See rust2rpm, pyp2rpm, gem2rpm and more tooling which generates spec files out of upstream metadata.

Is it possible to collect the data on how much actual rpm specs differ from those suggested by tools?

But if you need to change something in generator, you have to re-do all the spec files instead of just changing code in generator and rebuild all affected packages (without changing any single line in them).

Do you propose policy on mass rebuilds as a part of this Change? On which events and which requirements?

Is it possible to collect the data on how much actual rpm specs differ from those suggested by tools?

All rust-* packages (around 550 at this moment) are generated using rust2rpm. Then they are processed to remove some subpackages, fixup description/summary and such small tweaks. If dependencies in the spec file are different from what rust2rpm puts there, 99% package won't build.

Do you propose policy on mass rebuilds as a part of this Change? On which events and which requirements?

Best judgement of generator' maintainers. For example, if I change rust2rpm generator and I consider it being important for rebuild -- I'll just go and rebuild affected packages in rawhide.

Best judgement of generator' maintainers. For example, if I change rust2rpm generator and I consider it being important for rebuild -- I'll just go and rebuild affected packages in rawhide.

This part is actually not different from today. If you change dependency generators (which exist for last 10+ years), maintainer needs to decide whether to rebuild and what.


Speaking of which, https://pagure.io/fedora-rust/rust2rpm/issue/70 issue showed up today. We probably will need to change how dependencies are constructed. But that basically means we have to regenerate all spec files.

Have you considered building this tooling to be a part of packager toolbox (like rpmlint, rpmbuild new-spec or smth like this). You can build the autogenerator for packagers to run locally. Its output then is going to be a template for a spec file or recommended list of requirements.

That has been tried many times for many languages and it does not scale nor works long term because the

upstream artefact → generated spec → build

workflow is really a

upstream artefact → generated spec → packager adjusted spec → build

So unless you are so lucky your upstream always generates perfect artefacts you quickly end up with a mass of specs that can not be regenerated automatically without human review, and this is too resource intensive, the result slowly rots, resulting in an unmanageable pile of different style specs generated at different times with different tooling versions.

And, it never worked at all for the large class of projects that mix several kinds or artefacts within a single archive.

To keep something maintenable you need to split the generating functions in atoms and compose them within your spec like everything else.

That means either integrating the generators in rpm to keep rpm our primary tool, or add a templating layer over rpm to work around the facts you can not compose the generators at the rpm level (which means that, basically, your packager tool is no longer rpm but something that wraps it).

Switching the distro from plain rpm to something else over rpm would be a lot more dangerous and invasive than integrating the generators within rpm in the first place. It would imply redoing all out tooling from the ground up to take into account the wrapping template as primary package descriptor, not the spec file.

And yes you can avoid that by pretending the generation does not happen. What that means concretely is that you have large part of the package maintenance which is no longer tracked and audited by Fedora tools.

My main worry is we won't have the generated dependencies included in the SRPMs, which might make analysis and generating proper build order rather complicated.

Right now a SRPM is created as a build artifact in koji. How does this change with the proposed change? IIUC, there are two SRPMs — the original one, and the one augmented by generated BuildRequires. Will both be visible in koji? The change page says "TBD", but this is a rather important bit.

I don't know if we need both SRPMs to be kept. Maybe it it is enough if the generated list of BR is kept as a file or something.

1) let's limit the scope to Rust alone;
2) figure out how to save build requirements in the srpm produced;
3) define the impact on dnf;
4) define the impact on local developer who tries to rebuild the package on a different environment.

The reproducibility issue is covered in the proposal in a wrong way. If i reproduce build environment to every single bit, i get the same result. But that's not the point of a reproducibility. Or maybe better to say, you have reproducibility, but no transparency, no possiility to build on that.

I want to rebuild a package on my Fedora system, or my CentOS system, and I need to easily track dependencies i need to install and how these are changed. I don't have a task to reproduce bit by bit, I have a task to change in a controlled way.

1) let's limit the scope to Rust alone;

Python wants this too...

@churchyard I am not saying we should never try it for Python. But it was mentioned in the comments above that the first target is going to be Rust. So let's start with that and proceed step by step.

Especially since @otaylor mentioned that Python might have certain problem with this approach.

FESCo likes the idea in general, but will wait for a specification to review before accepting for use in Fedora. We will be happy to answer questions in ticket along the way (+8,0,-0)

Metadata Update from @jforbes:
- Issue untagged with: meeting

8 months ago

1) let's limit the scope to Rust alone;

This idea is to have a generic mechanism that works for all and any language. It is possible that only rust might use this in the beginning, but the change on the rpm/mock/koji side is not tied to any specific language stack.

2) figure out how to save build requirements in the srpm produced;

+1, this needs to be figured out.

3) define the impact on dnf;

There shouldn't be any. In this proposal (as opposed to the other pm_requests one), dnf is simply invoked another time with an additional list of deps to satisfy. It's the higher-level components that orchestrate this.

4) define the impact on local developer who tries to rebuild the package on a different environment.

+1. If 2) produced a "full" srpm, then this will be solved automatically.

The reproducibility issue is covered in the proposal in a wrong way. If i reproduce build environment to every single bit, i get the same result. But that's not the point of a reproducibility. Or maybe better to say, you have reproducibility, but no transparency, no possiility to build on that.
I want to rebuild a package on my Fedora system, or my CentOS system, and I need to easily track dependencies i need to install and how these are changed. I don't have a task to reproduce bit by bit, I have a task to change in a controlled way.

2) should also solve those reproducibility/transparency issues.

This idea is to have a generic mechanism that works for all and any language. It is possible that only rust might use this in the beginning, but the change on the rpm/mock/koji side is not tied to any specific language stack.

I have a feeling that we are mixing the generic effort on the RPM side and the change proposed to Fedora 31 here. And it also needs to be specifically explained in the proposal.

So the generic direction, the effort is "to enable Build Requires generator for everything".
Interesting idea, worth looking into, we agreed on that.

But what is the actual proposal for Fedora 31?

Is it "to update RPM to the upstream version which supports this functionality"?
Or "to enable this feature for all packages on Fedora Koji service"?
Or "enable it in MBS for Rust module"?

If it works for Rust but doesn't work for Python yet, should Change for F31 be declared Done or Deferred?

to technically enable this feature for all packages - enable, not enforce.

IMHO the change doesn't say "we will mass change our packages to use it" it says "we will make it possible for packages to use it".

So, assume i have three packages which, when enabled with this feature, get their BuildRequires right, and then I have 3000 packages, for which, if I enable this feature, it ends as garbage.

Is this change done or not?

Note, I don't have my own final opinion on that yet. But I believe we need to have this answered and this answer recorded in the proposal to move forward.

I don't think we are on the same page. This change doesn't calculate the buildrequires, it just gives you the opportunity to do so. You can write your own Perl script that outputs the package names. This change cannot turn packages into garbage, it only gives you the tool to do whatever you want.

1) let's limit the scope to Rust alone;

That would be massively unfriendly to all the Fedora contributors, that worked on this, in different SIGs. Just because a member of the rust SIG got sufficiently fed up with the lack of advancement to write the actual change request, does not mean rust people are the only ones who have been working or waiting on this.

So, assume i have three packages which, when enabled with this feature, get their BuildRequires right, and then I have 3000 packages, for which, if I enable this feature, it ends as garbage.

Then you do not enable it in the problem specs (you need to add the autobr call line to a spec to enable it). Or you add a | grep -v "bad BR" to the specs where a generator generates a problematic BuildRequires. It's no harder than that.

The change is a generic rpm mecanism where rpm accepts the output of spec commands as a BR list. If you don't like the BRs produced by those other commands you can replace them of massage their output the usual spec way.

1) let's limit the scope to Rust alone;

That would be massively unfriendly to all the Fedora contributors, that worked on this, in different SIGs. Just because a member of the rust SIG got sufficiently fed up with the lack of advancement to write the actual change request, does not mean rust people are the only ones who have been working or waiting on this.

Sorry, I don't get it.

We are discussing the proposed Change. Change Owners present it and scope it.

I am not trying to enforce additional limitations on the scope from my side here, I am trying to make sure that the change as it is proposed, and the change as it is going to be delivered are the same thing. Thus I am relying on Change Owner's comment here that the focus "for now" is going to be Rust.

If there other contributors who will cover other areas, I suggest you get onboard. Join forces, communicate with current Change Owners and add yourself to co-Owners. Or convince current Owners that support for other languages need to be in the focus as well and that you are going to help them.

In the end it is the Change Owner who supposed to be responsible for the result. We can not add tasks on them without them accepting it.

Well, change on its own is about implementing generic way of supplying BR generators and making sure it works fine and is supported by infrastructure. That's it.

Once this is there, I'll obviously implement such generators for Rust.

I don't think we are on the same page.

Actually yes, because there is no page. I am guessing and in a wrong way it seems.

This change doesn't calculate the buildrequires, it just gives you the opportunity to do so. You can write your own Perl script that outputs the package names. This change cannot turn packages into garbage, it only gives you the tool to do whatever you want.

So let me try again and check if I get it right this time:

1) there is going to be change in the rpm in the upstream which provides a way for packager to replace hardcoded list of Build Requires with a list of autogenerated ones.

Macro?

2) there will be release in upstream of several plugins (per language) which can be added to rpm to provide the implementation on how to autogenerate build deps in each case

3) Fedora Infra would need to update to this RPM version and install plugins in the build environment

4) Fedora Packaging Guidelines need to allow this kind of macro in the spec file

1) let's limit the scope to Rust alone;

See above. Change is about generic infrastructure in RPM which allows generating BRs and making sure it is supported by infra.

2) figure out how to save build requirements in the srpm produced;

What do you think about:

  • When rpmbuild is run, in phase %prep (or somewhere between %prep and %build) it generates BuildRequires as per configured BR generators. rpm itself produces new SRPM which contains original BRs + generated ones (annotated with some flag).
  • Then, rpmbuild will gain new option (something like --reuse-buildrequires) which controls whether it will regenerate all BRs or will use generated in previous step to perform checks.

This way we can ship RPMs which still have static BuildRequires which are sufficient to build RPM, but still when generators change their output, have possibility to regenerate BRs from the scratch.

3) define the impact on dnf;

No impact.

1) there is going to be change in the rpm in the upstream which provides a way for packager to replace hardcoded list of Build Requires with a list of autogenerated ones.

Yes.

Macro?

Hopefully not. Rather how dependency generators works today in RPM. They specify some file patterns on which some script is run.

2) there will be release in upstream of several plugins (per language) which can be added to rpm to provide the implementation on how to autogenerate build deps in each case

Yes, main functionality will go into RPM. But it should have enough flexibility so that there is no hardcoding in RPM for language-specific stuff.

https://rpm.org/user_doc/dependency_generators.html

3) Fedora Infra would need to update to this RPM version and install plugins in the build environment

Fedora Infra would need to upgrade to new release of mock which supports necessary functionality. Nothing additional goes into default buildroot.

4) Fedora Packaging Guidelines need to allow this kind of macro in the spec file

FPG doesn't prohibit any macro. But yes, FPG needs to gain some page which describes how to use dependency generators for BuildRequires. I'm willing to work on it (putting my Packaging Committee hat on).

We are discussing the proposed Change. Change Owners present it and scope it.

Then respect the change owner scoping. If the change owner wanted to limit it to rust, he would have written it so (and all that would have resulted in is cut and pasting of the change with rust replaced by other languages, and FESCO asking to consolidate everything in a single ticket).

The "let's limit to it rust" thing was added by FESCO during FESCO discussions, based on a deliberate misinterpretation and expansion of a single sentence in the change page.

Then respect the change owner scoping.

I do. I am still allowed to suggest things though. If these suggestions don't make sense, we discard them.

If the change owner wanted to limit it to rust, he would have written it so (and all that would have resulted in is cut and pasting of the change with rust replaced by other languages, and FESCO asking to consolidate everything in a single ticket).

The description of a change is mostly non-existent. Thus it is hard to argue about what is expected and what is not. Don't take me as an enemy here, I am trying to ask questions so that the proposal can be improved by adding answers to them.

"Let's limit to Rust" - "No, because.." - is a good answer too. Let's move on.

1) there is going to be change in the rpm in the upstream which provides a way for packager to replace hardcoded list of Build Requires with a list of autogenerated ones.

Yes.

rpmbuild will accept at a specific point of the spec (before %build, after %prep) the output of commands as a list of BuildRequires

Macro?

There are no constrains of the kind of commands rpmbuild will accept the output of (upstream tool, fedora wrapper, package-specific script, macro, whatever).
echo foo will cause foo to be installed if written in the correct part of the spec.

If upstream is nice enough to provide a command that produces buildrequires lists in a syntax rpm understands, it is likely to be used as-is (our current font provides, for example, use raw fontconfig output). The most likely case is something in between: upstream provides a build requires tool, and the output of this tool needs to be translated into rpm syntax at the rpm, distro, or individual packager level.

2) there will be release in upstream of several plugins (per language) which can be added to rpm to provide the implementation on how to autogenerate build deps in each case

I sort of doubt that will happen, people do not need rpm upstream help to write commands that output lists of things, and rpm upstream is not terribly interested in baby-sitting the writing of lots of domain-specific commands (or just inventorying them). If the rpm world was organized this way most of redhat-rpm-config would have been merged into the rpm project long ago.

Plus, the actual generator commands will depend on the software stack implementations each distribution ships. If python 3.x provides the command generator, and python 4.z the command bettergenerator, rpm upstream will just expect distributions to write generator or bettergenerator in their specs, depending on their python level target. (or write a python-generator macro that warps the correct generator depending on the result of distro-specific processing).

rpm upstream will only provide the facility to feed BR lists to the build env at a specified point of the spec, and distributions and packagers will do whatever they want with it (just like rpm upstream does not have any opinion, on the commands you put in %build or %install)

Yes, main functionality will go into RPM. But it should have enough flexibility so that there is no hardcoding in RPM for language-specific stuff.
https://rpm.org/user_doc/dependency_generators.html

The main difference I see with the existing dependency generators, is that "stuff" at the end of %prep is not nicely filtered and put in standard FHS places, so I doubt a fileattr mecanism could work without lots of efforts to restructure sources in %prep.

It's probably simpler and more efficient to just ask the packager to add explicit calls to the magic generator commands he needs in his spec.

3) Fedora Infra would need to update to this RPM version and install plugins in the build environment

Fedora Infra would need to upgrade to new release of mock which supports necessary functionality.

and use a rpmbuild version that includes the feature

Nothing additional goes into default buildroot.

The default buildroot is likely to gain more foo-srpm-macros packages to define a convenient wrapping of upstream generator commands. It is likely to shed more "just in case it's needed" parts because the feature permits finer control of buildrequires, on a per-srpm level.

4) Fedora Packaging Guidelines need to allow this kind of macro in the spec file

Fedora Packaging Guidelines would need to provide guidance on what generator commands to use in a rust project, in a golang project, a python project, and so on.

Nothing different from the guidance on the commands to use in other parts of a Fedora spec, with the same domain specificity, and the same maintenance as upstream commands evolve and are wrapped (or not) in Fedora macros.

@nim

rpmbuild will accept at a specific point of the spec (before %build, after %prep) the output of commands as a list of BuildRequires

Afaik, BuildRequires are already used by %prep stage

We have just recently had couple of issues with that, when python3-devel or autotools were required for %prep stage to finish.

So inserting new set of Build Required between steps makes it two different types of Build Requires (those in %prep and those in %build)

I wonder if there should be explicitly defined PrepRequires then? BuildRequires(prep)?

2) there will be release in upstream of several plugins (per language) which can be added to rpm to provide the implementation on how to autogenerate build deps in each case

(...)

rpm upstream will just expect distributions to write generator or bettergenerator in their specs, depending on their python level target. (or write a python-generator macro that warps the correct generator depending on the result of distro-specific processing).

So, should this also be done in this Change? Or should it be just an ongoing effort happening at its own pace?

The main difference I see with the existing dependency generators, is that "stuff" at the end of %prep is not nicely filtered and put in standard FHS places, so I doubt a fileattr mecanism could work without lots of efforts to restructure sources in %prep.
It's probably simpler and more efficient to just ask the packager to add explicit calls to the magic generator commands he needs in his spec.

Feels like a new %configure section in RPM spec.

rpmbuild will accept at a specific point of the spec (before %build, after %prep) the output of commands as a list of BuildRequires

Afaik, BuildRequires are already used by %prep stage

The change provides an additional BuildRequires entry point, the existing static one is unchanged. Static BuildRequires still exist, and you need them to cover at least the bits needed for %prep execution.

The build process becomes

instantiate cached default buildroot
install buildroot updates
install static buildrequires
unpack sources and execute %prep
execute dynamic buildrequires generators
install computed dynamic buildrequires
execute %build…

I wonder if there should be explicitly defined PrepRequires then? BuildRequires(prep)?

The proposed system adds the equivalent dynamic capability to BuildRequires, that already exists for %files.

The dynamic capability of %files is provided by the -f flag. A %files section can contain both static entries, and dynamic entries provided in the -f list. Both the static and dynamic entries use the same verbs and syntax, no need to invent a separate one.

For BuildRequires there is no existing %buildrequires section to hang a -f on, so list feeding just occurs at a specific point of the spec (a file containing a list and the output of commands are pretty much equivalent, you can transform files in output with cat and output in files with >>)

2) there will be release in upstream of several plugins (per language) which can be added to rpm to provide the implementation on how to autogenerate build deps in each case

(...)
rpm upstream will just expect distributions to write generator or bettergenerator in their specs, depending on their python level target. (or write a python-generator macro that warps the correct generator depending on the result of distro-specific processing).

So, should this also be done in this Change? Or should it be just an ongoing effort happening at its own pace?

As soon of the capability is available in rpm and mock Fedora packagers will start experimenting with it.

The recommended way to make use of it in specs will go through FPC like the rest of our spec stuff, probably domain by domain, as soon as someone bothers to write a nice domain application packaging guideline, in the usual iterative ongoing way.

Since several SIGs have been working on this and are waiting for the rpm/mock part to be finished, I expect several packaging guideline proposals will happen quickly after the feature is deployed infra-side (as soon as FPC has a build infra with the feature to evaluate proposals on).

You can simulate a big part of the feature with pm_request (not secure enough to deploy public infra side, but secure enough to run on your own private systems). Therefore its completion is not blocking early generator work.

And, regardless of what FPC decides, you will have some uses of the feature in Fedora specs not covered by guidelines, because FPC only has the time and energy to write guidelines for common needs, and less common needs are left to individual packager discretion and good sense.

Feels like a new %configure section in RPM spec.

Not really, configure allows restricting a build to things detected within the build root. This proposal has the inverse effect: expanding the build root to things that we realize are needed once sources have been expanded and inspected.

The proposed system adds the equivalent dynamic capability to BuildRequires, that already exists for %files.

There is a difference though. BuildRequires changes the flow of the build process.

Consider the autoreconf command. It makes decisions based on what is available in the build environment. Currently there is no difference between putting it in %prep or %build stage, but with update of BuildRequires hooked in between, it makes it a big deal.

And then approach by @ignatenkobrain wouldn't work, because "new spec", with added Build Requires will not reproduce the build process for package in the same way.

Essentially, listing the same file explicitly or dynamically produces the same result. But adding the same build requirement statically or dynamically would produce different packages.

The proposed system adds the equivalent dynamic capability to BuildRequires, that already exists for %files.

There is a difference though. BuildRequires changes the flow of the build process.
Consider the autoreconf command. It makes decisions based on what is available in the build environment. Currently there is no difference between putting it in %prep or %build stage, but with update of BuildRequires hooked in between, it makes it a big deal.

autoreconf generates files from sources. That's pretty much the definition of something that belongs in %build. rpm is very lenient and lets you execute commands in any order in any section. So you can get by with misplacing things a little. But, that will generally end up in woe later.

That happened to me recently. I didn't think hard if an action was of a prep build or install nature. I put it in build. I worked fine most of the times, except in special cases. I spent a few months piling up workarounds, before thinking hard and realising I was fighting rpm. And I was fighting rpm because I was doing preparation actions too late in build. Just moving the action in the right section removed the need for workarounds and simplified my code a lot.

Point being, that even if it does not look like it, correct action ordering is very important for reliable rpmbuild execution. Using the correct section matters. The implicit design rpmbuild enforces relies on them.

Essentially, listing the same file explicitly or dynamically produces the same result.

Only if you do not use things like %defattr. If you do, ordering matters. A lot of things in specs have ordering constrains that only trigger in specific cases.

We have the proposal in https://pagure.io/fesco/issue/2096#comment-561887. I think it's reasonable, but it needs putting in the Change page and fleshing out.

We've had quick (not really) call today with @ngompa and @ffesti. We've agreed that final SRPM will include auto-generated BuildRequires, so we are good with this. We're still fleshing out details about rpmbuild flags and some workflow though.

Also to clarify a bit more details, there will be special section in spec file where packagers will put scripts/macro to generate BRs. Same here, we're choosing name (one of those hard problems in programming).

I'll update change proposal in week or two.

Given https://pagure.io/fesco/issue/2096#comment-562909 I am taking this off of the agenda until the proposal is updated. Thanks very much @ignatenkobrain @ngompa and @ffesti

@bcotton @sgallagh @bookwar @psabata I have renamed change proposal to "DynamicBuildRequires" so that it better matches upstream name.

Also I have updated wiki page which hopefully resolves all concerns along with POC example.

https://fedoraproject.org/wiki/Changes/DynamicBuildRequires

Thanks @ignatenkobrain

I think we still need releng issue https://pagure.io/releng/issue/8129 to be clarified.

And a question: if I understand correctly the srpm produced with the feature enabled will contain all BuildRequires( this is needed for tools like repoquery to work). Will I be able to recover original spec file from this srpm?

Looks great. I'm still +1.

I'm confused by the ".nosrc.rpm" extension though — it would imply that that srpm does not include sources, but in the transcript, sources are unpacked from it.

Also, to clarify: in koji builds, do we keep the original "lean" src.rpm, or the second src.rpm with rpmlib(DynamicBuildRequires), or both?

And a question: if I understand correctly the srpm produced with the feature enabled will contain all BuildRequires( this is needed for tools like repoquery to work). Will I be able to recover original spec file from this srpm?

Yes. Spec file are not modified.

I'm confused by the ".nosrc.rpm" extension though — it would imply that that srpm does not include sources, but in the transcript, sources are unpacked from it.

buildreqs.nosrc.rpm is just temporary RPM which is generated just to be able to use dnf builddep. There are no sources, probably neither even spec (but I didn't check).

Also, to clarify: in koji builds, do we keep the original "lean" src.rpm, or the second src.rpm with rpmlib(DynamicBuildRequires), or both?

In koji builds, we will store only one src.rpm. The one which contains all dynamic BuildRequires.

Funny Python story: We would need to do this twice (there is a two-phase dependencies check - the first gives you a tool like setuptools or flit and the second uses this tool to get stuff like Cython). I suppose we need to take that to RPM upstream, right?

Nope, RPM upstream already is capable of doing that... It is just need some implementation on mock side... Which was not trivial, so I didn't implement it.

Also, I disagree that there is no impact on dnf and on end user.

I understand that it doesn't change the basics of how packages are managed. And I guess there are no parts which are explicitly blocking for this change on dnf side.

But

1) we create confusion for regular users and dnf is going to be the first point of contact for it
2) there might be feature requests to support new behavior from UX side of dnf. For example requests for additional repoquery flags.
3) For things like dnf builddepthe impact is unclear, but at the very least the doc [1] needs to be updated

So I'd suggest to stop declaring that this feature has no visible effect, rather embrace the fact that it has, and look into what can be done to make it visible for end users in a proper way.

And by users we shouldn't mean just fedora desktop people. There are tools like rpmdeplint, rpmdiff (aka rpm inspect), there are rpm templating engines, like renderspec [2] and other consumers who build their infrastructure on top of this tooling.

I am not saying we should explicitly deal with each one of them, and block on them, but let's think about them as users too and look into this topic instead of discarding it right away.

[1] https://dnf-plugins-core.readthedocs.io/en/latest/builddep.html
[2] https://github.com/openstack/renderspec

1) we create confusion for regular users and dnf is going to be the first point of contact for it

What kind of confusion?

2) there might be feature requests to support new behavior from UX side of dnf. For example requests for additional repoquery flags.

In that case, it won't be only DNF because you will need to propagate more data through rpm-md.. That means you will need libsolv, createrepo_c, libdnf changes at the very least.

3) For things like dnf builddepthe impact is unclear, but at the very least the doc [1] needs to be updated

As I said multiple times, dnf builddep is not affected. There is absolutely no changes needed.

So I'd suggest to stop declaring that this feature has no visible effect, rather embrace the fact that it has, and look into what can be done to make it visible for end users in a proper way.

Do you have proposal what I should put in the wiki page?

What kind of confusion?
...
As I said multiple times, dnf builddep is not affected. There is absolutely no changes needed.

Let's consider the question: What does the command dnf builddep foobar.spec do?

With the change enabled, the line "Install the needed build requirements, defined in the foobar.spec file" is confusing. And adding couple of lines to explain it will help.

(UPDATE: Also adding a flag to install initial BRs only without dynamic ones could be a valid feature request. I already have a use case for it.)

Now how is it going to work with distro-specific macro definitions? Would I need to install them beforehand? Would it be dependency for dnf plugin? Will dnf plugin show a proper error message in case it can not find the dependency generator required by this spec file and in case it failed by some reason?

Btw, is there an impact on fedora-review? rpmlint?

Do you have proposal what I should put in the wiki page?

Will dynamic_buildrequires for mock be enabled by default? If yes, then it is a user impact, if no - then it is also user impact as user needs to enable it to build certain packages.

Will it be required to explicitly install distro-specific language-specific dependency generators before building packages or running dnf builddep? Then it is a user impact as well.

New section in rpm spec also can be considered a user impact but this one can be skipped i guess.

Let's consider the question: What does the command dnf builddep foobar.spec do?

Installs BuildRequires specified in foobar.spec (not the dynamic ones).

Now how is it going to work with distro-specific macro definitions?

Would I need to install them beforehand?

Well, it is not different from today, if you want some macro expanded in spec, you need to install some package which is providing it. So, imagine case with Rust: you need following lines in spec:

%generate_buildrequires
%cargo_generate_buildrequires

You need to have rust-packaging installed to expand that macro, so you have 2 options:

BuildRequires: rust-packaging
%generate_buildrequires
%cargo_generate_buildrequires

or

%generate_buildrequires
%if %{defined cargo_generate_buildrequires}
%cargo_generate_buildrequires
%else
echo "rust-packaging"
%endif

Both are valid.

Would it be dependency for dnf plugin?

No.

Will dnf plugin show a proper error message in case it can not find the dependency generator required by this spec file and in case it failed by some reason?

DNF doesn't touch dynamic BuildRequires.

Btw, is there an impact on fedora-review? rpmlint?

fedora-review is using mock, so I doubt. rpmlint should not be affected. Both obviously can grow some features to deal with dynamic BuildRequires better or do some additional checks, but they will keep working as is and there is no loss in functionality.

Will dynamic_buildrequires for mock be enabled by default? If yes, then it is a user impact, if no - then it is also user impact as user needs to enable it to build certain packages.

Yes, for f31+ chroots only.

Will it be required to explicitly install distro-specific language-specific dependency generators before building packages or running dnf builddep? Then it is a user impact as well.

As I said above, this is not changing at all. Packagers have today them in BuildRequires. Tomorrow they will have to run dnf builddep multiple times if rpmbuild aborts.

(UPDATE: Also adding a flag to install initial BRs only without dynamic ones could be a valid feature request. I already have a use case for it.)

Please share it. But anyway, dnf builddep installs only manually-specified BuildRequires.

But anyway, dnf builddep installs only manually-specified BuildRequires.

Wait, but then it is a bug, and an impact. And actually a blocking one, because it heavily affects the user workflow.

Wait, but then it is a bug, and an impact. And actually a blocking one, because it heavily affects the user workflow.

If it affects user workflow, it is not blocking bug. Many changes are affecting user workflows. This is just one of them.

What do you suggest DNF to do? Run (potentially) untrusted code on user' machines? It is job of mock (or any other system which is orchestrating build of RPM packages).

⋊> ~/P/s/r/rust-rpick on master ⨯ sudo dnf builddep rust-rpick.spec     13:20:02
Last metadata expiration check: 0:00:47 ago on Fri 12 Apr 2019 01:19:22 PM CEST.
Package rust-packaging-6-28.fc31+buildreqs.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
⋊> ~/P/s/r/rust-rpick on master ⨯ fedpkg local                          13:20:10

Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.Ydztwl
+ umask 022
+ cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick
+ cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick
+ rm -rf rpick-0.3.0
+ /usr/bin/gzip -dc /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick/rpick-0.3.0.crate
+ /usr/bin/tar -xof -
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd rpick-0.3.0
+ /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ set -eu
+ /usr/bin/mkdir -p .cargo
+ cat
+ /usr/bin/rm -f Cargo.lock
+ exit 0
Executing(buildreqs): /bin/sh -e /var/tmp/rpm-tmp.SckKAl
+ umask 022
+ cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick
+ cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick
+ cd rpick-0.3.0
+ /usr/bin/cargo-inspector -BR Cargo.toml
+ /usr/bin/cargo-inspector -TR Cargo.toml
+ exit 0
error: Failed build dependencies:
    (crate(dirs/default) >= 1.0.0 with crate(dirs/default) < 2.0.0) is needed by rust-rpick-0.3.0-1.fc31.x86_64
    (crate(rand/default) >= 0.6.0 with crate(rand/default) < 0.7.0) is needed by rust-rpick-0.3.0-1.fc31.x86_64
    (crate(serde/default) >= 1.0.0 with crate(serde/default) < 2.0.0) is needed by rust-rpick-0.3.0-1.fc31.x86_64
    (crate(serde/derive) >= 1.0.0 with crate(serde/derive) < 2.0.0) is needed by rust-rpick-0.3.0-1.fc31.x86_64
    (crate(serde_yaml/default) >= 0.8.0 with crate(serde_yaml/default) < 0.9.0) is needed by rust-rpick-0.3.0-1.fc31.x86_64
    (crate(structopt/default) >= 0.2.0 with crate(structopt/default) < 0.3.0) is needed by rust-rpick-0.3.0-1.fc31.x86_64
Wrote: /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick/rust-rpick-0.3.0-1.fc31.buildreqs.nosrc.rpm
Could not execute local: rpmbuild --define '_sourcedir /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick' --define '_specdir /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick' --define '_builddir /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick' --define '_srcrpmdir /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick' --define '_rpmdir /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick' --define 'dist .fc31' --define 'fedora 31' --eval '%undefine rhel' --define 'fc31 1' -ba /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick/rust-rpick.spec | tee .build-0.3.0-1.fc31.log
⋊> ~/P/s/r/rust-rpick on master ⨯ 
sudo dnf builddep /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick/rust-rpick-0.3.0-1.fc31.buildreqs.nosrc.rpm
Last metadata expiration check: 0:01:01 ago on Fri 12 Apr 2019 01:19:22 PM CEST.
Package rust-packaging-6-28.fc31+buildreqs.x86_64 is already installed.
Dependencies resolved.
================================================================================
 Package                                 Arch   Version           Repo     Size
================================================================================
Installing:
 rust-dirs+default-devel                 noarch 1.0.5-1.fc31      rawhide 7.6 k
 rust-rand+default-devel                 noarch 0.6.5-3.fc31      rawhide 8.4 k
 rust-serde+default-devel                noarch 1.0.90-1.fc31     rawhide  12 k
 rust-serde+derive-devel                 noarch 1.0.90-1.fc31     rawhide  12 k
 rust-serde_yaml+default-devel           noarch 0.8.8-2.fc30      rawhide 8.3 k
 rust-structopt+default-devel            noarch 0.2.15-2.fc31     rawhide 8.2 k
Installing dependencies:
[…]
Transaction Summary
================================================================================
Install  87 Packages

Total download size: 2.0 M
Installed size: 8.0 M
Is this ok [y/N]: y
[…]
Complete!
⋊> ~/P/s/r/rust-rpick on master ⨯ fedpkg local                          13:20:36

Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.PmWhFa
+ umask 022
+ cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick
+ cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick
+ rm -rf rpick-0.3.0
+ /usr/bin/gzip -dc /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick/rpick-0.3.0.crate
+ /usr/bin/tar -xof -
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd rpick-0.3.0
+ /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ set -eu
+ /usr/bin/mkdir -p .cargo
+ cat
+ /usr/bin/rm -f Cargo.lock
+ exit 0
Executing(buildreqs): /bin/sh -e /var/tmp/rpm-tmp.b2Aszb
+ umask 022
+ cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick
+ cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick
+ cd rpick-0.3.0
+ /usr/bin/cargo-inspector -BR Cargo.toml
+ /usr/bin/cargo-inspector -TR Cargo.toml
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.8KsW4c
+ umask 022
+ cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick
+ cd rpick-0.3.0
+ /usr/bin/env CARGO_HOME=.cargo RUSTC_BOOTSTRAP=1 /usr/bin/cargo build -j8 -Z avoid-dev-deps --release
[…]

If you use fedpkg mockbuild, you will not need to do anything.

@ignatenkobrain

To be honest, I have a problem with your assessment of this change. You repeatedly say that there is no impact at all, just so that 20 comments later we find out that there is actually change in the user workflow which you say users just need to accept. If this is not an impact, then I don't know what is.

Please document it explicitly.

As for what to do in this case:
- how about adding an option to dnf builddep to execute dep generators on request?
- or leave the behavior as it is but document the limitations explicitly in the plugin, provide helpful messages and alternative way of achieving the same result?

Please document it explicitly.

I have added following text to wiki:

For people/tooling who build RPMs, it might be needed to run <code>dnf builddep</code> on new file generated by RPM when dynamic BuildRequires detected and unsatisfied ones found - <code>buildreqs.nosrc.rpm</code>. RPM will guide you where it stored it and which name file has.
  • how about adding an option to dnf builddep to execute dep generators on request?

This kind of problems is there since day 0. If you use %ifarch s390x BuildRequires: s390x-utils %endif, then you can't run dnf builddep on x86_64 on it because you get src.rpm from s390x builder (since Fedora ships only one srpm).

  • or leave the behavior as it is but document the limitations explicitly in the plugin, provide helpful messages and alternative way of achieving the same result?

Can be, I can open bug (or anybody else can do that) against dnf-plugins-core. But I don't see how this can block this Change proposal going forward.

Would I need to install them beforehand?

Well, it is not different from today, if you want some macro expanded in spec, you need to install some package which is providing it.

Same for Golang. If you want to process Go code in Fedora, you need to have a %gometa call somewhere in the spec today.

The %gometa macro is provided by go-srpm-macros (part of the Fedora default build root). One of the things it does is to generate a static BuildRequires on go-rpm-macros (not part of the default build root). Today, go-rpm-macros (among other things) depends on the Go compiler. Tomorrow it will also depend on the Go dynamic build dependency generator. Actually, it already depends on it, because dynamic build dependencies will use the same generator utility, that is already used for Provides and Requires.

So adding dynamic build dependencies to the mix does not change initial build root provisioning one iota. build dependency generator seeding is handled by traditional rpm static BuildRequires, if you know how to process static BuildRequires, you know how to provision dynamic dependency generators.

The beauty of the dynamic build dependency concept, is that is reuses existing rpm/mock infrastructure and tooling. So it is both more powerful, and massively less invasive in tooling footprint, than traditional out-of-rpm foo2spec templating utilities.

I plan to generalise the foo-srpm-macrosfoo-rpm-macros dance to font packages for example because it is a good robust way to balance a minimal generic build root with content-specific build requirements (I'd already have done it if redhat-rpm-config maintainers had finished reviewing the macro parts of PR51)

Will dynamic_buildrequires for mock be enabled by default? If yes, then it is a user impact, if no - then it is also user impact as user needs to enable it to build certain packages.

Yes, for f31+ chroots only.

At first. For languages that can make use of them, the dynamic dependency generator concept is so convenient, that going back to pre-generation times quickly lose any appeal. So if it is only available in f31+, some packagers will quickly balk at doing pre-f31 or EL packaging updates.

At first. For languages that can make use of them, the dynamic dependency generator concept is so convenient, that going back to pre-generation times quickly lose any appeal. So if it is only available in f31+, some packagers will quickly balk at doing pre-f31 or EL packaging updates.

Sorry, but new features are getting implemented in next version of Fedora. If you wish to backport any of them later, you still can do so. This change proposal is only about adding this feature in Fedora 31. Also I don't think RPM maintainers will backport anything like that into F29/F30's RPM.

@ignatenkobrain sure, sorry I wasn't clear, I'm not arguing to modify the change proposal target, which is f31+.

I do believe, however, that the desire of a backport will be quickly felt with or without my involvement. As @tibbs argued in the past, not making our tooling improvements available in all supported releases, is a serious drag on the project effectiveness and attractiveness.

@ignatenkobrain I see it's been about a week since your last edit to the proposal. Are you ready for FESCo to resume voting?

@bcotton, I thought that it already is :)

So the answer is definitely yes.

Since the proposal has changed, I'd like to start the vote timer again.

Metadata Update from @bcotton:
- Issue tagged with: meeting

7 months ago

@bcotton Did you want this on this week's meeting agenda? I ask because of the tag, but it also sounded like you wanted to give it the usual week of voting.

@bcotton Did you want this on this week's meeting agenda? I ask because of the tag, but it also sounded like you wanted to give it the usual week of voting.

@bowlofeggs if time permits, we can put it on the agenda just to close it out. If other topics take up all the time, I'll just wait until the 7 days have passed.

We will discuss this during the FESCo meeting on Friday at 15:00UTC in #fedora-meeting-1 on
irc.freenode.net.

Thanks! Please ping me when you will discuss that change so that I can chime in and answer any questions.

Correction: the meeting will be in #fedora-meeting.

Metadata Update from @bowlofeggs:
- Issue close_status updated to: Accepted
- Issue status updated to: Closed (was: Open)

7 months ago

Metadata Update from @zbyszek:
- Issue untagged with: meeting

6 months ago

Login to comment on this ticket.

Metadata