Add possibility to generate build-time dependencies within RPM spec file and teach RPM and mock how to handle this.
There needs to be an upstream rpm release with this feature included, we do not do feature backports in Fedora rpm. As of now, this is only being discussed at upstream.
This is something to consider, however I'm +1 for allowing this to happen. I guess there is always a chance that that a change will not happen in time.
I am +1. I think it's OK to approve it even if the upstream doesn't have it yet. If it doesn't get in in time, we can always postpone to a later Fedora release.
+1 also, we do need to keep track of where upstream is here (like the other rpm change)
+1
Metadata Update from @zbyszek: - Issue tagged with: meeting
[ Random note: if this is enabled for Python, I'm guessing we're going to have quite a few circular build dependencies, because build/runtime dependencies aren't distinguished. And while circular build dependencies don't typically matter, they do for bootstrapping, and for automatically determining build ordering. But if the facility is available it could be enabled for other languages and not Python if that turned out to be a problem. ]
For now, my ultimate goal is to have it just for Rust ecosystem. In future we might want to use it for other languages / ecosystems, but that is definitely not in scope for this change. Even Rust generators are not in this scope.
So basically Python projects usually only have very few build dependencies (setuptools, numpy, cython). It's test dependdencies and we could have a higher level macro that controls whether you want them or not (depending on %with check) etc. as we already need to bootstrap such deps manually.
%with check
This was discussed in today's meeting. We didn't reach a formal conclusion, because there are still questions about the details of the proposal. In particular contyk said he'll post some questions.
Metadata Update from @zbyszek: - Issue untagged with: meeting
A generator code is already written and used out of Fedora for Go, it is only waiting for the rpm part to be used within Fedora. (a second-gen Go BR generator is being written to handle Go modules, including a form of BR version constrains
After 10 days, I count the vote as (+5,0,-0). Are @psabata's questions answered, or should I delay processing this proposal as approved?
@contyk: please post your questions now, or remain forever silent.
My main worry is we won't have the generated dependencies included in the SRPMs, which might make analysis and generating proper build order rather complicated. The change proposal includes a bunch of unknowns.
To me, voting on this proposal at this stage feels rather rushed. If the proposal owners don't think it's worth working on the code unless we approve it right now, it probably isn't.
Metadata Update from @psabata: - Issue tagged with: meeting
I'd just like to not end up in the situation where we've implement something in upstream and then have FESCo said that we don't like this and that. I would prefer to hear all concerns and requirements in advance and make implementation to account for this.
I know that it should have been sent to devel ML for such discussion, but unfortunately such topics do not lead to getting such answers.
Now to your point about dependencies in SRPMs. I think it should be possible to add some flag to dependencies which would mean that those are autogenerated and then flag in rpmbuild/mock which would regenerate (or not) those dependencies. This way we can still inspect what dependencies have been used. Definitely this raises question about arch-dependent things..
Even today, you don't have correct dependencies in SRPMs if one is using %ifarch construction for BuildRequires. Because we ship just one SRPM instead of one per architecture.
And it is not only about ifarch, but dependencies can be injected depending on other RPM variables and even depending on file/time/whatsoever.
So what I would like to do from FESCo is list of concerns, I can propose ways to resolve them.
Have you considered building this tooling to be a part of packager toolbox (like rpmlint, rpmbuild new-spec or smth like this). You can build the autogenerator for packagers to run locally. Its output then is going to be a template for a spec file or recommended list of requirements.
This will create the initial userbase which is going to provide feedback, verify corner cases and so on. This userbase will then provide the source of concerns and requirements, which will give us the better ground to decide whether it is worth it to built it in the internals of the rpmbuild process.
And i don't really get your reason not to have a mailing list discussion for it.
Well, we do that for long long time. But if you need to change something in generator, you have to re-do all the spec files instead of just changing code in generator and rebuild all affected packages (without changing any single line in them).
See rust2rpm, pyp2rpm, gem2rpm and more tooling which generates spec files out of upstream metadata.
rust2rpm
pyp2rpm
gem2rpm
Is it possible to collect the data on how much actual rpm specs differ from those suggested by tools?
But if you need to change something in generator, you have to re-do all the spec files instead of just changing code in generator and rebuild all affected packages (without changing any single line in them).
Do you propose policy on mass rebuilds as a part of this Change? On which events and which requirements?
All rust-* packages (around 550 at this moment) are generated using rust2rpm. Then they are processed to remove some subpackages, fixup description/summary and such small tweaks. If dependencies in the spec file are different from what rust2rpm puts there, 99% package won't build.
rust-*
Best judgement of generator' maintainers. For example, if I change rust2rpm generator and I consider it being important for rebuild -- I'll just go and rebuild affected packages in rawhide.
This part is actually not different from today. If you change dependency generators (which exist for last 10+ years), maintainer needs to decide whether to rebuild and what.
Speaking of which, https://pagure.io/fedora-rust/rust2rpm/issue/70 issue showed up today. We probably will need to change how dependencies are constructed. But that basically means we have to regenerate all spec files.
That has been tried many times for many languages and it does not scale nor works long term because the
upstream artefact → generated spec → build
workflow is really a
upstream artefact → generated spec → packager adjusted spec → build
So unless you are so lucky your upstream always generates perfect artefacts you quickly end up with a mass of specs that can not be regenerated automatically without human review, and this is too resource intensive, the result slowly rots, resulting in an unmanageable pile of different style specs generated at different times with different tooling versions.
And, it never worked at all for the large class of projects that mix several kinds or artefacts within a single archive.
To keep something maintenable you need to split the generating functions in atoms and compose them within your spec like everything else.
That means either integrating the generators in rpm to keep rpm our primary tool, or add a templating layer over rpm to work around the facts you can not compose the generators at the rpm level (which means that, basically, your packager tool is no longer rpm but something that wraps it).
Switching the distro from plain rpm to something else over rpm would be a lot more dangerous and invasive than integrating the generators within rpm in the first place. It would imply redoing all out tooling from the ground up to take into account the wrapping template as primary package descriptor, not the spec file.
And yes you can avoid that by pretending the generation does not happen. What that means concretely is that you have large part of the package maintenance which is no longer tracked and audited by Fedora tools.
My main worry is we won't have the generated dependencies included in the SRPMs, which might make analysis and generating proper build order rather complicated.
Right now a SRPM is created as a build artifact in koji. How does this change with the proposed change? IIUC, there are two SRPMs — the original one, and the one augmented by generated BuildRequires. Will both be visible in koji? The change page says "TBD", but this is a rather important bit.
I don't know if we need both SRPMs to be kept. Maybe it it is enough if the generated list of BR is kept as a file or something.
1) let's limit the scope to Rust alone; 2) figure out how to save build requirements in the srpm produced; 3) define the impact on dnf; 4) define the impact on local developer who tries to rebuild the package on a different environment.
The reproducibility issue is covered in the proposal in a wrong way. If i reproduce build environment to every single bit, i get the same result. But that's not the point of a reproducibility. Or maybe better to say, you have reproducibility, but no transparency, no possiility to build on that.
I want to rebuild a package on my Fedora system, or my CentOS system, and I need to easily track dependencies i need to install and how these are changed. I don't have a task to reproduce bit by bit, I have a task to change in a controlled way.
1) let's limit the scope to Rust alone;
Python wants this too...
@churchyard I am not saying we should never try it for Python. But it was mentioned in the comments above that the first target is going to be Rust. So let's start with that and proceed step by step.
Especially since @otaylor mentioned that Python might have certain problem with this approach.
FESCo likes the idea in general, but will wait for a specification to review before accepting for use in Fedora. We will be happy to answer questions in ticket along the way (+8,0,-0)
Metadata Update from @jforbes: - Issue untagged with: meeting
This idea is to have a generic mechanism that works for all and any language. It is possible that only rust might use this in the beginning, but the change on the rpm/mock/koji side is not tied to any specific language stack.
2) figure out how to save build requirements in the srpm produced;
+1, this needs to be figured out.
3) define the impact on dnf;
There shouldn't be any. In this proposal (as opposed to the other pm_requests one), dnf is simply invoked another time with an additional list of deps to satisfy. It's the higher-level components that orchestrate this.
4) define the impact on local developer who tries to rebuild the package on a different environment.
+1. If 2) produced a "full" srpm, then this will be solved automatically.
The reproducibility issue is covered in the proposal in a wrong way. If i reproduce build environment to every single bit, i get the same result. But that's not the point of a reproducibility. Or maybe better to say, you have reproducibility, but no transparency, no possiility to build on that. I want to rebuild a package on my Fedora system, or my CentOS system, and I need to easily track dependencies i need to install and how these are changed. I don't have a task to reproduce bit by bit, I have a task to change in a controlled way.
2) should also solve those reproducibility/transparency issues.
I have a feeling that we are mixing the generic effort on the RPM side and the change proposed to Fedora 31 here. And it also needs to be specifically explained in the proposal.
So the generic direction, the effort is "to enable Build Requires generator for everything". Interesting idea, worth looking into, we agreed on that.
But what is the actual proposal for Fedora 31?
Is it "to update RPM to the upstream version which supports this functionality"? Or "to enable this feature for all packages on Fedora Koji service"? Or "enable it in MBS for Rust module"?
If it works for Rust but doesn't work for Python yet, should Change for F31 be declared Done or Deferred?
to technically enable this feature for all packages - enable, not enforce.
IMHO the change doesn't say "we will mass change our packages to use it" it says "we will make it possible for packages to use it".
So, assume i have three packages which, when enabled with this feature, get their BuildRequires right, and then I have 3000 packages, for which, if I enable this feature, it ends as garbage.
Is this change done or not?
Note, I don't have my own final opinion on that yet. But I believe we need to have this answered and this answer recorded in the proposal to move forward.
I don't think we are on the same page. This change doesn't calculate the buildrequires, it just gives you the opportunity to do so. You can write your own Perl script that outputs the package names. This change cannot turn packages into garbage, it only gives you the tool to do whatever you want.
That would be massively unfriendly to all the Fedora contributors, that worked on this, in different SIGs. Just because a member of the rust SIG got sufficiently fed up with the lack of advancement to write the actual change request, does not mean rust people are the only ones who have been working or waiting on this.
Then you do not enable it in the problem specs (you need to add the autobr call line to a spec to enable it). Or you add a | grep -v "bad BR" to the specs where a generator generates a problematic BuildRequires. It's no harder than that.
| grep -v "bad BR"
The change is a generic rpm mecanism where rpm accepts the output of spec commands as a BR list. If you don't like the BRs produced by those other commands you can replace them of massage their output the usual spec way.
1) let's limit the scope to Rust alone; That would be massively unfriendly to all the Fedora contributors, that worked on this, in different SIGs. Just because a member of the rust SIG got sufficiently fed up with the lack of advancement to write the actual change request, does not mean rust people are the only ones who have been working or waiting on this.
Sorry, I don't get it.
We are discussing the proposed Change. Change Owners present it and scope it.
I am not trying to enforce additional limitations on the scope from my side here, I am trying to make sure that the change as it is proposed, and the change as it is going to be delivered are the same thing. Thus I am relying on Change Owner's comment here that the focus "for now" is going to be Rust.
If there other contributors who will cover other areas, I suggest you get onboard. Join forces, communicate with current Change Owners and add yourself to co-Owners. Or convince current Owners that support for other languages need to be in the focus as well and that you are going to help them.
In the end it is the Change Owner who supposed to be responsible for the result. We can not add tasks on them without them accepting it.
Well, change on its own is about implementing generic way of supplying BR generators and making sure it works fine and is supported by infrastructure. That's it.
Once this is there, I'll obviously implement such generators for Rust.
I don't think we are on the same page.
Actually yes, because there is no page. I am guessing and in a wrong way it seems.
This change doesn't calculate the buildrequires, it just gives you the opportunity to do so. You can write your own Perl script that outputs the package names. This change cannot turn packages into garbage, it only gives you the tool to do whatever you want.
So let me try again and check if I get it right this time:
1) there is going to be change in the rpm in the upstream which provides a way for packager to replace hardcoded list of Build Requires with a list of autogenerated ones.
Macro?
2) there will be release in upstream of several plugins (per language) which can be added to rpm to provide the implementation on how to autogenerate build deps in each case
3) Fedora Infra would need to update to this RPM version and install plugins in the build environment
4) Fedora Packaging Guidelines need to allow this kind of macro in the spec file
See above. Change is about generic infrastructure in RPM which allows generating BRs and making sure it is supported by infra.
What do you think about:
--reuse-buildrequires
This way we can ship RPMs which still have static BuildRequires which are sufficient to build RPM, but still when generators change their output, have possibility to regenerate BRs from the scratch.
No impact.
Yes.
Hopefully not. Rather how dependency generators works today in RPM. They specify some file patterns on which some script is run.
Yes, main functionality will go into RPM. But it should have enough flexibility so that there is no hardcoding in RPM for language-specific stuff.
https://rpm.org/user_doc/dependency_generators.html
Fedora Infra would need to upgrade to new release of mock which supports necessary functionality. Nothing additional goes into default buildroot.
FPG doesn't prohibit any macro. But yes, FPG needs to gain some page which describes how to use dependency generators for BuildRequires. I'm willing to work on it (putting my Packaging Committee hat on).
Then respect the change owner scoping. If the change owner wanted to limit it to rust, he would have written it so (and all that would have resulted in is cut and pasting of the change with rust replaced by other languages, and FESCO asking to consolidate everything in a single ticket).
The "let's limit to it rust" thing was added by FESCO during FESCO discussions, based on a deliberate misinterpretation and expansion of a single sentence in the change page.
Then respect the change owner scoping.
I do. I am still allowed to suggest things though. If these suggestions don't make sense, we discard them.
If the change owner wanted to limit it to rust, he would have written it so (and all that would have resulted in is cut and pasting of the change with rust replaced by other languages, and FESCO asking to consolidate everything in a single ticket).
The description of a change is mostly non-existent. Thus it is hard to argue about what is expected and what is not. Don't take me as an enemy here, I am trying to ask questions so that the proposal can be improved by adding answers to them.
"Let's limit to Rust" - "No, because.." - is a good answer too. Let's move on.
1) there is going to be change in the rpm in the upstream which provides a way for packager to replace hardcoded list of Build Requires with a list of autogenerated ones. Yes.
rpmbuild will accept at a specific point of the spec (before %build, after %prep) the output of commands as a list of BuildRequires
There are no constrains of the kind of commands rpmbuild will accept the output of (upstream tool, fedora wrapper, package-specific script, macro, whatever). echo foo will cause foo to be installed if written in the correct part of the spec.
echo foo
foo
If upstream is nice enough to provide a command that produces buildrequires lists in a syntax rpm understands, it is likely to be used as-is (our current font provides, for example, use raw fontconfig output). The most likely case is something in between: upstream provides a build requires tool, and the output of this tool needs to be translated into rpm syntax at the rpm, distro, or individual packager level.
I sort of doubt that will happen, people do not need rpm upstream help to write commands that output lists of things, and rpm upstream is not terribly interested in baby-sitting the writing of lots of domain-specific commands (or just inventorying them). If the rpm world was organized this way most of redhat-rpm-config would have been merged into the rpm project long ago.
Plus, the actual generator commands will depend on the software stack implementations each distribution ships. If python 3.x provides the command generator, and python 4.z the command bettergenerator, rpm upstream will just expect distributions to write generator or bettergenerator in their specs, depending on their python level target. (or write a python-generator macro that warps the correct generator depending on the result of distro-specific processing).
rpm upstream will only provide the facility to feed BR lists to the build env at a specified point of the spec, and distributions and packagers will do whatever they want with it (just like rpm upstream does not have any opinion, on the commands you put in %build or %install)
Yes, main functionality will go into RPM. But it should have enough flexibility so that there is no hardcoding in RPM for language-specific stuff. https://rpm.org/user_doc/dependency_generators.html
The main difference I see with the existing dependency generators, is that "stuff" at the end of %prep is not nicely filtered and put in standard FHS places, so I doubt a fileattr mecanism could work without lots of efforts to restructure sources in %prep.
%prep
fileattr
It's probably simpler and more efficient to just ask the packager to add explicit calls to the magic generator commands he needs in his spec.
3) Fedora Infra would need to update to this RPM version and install plugins in the build environment Fedora Infra would need to upgrade to new release of mock which supports necessary functionality.
Fedora Infra would need to upgrade to new release of mock which supports necessary functionality.
and use a rpmbuild version that includes the feature
Nothing additional goes into default buildroot.
The default buildroot is likely to gain more foo-srpm-macros packages to define a convenient wrapping of upstream generator commands. It is likely to shed more "just in case it's needed" parts because the feature permits finer control of buildrequires, on a per-srpm level.
Fedora Packaging Guidelines would need to provide guidance on what generator commands to use in a rust project, in a golang project, a python project, and so on.
Nothing different from the guidance on the commands to use in other parts of a Fedora spec, with the same domain specificity, and the same maintenance as upstream commands evolve and are wrapped (or not) in Fedora macros.
@nim
Afaik, BuildRequires are already used by %prep stage
We have just recently had couple of issues with that, when python3-devel or autotools were required for %prep stage to finish.
So inserting new set of Build Required between steps makes it two different types of Build Requires (those in %prep and those in %build)
I wonder if there should be explicitly defined PrepRequires then? BuildRequires(prep)?
2) there will be release in upstream of several plugins (per language) which can be added to rpm to provide the implementation on how to autogenerate build deps in each case (...) rpm upstream will just expect distributions to write generator or bettergenerator in their specs, depending on their python level target. (or write a python-generator macro that warps the correct generator depending on the result of distro-specific processing).
(...)
rpm upstream will just expect distributions to write generator or bettergenerator in their specs, depending on their python level target. (or write a python-generator macro that warps the correct generator depending on the result of distro-specific processing).
So, should this also be done in this Change? Or should it be just an ongoing effort happening at its own pace?
The main difference I see with the existing dependency generators, is that "stuff" at the end of %prep is not nicely filtered and put in standard FHS places, so I doubt a fileattr mecanism could work without lots of efforts to restructure sources in %prep. It's probably simpler and more efficient to just ask the packager to add explicit calls to the magic generator commands he needs in his spec.
Feels like a new %configure section in RPM spec.
%configure
rpmbuild will accept at a specific point of the spec (before %build, after %prep) the output of commands as a list of BuildRequires Afaik, BuildRequires are already used by %prep stage
The change provides an additional BuildRequires entry point, the existing static one is unchanged. Static BuildRequires still exist, and you need them to cover at least the bits needed for %prep execution.
The build process becomes
instantiate cached default buildroot install buildroot updates install static buildrequires unpack sources and execute %prep execute dynamic buildrequires generators install computed dynamic buildrequires execute %build…
The proposed system adds the equivalent dynamic capability to BuildRequires, that already exists for %files.
BuildRequires
%files
The dynamic capability of %files is provided by the -f flag. A %files section can contain both static entries, and dynamic entries provided in the -f list. Both the static and dynamic entries use the same verbs and syntax, no need to invent a separate one.
-f
For BuildRequires there is no existing %buildrequires section to hang a -f on, so list feeding just occurs at a specific point of the spec (a file containing a list and the output of commands are pretty much equivalent, you can transform files in output with cat and output in files with >>)
%buildrequires
cat
>>
2) there will be release in upstream of several plugins (per language) which can be added to rpm to provide the implementation on how to autogenerate build deps in each case (...) rpm upstream will just expect distributions to write generator or bettergenerator in their specs, depending on their python level target. (or write a python-generator macro that warps the correct generator depending on the result of distro-specific processing). So, should this also be done in this Change? Or should it be just an ongoing effort happening at its own pace?
(...) rpm upstream will just expect distributions to write generator or bettergenerator in their specs, depending on their python level target. (or write a python-generator macro that warps the correct generator depending on the result of distro-specific processing).
As soon of the capability is available in rpm and mock Fedora packagers will start experimenting with it.
The recommended way to make use of it in specs will go through FPC like the rest of our spec stuff, probably domain by domain, as soon as someone bothers to write a nice domain application packaging guideline, in the usual iterative ongoing way.
Since several SIGs have been working on this and are waiting for the rpm/mock part to be finished, I expect several packaging guideline proposals will happen quickly after the feature is deployed infra-side (as soon as FPC has a build infra with the feature to evaluate proposals on).
You can simulate a big part of the feature with pm_request (not secure enough to deploy public infra side, but secure enough to run on your own private systems). Therefore its completion is not blocking early generator work.
And, regardless of what FPC decides, you will have some uses of the feature in Fedora specs not covered by guidelines, because FPC only has the time and energy to write guidelines for common needs, and less common needs are left to individual packager discretion and good sense.
Not really, configure allows restricting a build to things detected within the build root. This proposal has the inverse effect: expanding the build root to things that we realize are needed once sources have been expanded and inspected.
There is a difference though. BuildRequires changes the flow of the build process.
Consider the autoreconf command. It makes decisions based on what is available in the build environment. Currently there is no difference between putting it in %prep or %build stage, but with update of BuildRequires hooked in between, it makes it a big deal.
autoreconf
And then approach by @ignatenkobrain wouldn't work, because "new spec", with added Build Requires will not reproduce the build process for package in the same way.
Essentially, listing the same file explicitly or dynamically produces the same result. But adding the same build requirement statically or dynamically would produce different packages.
The proposed system adds the equivalent dynamic capability to BuildRequires, that already exists for %files. There is a difference though. BuildRequires changes the flow of the build process. Consider the autoreconf command. It makes decisions based on what is available in the build environment. Currently there is no difference between putting it in %prep or %build stage, but with update of BuildRequires hooked in between, it makes it a big deal.
There is a difference though. BuildRequires changes the flow of the build process. Consider the autoreconf command. It makes decisions based on what is available in the build environment. Currently there is no difference between putting it in %prep or %build stage, but with update of BuildRequires hooked in between, it makes it a big deal.
autoreconf generates files from sources. That's pretty much the definition of something that belongs in %build. rpm is very lenient and lets you execute commands in any order in any section. So you can get by with misplacing things a little. But, that will generally end up in woe later.
%build
That happened to me recently. I didn't think hard if an action was of a prep build or install nature. I put it in build. I worked fine most of the times, except in special cases. I spent a few months piling up workarounds, before thinking hard and realising I was fighting rpm. And I was fighting rpm because I was doing preparation actions too late in build. Just moving the action in the right section removed the need for workarounds and simplified my code a lot.
Point being, that even if it does not look like it, correct action ordering is very important for reliable rpmbuild execution. Using the correct section matters. The implicit design rpmbuild enforces relies on them.
Essentially, listing the same file explicitly or dynamically produces the same result.
Only if you do not use things like %defattr. If you do, ordering matters. A lot of things in specs have ordering constrains that only trigger in specific cases.
We have the proposal in https://pagure.io/fesco/issue/2096#comment-561887. I think it's reasonable, but it needs putting in the Change page and fleshing out.
We've had quick (not really) call today with @ngompa and @ffesti. We've agreed that final SRPM will include auto-generated BuildRequires, so we are good with this. We're still fleshing out details about rpmbuild flags and some workflow though.
Also to clarify a bit more details, there will be special section in spec file where packagers will put scripts/macro to generate BRs. Same here, we're choosing name (one of those hard problems in programming).
I'll update change proposal in week or two.
That's terrific news, thanks @ignatenkobrain @ngompa and @ffesti
Given https://pagure.io/fesco/issue/2096#comment-562909 I am taking this off of the agenda until the proposal is updated. Thanks very much @ignatenkobrain @ngompa and @ffesti
@bcotton @sgallagh @bookwar @psabata I have renamed change proposal to "DynamicBuildRequires" so that it better matches upstream name.
Also I have updated wiki page which hopefully resolves all concerns along with POC example.
https://fedoraproject.org/wiki/Changes/DynamicBuildRequires
Still +1.
Thanks @ignatenkobrain
I think we still need releng issue https://pagure.io/releng/issue/8129 to be clarified.
And a question: if I understand correctly the srpm produced with the feature enabled will contain all BuildRequires( this is needed for tools like repoquery to work). Will I be able to recover original spec file from this srpm?
Looks great. I'm still +1.
I'm confused by the ".nosrc.rpm" extension though — it would imply that that srpm does not include sources, but in the transcript, sources are unpacked from it.
Also, to clarify: in koji builds, do we keep the original "lean" src.rpm, or the second src.rpm with rpmlib(DynamicBuildRequires), or both?
rpmlib(DynamicBuildRequires)
Yes. Spec file are not modified.
buildreqs.nosrc.rpm is just temporary RPM which is generated just to be able to use dnf builddep. There are no sources, probably neither even spec (but I didn't check).
dnf builddep
In koji builds, we will store only one src.rpm. The one which contains all dynamic BuildRequires.
Funny Python story: We would need to do this twice (there is a two-phase dependencies check - the first gives you a tool like setuptools or flit and the second uses this tool to get stuff like Cython). I suppose we need to take that to RPM upstream, right?
Nope, RPM upstream already is capable of doing that... It is just need some implementation on mock side... Which was not trivial, so I didn't implement it.
See here: https://github.com/rpm-software-management/mock/pull/248/commits/3d8815d35c5ab178542b6a899c97420ffe96b683#diff-a3ef273b52c1ee7fd2846e6fb135fc28R522
Also, I disagree that there is no impact on dnf and on end user.
I understand that it doesn't change the basics of how packages are managed. And I guess there are no parts which are explicitly blocking for this change on dnf side.
But
1) we create confusion for regular users and dnf is going to be the first point of contact for it 2) there might be feature requests to support new behavior from UX side of dnf. For example requests for additional repoquery flags. 3) For things like dnf builddepthe impact is unclear, but at the very least the doc [1] needs to be updated
So I'd suggest to stop declaring that this feature has no visible effect, rather embrace the fact that it has, and look into what can be done to make it visible for end users in a proper way.
And by users we shouldn't mean just fedora desktop people. There are tools like rpmdeplint, rpmdiff (aka rpm inspect), there are rpm templating engines, like renderspec [2] and other consumers who build their infrastructure on top of this tooling.
I am not saying we should explicitly deal with each one of them, and block on them, but let's think about them as users too and look into this topic instead of discarding it right away.
[1] https://dnf-plugins-core.readthedocs.io/en/latest/builddep.html [2] https://github.com/openstack/renderspec
1) we create confusion for regular users and dnf is going to be the first point of contact for it
What kind of confusion?
2) there might be feature requests to support new behavior from UX side of dnf. For example requests for additional repoquery flags.
In that case, it won't be only DNF because you will need to propagate more data through rpm-md.. That means you will need libsolv, createrepo_c, libdnf changes at the very least.
3) For things like dnf builddepthe impact is unclear, but at the very least the doc [1] needs to be updated
As I said multiple times, dnf builddep is not affected. There is absolutely no changes needed.
Do you have proposal what I should put in the wiki page?
What kind of confusion? ... As I said multiple times, dnf builddep is not affected. There is absolutely no changes needed.
Let's consider the question: What does the command dnf builddep foobar.spec do?
dnf builddep foobar.spec
With the change enabled, the line "Install the needed build requirements, defined in the foobar.spec file" is confusing. And adding couple of lines to explain it will help.
(UPDATE: Also adding a flag to install initial BRs only without dynamic ones could be a valid feature request. I already have a use case for it.)
Now how is it going to work with distro-specific macro definitions? Would I need to install them beforehand? Would it be dependency for dnf plugin? Will dnf plugin show a proper error message in case it can not find the dependency generator required by this spec file and in case it failed by some reason?
Btw, is there an impact on fedora-review? rpmlint?
fedora-review
Will dynamic_buildrequires for mock be enabled by default? If yes, then it is a user impact, if no - then it is also user impact as user needs to enable it to build certain packages.
dynamic_buildrequires
Will it be required to explicitly install distro-specific language-specific dependency generators before building packages or running dnf builddep? Then it is a user impact as well.
New section in rpm spec also can be considered a user impact but this one can be skipped i guess.
Installs BuildRequires specified in foobar.spec (not the dynamic ones).
foobar.spec
Now how is it going to work with distro-specific macro definitions?
…
Would I need to install them beforehand?
Well, it is not different from today, if you want some macro expanded in spec, you need to install some package which is providing it. So, imagine case with Rust: you need following lines in spec:
%generate_buildrequires %cargo_generate_buildrequires
You need to have rust-packaging installed to expand that macro, so you have 2 options:
rust-packaging
BuildRequires: rust-packaging %generate_buildrequires %cargo_generate_buildrequires
or
%generate_buildrequires %if %{defined cargo_generate_buildrequires} %cargo_generate_buildrequires %else echo "rust-packaging" %endif
Both are valid.
Would it be dependency for dnf plugin?
No.
Will dnf plugin show a proper error message in case it can not find the dependency generator required by this spec file and in case it failed by some reason?
DNF doesn't touch dynamic BuildRequires.
fedora-review is using mock, so I doubt. rpmlint should not be affected. Both obviously can grow some features to deal with dynamic BuildRequires better or do some additional checks, but they will keep working as is and there is no loss in functionality.
rpmlint
Yes, for f31+ chroots only.
As I said above, this is not changing at all. Packagers have today them in BuildRequires. Tomorrow they will have to run dnf builddep multiple times if rpmbuild aborts.
Please share it. But anyway, dnf builddep installs only manually-specified BuildRequires.
But anyway, dnf builddep installs only manually-specified BuildRequires.
Wait, but then it is a bug, and an impact. And actually a blocking one, because it heavily affects the user workflow.
If it affects user workflow, it is not blocking bug. Many changes are affecting user workflows. This is just one of them.
What do you suggest DNF to do? Run (potentially) untrusted code on user' machines? It is job of mock (or any other system which is orchestrating build of RPM packages).
⋊> ~/P/s/r/rust-rpick on master ⨯ sudo dnf builddep rust-rpick.spec 13:20:02 Last metadata expiration check: 0:00:47 ago on Fri 12 Apr 2019 01:19:22 PM CEST. Package rust-packaging-6-28.fc31+buildreqs.x86_64 is already installed. Dependencies resolved. Nothing to do. Complete! ⋊> ~/P/s/r/rust-rpick on master ⨯ fedpkg local 13:20:10 Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.Ydztwl + umask 022 + cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick + cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick + rm -rf rpick-0.3.0 + /usr/bin/gzip -dc /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick/rpick-0.3.0.crate + /usr/bin/tar -xof - + STATUS=0 + '[' 0 -ne 0 ']' + cd rpick-0.3.0 + /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w . + set -eu + /usr/bin/mkdir -p .cargo + cat + /usr/bin/rm -f Cargo.lock + exit 0 Executing(buildreqs): /bin/sh -e /var/tmp/rpm-tmp.SckKAl + umask 022 + cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick + cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick + cd rpick-0.3.0 + /usr/bin/cargo-inspector -BR Cargo.toml + /usr/bin/cargo-inspector -TR Cargo.toml + exit 0 error: Failed build dependencies: (crate(dirs/default) >= 1.0.0 with crate(dirs/default) < 2.0.0) is needed by rust-rpick-0.3.0-1.fc31.x86_64 (crate(rand/default) >= 0.6.0 with crate(rand/default) < 0.7.0) is needed by rust-rpick-0.3.0-1.fc31.x86_64 (crate(serde/default) >= 1.0.0 with crate(serde/default) < 2.0.0) is needed by rust-rpick-0.3.0-1.fc31.x86_64 (crate(serde/derive) >= 1.0.0 with crate(serde/derive) < 2.0.0) is needed by rust-rpick-0.3.0-1.fc31.x86_64 (crate(serde_yaml/default) >= 0.8.0 with crate(serde_yaml/default) < 0.9.0) is needed by rust-rpick-0.3.0-1.fc31.x86_64 (crate(structopt/default) >= 0.2.0 with crate(structopt/default) < 0.3.0) is needed by rust-rpick-0.3.0-1.fc31.x86_64 Wrote: /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick/rust-rpick-0.3.0-1.fc31.buildreqs.nosrc.rpm Could not execute local: rpmbuild --define '_sourcedir /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick' --define '_specdir /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick' --define '_builddir /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick' --define '_srcrpmdir /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick' --define '_rpmdir /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick' --define 'dist .fc31' --define 'fedora 31' --eval '%undefine rhel' --define 'fc31 1' -ba /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick/rust-rpick.spec | tee .build-0.3.0-1.fc31.log ⋊> ~/P/s/r/rust-rpick on master ⨯ sudo dnf builddep /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick/rust-rpick-0.3.0-1.fc31.buildreqs.nosrc.rpm Last metadata expiration check: 0:01:01 ago on Fri 12 Apr 2019 01:19:22 PM CEST. Package rust-packaging-6-28.fc31+buildreqs.x86_64 is already installed. Dependencies resolved. ================================================================================ Package Arch Version Repo Size ================================================================================ Installing: rust-dirs+default-devel noarch 1.0.5-1.fc31 rawhide 7.6 k rust-rand+default-devel noarch 0.6.5-3.fc31 rawhide 8.4 k rust-serde+default-devel noarch 1.0.90-1.fc31 rawhide 12 k rust-serde+derive-devel noarch 1.0.90-1.fc31 rawhide 12 k rust-serde_yaml+default-devel noarch 0.8.8-2.fc30 rawhide 8.3 k rust-structopt+default-devel noarch 0.2.15-2.fc31 rawhide 8.2 k Installing dependencies: […] Transaction Summary ================================================================================ Install 87 Packages Total download size: 2.0 M Installed size: 8.0 M Is this ok [y/N]: y […] Complete! ⋊> ~/P/s/r/rust-rpick on master ⨯ fedpkg local 13:20:36 Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.PmWhFa + umask 022 + cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick + cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick + rm -rf rpick-0.3.0 + /usr/bin/gzip -dc /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick/rpick-0.3.0.crate + /usr/bin/tar -xof - + STATUS=0 + '[' 0 -ne 0 ']' + cd rpick-0.3.0 + /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w . + set -eu + /usr/bin/mkdir -p .cargo + cat + /usr/bin/rm -f Cargo.lock + exit 0 Executing(buildreqs): /bin/sh -e /var/tmp/rpm-tmp.b2Aszb + umask 022 + cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick + cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick + cd rpick-0.3.0 + /usr/bin/cargo-inspector -BR Cargo.toml + /usr/bin/cargo-inspector -TR Cargo.toml + exit 0 Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.8KsW4c + umask 022 + cd /home/brain/Projects/src.fedoraproject.org/rpms/rust-rpick + cd rpick-0.3.0 + /usr/bin/env CARGO_HOME=.cargo RUSTC_BOOTSTRAP=1 /usr/bin/cargo build -j8 -Z avoid-dev-deps --release […]
If you use fedpkg mockbuild, you will not need to do anything.
fedpkg mockbuild
@ignatenkobrain
To be honest, I have a problem with your assessment of this change. You repeatedly say that there is no impact at all, just so that 20 comments later we find out that there is actually change in the user workflow which you say users just need to accept. If this is not an impact, then I don't know what is.
Please document it explicitly.
As for what to do in this case: - how about adding an option to dnf builddep to execute dep generators on request? - or leave the behavior as it is but document the limitations explicitly in the plugin, provide helpful messages and alternative way of achieving the same result?
I have added following text to wiki:
For people/tooling who build RPMs, it might be needed to run <code>dnf builddep</code> on new file generated by RPM when dynamic BuildRequires detected and unsatisfied ones found - <code>buildreqs.nosrc.rpm</code>. RPM will guide you where it stored it and which name file has.
how about adding an option to dnf builddep to execute dep generators on request?
This kind of problems is there since day 0. If you use %ifarch s390x BuildRequires: s390x-utils %endif, then you can't run dnf builddep on x86_64 on it because you get src.rpm from s390x builder (since Fedora ships only one srpm).
%ifarch s390x BuildRequires: s390x-utils %endif
or leave the behavior as it is but document the limitations explicitly in the plugin, provide helpful messages and alternative way of achieving the same result?
Can be, I can open bug (or anybody else can do that) against dnf-plugins-core. But I don't see how this can block this Change proposal going forward.
Would I need to install them beforehand? Well, it is not different from today, if you want some macro expanded in spec, you need to install some package which is providing it.
Well, it is not different from today, if you want some macro expanded in spec, you need to install some package which is providing it.
Same for Golang. If you want to process Go code in Fedora, you need to have a %gometa call somewhere in the spec today.
%gometa
The %gometa macro is provided by go-srpm-macros (part of the Fedora default build root). One of the things it does is to generate a static BuildRequires on go-rpm-macros (not part of the default build root). Today, go-rpm-macros (among other things) depends on the Go compiler. Tomorrow it will also depend on the Go dynamic build dependency generator. Actually, it already depends on it, because dynamic build dependencies will use the same generator utility, that is already used for Provides and Requires.
go-srpm-macros
go-rpm-macros
Provides
Requires
So adding dynamic build dependencies to the mix does not change initial build root provisioning one iota. build dependency generator seeding is handled by traditional rpm static BuildRequires, if you know how to process static BuildRequires, you know how to provision dynamic dependency generators.
The beauty of the dynamic build dependency concept, is that is reuses existing rpm/mock infrastructure and tooling. So it is both more powerful, and massively less invasive in tooling footprint, than traditional out-of-rpm foo2spec templating utilities.
foo2spec
I plan to generalise the foo-srpm-macros → foo-rpm-macros dance to font packages for example because it is a good robust way to balance a minimal generic build root with content-specific build requirements (I'd already have done it if redhat-rpm-config maintainers had finished reviewing the macro parts of PR51)
foo-srpm-macros
foo-rpm-macros
Will dynamic_buildrequires for mock be enabled by default? If yes, then it is a user impact, if no - then it is also user impact as user needs to enable it to build certain packages. Yes, for f31+ chroots only.
At first. For languages that can make use of them, the dynamic dependency generator concept is so convenient, that going back to pre-generation times quickly lose any appeal. So if it is only available in f31+, some packagers will quickly balk at doing pre-f31 or EL packaging updates.
Sorry, but new features are getting implemented in next version of Fedora. If you wish to backport any of them later, you still can do so. This change proposal is only about adding this feature in Fedora 31. Also I don't think RPM maintainers will backport anything like that into F29/F30's RPM.
@ignatenkobrain sure, sorry I wasn't clear, I'm not arguing to modify the change proposal target, which is f31+.
I do believe, however, that the desire of a backport will be quickly felt with or without my involvement. As @tibbs argued in the past, not making our tooling improvements available in all supported releases, is a serious drag on the project effectiveness and attractiveness.
@ignatenkobrain I see it's been about a week since your last edit to the proposal. Are you ready for FESCo to resume voting?
@bcotton, I thought that it already is :)
So the answer is definitely yes.
Since the proposal has changed, I'd like to start the vote timer again.
Metadata Update from @bcotton: - Issue tagged with: meeting
@bcotton Did you want this on this week's meeting agenda? I ask because of the tag, but it also sounded like you wanted to give it the usual week of voting.
@bowlofeggs if time permits, we can put it on the agenda just to close it out. If other topics take up all the time, I'll just wait until the 7 days have passed.
We will discuss this during the FESCo meeting on Friday at 15:00UTC in #fedora-meeting-1 on irc.freenode.net.
Thanks! Please ping me when you will discuss that change so that I can chime in and answer any questions.
Correction: the meeting will be in #fedora-meeting.
I'm also still +1 .
+1 from me
+1 here.
APPROVED (+7, 0, -0)
Metadata Update from @bowlofeggs: - Issue close_status updated to: Accepted - Issue status updated to: Closed (was: Open)
https://meetbot.fedoraproject.org/fedora-meeting/2019-04-26/fesco.2019-04-26-15.00.html
The change in RPM was merged: https://github.com/rpm-software-management/rpm/commit/58dcfddc376a7c97de1432f0082be0d5f01adbcd.
Login to comment on this ticket.