+1, this is effectively blocking postgresql RFE bug [1]; IOW: there already are all the build-time knobs needed by modularity team in postgresql.spec file, but we can not use them in module builds (== the issue). Mis-using binary macro %_module_build to build postgresql "without-feature-A" is just a hack, as some postgresql modules might need that "feature-A" and some not.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1444865

+1, we definitely need a solution here. I need to disable quite a few dependencies in order to build the shared-userspace module, but disabling them for every build other than that doesn't look right to me.

+1, seconding @praiskup's concerns.

I'm not so sure about @ppisar's proposed solution though, one macro could be affecting several packages and we may want to set it differently for each, e.g. to build the documentation of an API package but forgo that for mere build requirements.

I'd prefer something a little more idiomatic that lives with the component, e.g. adding define, with and without directives to data/components/rpms/*:

document: modulemd
version: 1
data:
  …
  api:
    rpms:
      - postgresql-server
  components:
    …
    rpms:
      …
      uuid:
        buildorder: …
        rationale: 'Build dependency (for postgresql).'
        without: docs
      …
      postgresql:
        buildorder: …
        rationale: 'The API we want to provide.'
        define:
          - sdt: 0
          - docs: 1

So if the uuid spec file had a %bcond_with docs or %bcond_without docs directive, i.e. honors a --with-docs or --without-docs flag on the rpmbuild command line and postgresql had %{!?sdt:%global sdt 1} and %{!?docs:%global docs 1}, the above modulemd would build uuid without docs and postgresql without systemtap support but with accompanying documentation. Ideally all of define, with, without would accept lists or single items interchangeably, as in the snippet above.

If you want to formalize the macros with per-component granularity in the modulemd file, why not. I like it. In my original proposal, I simulated the component assignment with namespacing:

%_without_uuid_enables_docs 1
%_without_postgresql_enables_std 1
%_with_postgresql_enables_docs 1

However, your approach does not allow changing the macro for all components. E.g. if you want to disable a documentation everywhere, a global section for "docs" switch would be helpful.

Also don't forget on included module components. I foresee many modulemd files existing for the sole purpose of reusing them in other modules with /data/components/modules/ entries. Then the possibility to parametrize builds of included module components from the including module will be very needed. And this can be achieved only with the global-level macros.

But whatever approach modularity implements, it will be better than nothing.

However, your approach does not allow changing the macro for all components. E.g. if you want to disable a documentation everywhere, a global section for "docs" switch would be helpful.

I don't see why we couldn't do this using a wildcard node(*) that won't conflict with any actual package name and that could still be overridden per package:

data:
  …
  components:
    …
    rpms:
      *: 
        define: pi 3
        with:
          - docs
          - sprinkles
        without:
          - build
          - errors
          - please
      uuid:
        …
        without: docs
      postgresql:
        …
        with: systemtap

(*): I'm not proposing that there needs to be actual wildcard expansion

Also don't forget on included module components. I foresee many modulemd files existing for the sole purpose of reusing them in other modules with /data/components/modules/ entries. Then the possibility to parametrize builds of included module components from the including module will be very needed. And this can be achieved only with the global-level macros.

Modules that are pulled in by others aren't rebuilt, so there's no need to set build time flags for them. If you need a feature of a module you depend on, it must be enabled in that module.

Big +1 here.

It would be awesome if we could discuss this proposal with other modularity stakeholders (/summon @psabata @ralph @langdon) and have this available in infrastructure soon-ish.

My only concern is that doing per-package bconds and macro definitions is not trivial to implement and hence could take quite some time. On the other hand, if we could just define a set of macros to be available to all packages within a module build (= adding those macros into module-build-macros spec), that should be easy to implement. We can always improve this -- start small, improve over time.

Modules that are pulled in by others aren't rebuilt

I do not talk about /data/dependencies/buildrequires/*. I talk about /data/components/modules/includemodules/* entries. My impression was that they are built in the current build root and outputs it's RPM packages into the current module.

@ppisar

I talk about /data/components/modules/includemodules/ entries. My impression was that they are built in the current build root and outputs it's RPM packages into the current module.

That's correct.

AFAIK the only user of includemodules is base runtime now, which includes base runtime module in bootstrap module.

I got a nice pointer by @rdieter to bootstrap process defined inside packaging guidelines:
https://fedoraproject.org/wiki/Packaging:Guidelines#Bootstrapping Hence we could turn this issue into: "modular infrastructure supporting official bootstrap process".

This issue is not only about bootstrapping. If you build a module to provide a feature, you will try to disable other all unnecessary features. Otherwise your module will bloat and pollute public name space (i.e. will install files into standard paths). Therefore it's natural requirement to build different modules from the same sources with different features sets. Hence even without bootstrapping, there is a desire for defining macros at module level.

I will have to think about this a bit but I'd prefer setting macros for the entire module rather than per component; while it might look desirable at first, it would seriously slow down module builds and might also negatively affect component re-use. The reason is you would need to build the module macros package before and after each component that defines its own macros and wait for repo regen after each step. Furthermore having several components defining their own and possibly conflicting macros within one build group would effectively mandate the implementation of "sub build groups", with clearly defined behavior that everyone would need to study and understand first... I'd prefer keeping things smple.

Regarding with and without -- unless I'm mistaken and there's more magic to it, this is just a fancy way to define a macro as either 1 or 0, which is generally more flexible. Rather than adding new keywords for every RPM feature, I'm inclined to just add a module-wide custom macro definition block.

In case there is more to this feature, is there a way to make koji apply these anyway?

I'm not sure we will be able to set defines per component build. Koji allows us to only set that for whole module if I'm not mistaken. Maybe we could do some hacks to do that per component, but it won't be trivial. I will read more about what's possible in Koji - maybe there is a way I don't see.

My only concern is that doing per-package bconds and macro definitions is not trivial to implement and hence could take quite some time. On the other hand, if we could just define a set of macros to be available to all packages within a module build (= adding those macros into module-build-macros spec), that should be easy to implement. We can always improve this -- start small, improve over time.

Agreed.

I'm not sure we will be able to set defines per component build. Koji allows us to only set that for whole module if I'm not mistaken. Maybe we could do some hacks to do that per component, but it won't be trivial.

Yeah, this would be much easier (read: possible) to do for the entire module. Defining macros per-component would be a major undertaking.

Are module-level macros an acceptable solution?

I'm fine with module-level solution.

Summary from IRC:

• @nils is doing research on a component-level solution and we agreed not to block on it for now.
• @contyk will write the modulemd spec change for this.
• Once done, ping Factory 2 and we'll write the change into MBS to leverage the macro data, if present.

+1 on Ralph's summary.

+1 from me, too.

I'm onto something for faking component-level setting of macros, stay tuned (but don't hold the presses).

Can you uncover some base principle? :) I'm just curious as I don't see other options than per-component package with separate set of macros ...

@praiskup per-component macros doesn't seem to be feasible (unless @nphilipp figures something out): it would have horrible performance in koji as for every build of rpm package, a package with macro definitions would have to have been built.

For now all of us decided to create one rpm with macro definitions for the whole module build. With this approach you can define macros targeting specific rpm. It's really up to you to figure out the hierarchy.

Pavle, what is your concern here?

Sure, I just wanted to do some testing with mock first. :wink:

Brain dump:

• LUA in RPM can expand and (un)define macros from within a macro.
• The LUA code gets executed when the macro is expanded.
• ­→ We'll have to piggy-back onto a macro that is expanded early in the build process.
• → It also must be stable between different package builds in the same module because we can't read the previously set value of the macro from within the macro (or else: infinite recursion). I.e. expand the value at module-build-macros build time.
• The macro in question mustn't be overridden in a higher prio macro file from mock or koji, or another part of the build system.

It took me a while to find something suitable, %_topdir was my first serious candidate because it gets expanded by way of %__spec_*_pre → %___build_pre → … which is prepended to each of the %prep, %build, … scripts when building. Unfortunately, mock overrides this in /builddir/.rpmmacros, so putting it in /usr/lib/rpm/macros.d/macros.modules has no effect. Thankfully, there is %_buildrootdir which is used in %buildroot (which is also used in the same place) and that doesn't suffer from this limitation (and isn't overridden even if the user specifies BuildRoot: … in the spec file, I checked that :smile:).

Here's an example of a snippet we could add to /usr/lib/rpm/macros.d/macros.modules that should do the trick:

%_buildrootdir %{lua: \
  local name = rpm.expand("%name"); \
  if name == "one_package" then rpm.define("docs 1") \
  elseif name == "another_package" then rpm.define("docs 0") \
  end; \
  print("/builddir/build/BUILDROOT"); \
}

That last print line actually would use the %_buildrootdir macro as set during the build of module-build-macros package.

@praiskup per-component macros doesn't seem to be feasible (unless
@nphilipp figures something out): it would have horrible performance in
koji as for every build of rpm package, a package with macro definitions
would have to have been built.

Well, you still can have one rpm package with many sub-packages (say
module-macros-postgresql for postgresql). That package would be installed
just for the build of package 'postgresql'. That's the only one (and
pretty ugly) way of implementing that (unless we fix Koji to accept
--define).

Pavle, what is your concern here?

None. I'm fine with per-module macro file (at least now), I was just
curious what is the Nils' proposal.

@praiskup What would install that package to buildroot?

I've checked the above with a normal RPM build and mock. I'll give it another round with COPR just to be safe.

@jkaluza, I'm not sure what's the architecture, I'm just guessing ... :)

@nphilipp, yup, hooking the macro definition somewhere is (perhaps) possible, but I would be afraid of a lot of side-effects .. (maybe the %_buildrootdir macro is really the right and innocent place, but I've seen such hooks in epel-rpm-config package, and that's really maintenance hell...).
What if such "work-around" is installed twice?

@praiskup I know I make a couple of assumptions which need to be met if this is to work, but so far they seem to be holding up fine :wink::

• %_buildrootdir is never unset → setting it in a macro file is safe because we don't need to be compatible to it being unset
• %_buildrootdir is stable between component builds of the same module, i.e. /builddir/build in mock/koji.

I would be afraid of a lot of side-effects

The only possible remaining side-effect I can think of is that the lua code could be run multiple times and overwrite a macro that was modified in the spec file. Adding code so that it is only run once is simple, though. What else do you have in mind?

What if such "work-around" is installed twice?

You mean inadvertently installing module-build-macros in two versions? Or if there is another party which added their macro setting trickery using a similar hack? If it's the latter, then we have a use case for an interface in the build system that doesn't require us to pulling off stunts like this, don't you think?

module-build-macros in Koji will be always installed just once in the buildroot.

Good observation, I must say. While I'm not saying hack is safe and nice (you never know, and also debugging issues in such hacks are a bit "advanced" packaging), I don't see a weak point in %_buildrootdir hook now. The per-component macro definition definitely would be more beneficial. I would be leery to start the efforts on this hack personally, though that's not my responsibility.. So I'm +1. Perhaps macro experts @pmatilai or @tibbs could tell us their opinion.

But I have to say it: good solution IMO would be a native support for this in 'rpmbuild' (e.g. some predefined rule that if FOO.spec is built, then a macro file /some/nondefault-path/macros.FOO` is parsed first). So the behavior would be well defined.

BuildRoot: … in the spec file, I checked that 😄).

Redefining %buildroot by 'rpmbuild --buildroot ...` seems to break this hack.

The only possible remaining side-effect I can think of is that the lua code
could be run multiple times and overwrite a macro that was modified in the
spec file. Adding code so that it is only run once is simple, though. What
else do you have in mind?

Note that the lua code is evaluated twice even before the %name is
actually defined (so that rpm.expand('%name') expands as '%name' and thus you
have unexpected behavior). Is this guaranteed to be idempotent? (to debug, try
e.g. to call the io.stderr:write(name) statement.

If it's the latter, then we have a use case for an interface in the build
system that doesn't require us to pulling off stunts like this, don't you
think?

Uh, I can't parse this O:). I meant: What if e.g. other
initiative (say SCL or so) decided to use that same hack?

Uh, I can't parse this O:). I meant: What if e.g. other
initiative (say SCL or so) decided to use that same hack?

IOW, we should at least ensure that (a) if we agree that this is correct
and safe solution, then (b) we don't waste the "single possible hook point" for
one "privileged" initiative.

So instead of 'for %_buildrootdir call some_hardwired_if_else_hell' we should rather 'for %_buildrootdir call %some_plug-in_aware_mechanism'.

Thankfully, there is %_buildrootdir which is used in %buildroot (which is also used in the same place) and that doesn't suffer from this limitation (and isn't overridden even if the user specifies BuildRoot: … in the spec file, I checked that 😄).

I recommend you to talk to RPM developers to make sure that this isn't just an rpmbuild implementation detail that can stop work any time. I fear building a new build system on such fragile foundations.

I'm sorry for the multiple post. Pagure returned 500 on each submission, so my impression was it did not store any data.

Good observation, I must say. While I'm not saying hack is safe and nice (you never know, and also debugging issues in such hacks are a bit "advanced" packaging), I don't see a weak point in %_buildrootdir hook now.

I totally agree about this not being as safe and nice as I like. As you said, it's a hack :smile:.

The per-component macro definition definitely would be more beneficial. I would be leery to start the efforts on this hack personally, though that's not my responsibility.. So I'm +1. Perhaps macro experts @pmatilai or @tibbs could tell us their opinion.
But I have to say it: good solution IMO would be a native support for this in 'rpmbuild' (e.g. some predefined rule that if FOO.spec is built, then a macro file /some/nondefault-path/macros.FOO` is parsed first). So the behavior would be well defined.

IMO, the well-defined interface to pass in build options into rpmbuild are the --with-* and --without-* options, it's just that koji doesn't expose a way to set these for package builds (because so far there was no reason to do it). Enhancing rpmbuild to parse package-name-dependent macro files is pretty non-obvious and I'd much prefer that koji adds an official interface to do these settings in the context of e.g. module builds.

Redefining %buildroot by 'rpmbuild --buildroot ...` seems to break this hack.

Is this what COPR does? So far I didn't succeed with any of my test builds in https://copr.fedorainfracloud.org/coprs/nphilipp/module-component-build-macros/ even though local builds and mock builds worked for me.

Note that the lua code is evaluated twice even before the %name is
actually defined (so that rpm.expand('%name') expands as '%name' and thus you
have unexpected behavior). Is this guaranteed to be idempotent? (to debug, try
e.g. to call the io.stderr:write(name) statement.

Oh well, I guess I'll have to make it delay until %name is set… But that figures, where there's one hack, another usually isn't very far. :wink:

If it's the latter, then we have a use case for an interface in the build
system that doesn't require us to pulling off stunts like this, don't you
think?

Uh, I can't parse this O:). I meant: What if e.g. other
initiative (say SCL or so) decided to use that same hack?

I meant: this hack can only be effectively used by one party, and if more than one party see the need to use it (if it can be made to work reliably, see my COPR comment above), then it's a sign that we definitely have a convincing reason that koji (and whatever else is involved) give us a real interface.

So what I'm going to do "for now" is add a new toplevel block named buildopts with per content type subsections. I intend to introduce a simple string field in buildopts/rpms/macros that would hold any macro definitions that would be appended to the ones we have in module-build-macros today, verbatim. I think this is the most flexible approach and even allows for per package macro definitions using Nils' method, if you decide to do it.

Guys, the topmost thought on my mind after grokking what you're planning to do with %_buildrootdir is to prevent such hacks once and for all by making all the relevant macros read-only or something.

I'll much, much rather add a special-purpose hook into rpm to achieve whatever it is you want to get done, if that's what's needed. I just need to understand what it is that you really need - but if it's actually koji and fedpkg to understand rpm's --with/--without then I'm the wrong person to be talking to. Those options were originally intentionally omitted from Fedora build infra, but the world has moved on since those days...

@pmatilai:

Guys, the topmost thought on my mind after grokking what you're planning to do with %_buildrootdir is to prevent such hacks once and for all by making all the relevant macros read-only or something.

You're preaching to the choir as far as "this is a bloody hack and we'd rather have something clean" goes. My motivation to look for something like that was that there are several layers in between that don't give us a means to do that so we have to bypass all these layers at once.

I'll much, much rather add a special-purpose hook into rpm to achieve whatever it is you want to get done, if that's what's needed. I just need to understand what it is that you really need - but if it's actually koji and fedpkg to understand rpm's --with/--without then I'm the wrong person to be talking to. Those options were originally intentionally omitted from Fedora build infra, but the world has moved on since those days...

Exactly, there was no good reason to give access to these options back then (i.e. for normal package builds: just set the options according to %fedora or whatever), but in the context of modular builds we have a pretty convincing use case (IMO). I'm just concerned—even assuming buy-in from all involved parties and projects—that a clean way using all official interfaces will take very long not to implement per se, but to trickle down into the production instances.

@nphilipp, understood, and proof-of-concept hacks are ... well, proof-of-concept hacks.

My point is that I much rather add that one-liner
rpmExpand("%{_hook_your_strange_hacks_here}" call into rpm than see you go with overriding something as fundamental as buildroot going into production. epel-rpm-macros was noted to do something like this so maybe there is a more general use-case underneath.

Exactly, there was no good reason to give access to these options back then
(i.e. for normal package builds: just set the options according to %fedora or
whatever)

There are still valid reasons to not give us the access to those options; as
we want to have the builds reproducible -> so that buildroot only defines the
build-time options.

I have an implementation of a new buildopts section that allows you to specify your macros as a string, module-wide. However, @ppisar asked about module inclusion and whether we will inherit the included modules' buildopts, too.

My first response would be no -- just like we don't inherit their buildroot and we build their components in our own, we should also build them with our buildopts (only), but I'm willing to reconsider if there's a good enough reason to do this.

If you believe we should also include buildopts as well, could you give me an example of how you would use it and how negatively it would affect you if you didn't have that feature? I expect to push the change and make a new release tomorrow so be quick ;)

I pushed the abovementioned implementation in bf1f08a.

My use case for inheriting buildopts is this:

I have a modulemd file that requires buildopts to build. If I include the modemd file, it will not build because it will miss buildopts.

But the truth is I did not know that buildrequires are not inherited. In that case simple inclusion won't work either, and thus inheriting buildopts wouldn't be a big help. I can live with the top-level buildopts only.

The only downside is people including modulemd files will probably copy and paste buildopts from included modulemd files. And this copying means duplication and that means more difficult maintenance. It goes against idea of modules as a reusable units.

Thank you for your feedback.

Maybe they'll be copypasting, maybe not. Not inheriting the buildroot was a deliberate design decision, making it the including module's author's responsibility to provide a sufficient build environment for the included components, also allowing them to alter it to their liking. This is one of the key differences between module inclusion and module dependency. Not inheriting buildopts will feel consistent with current behavior.

I'll clarify this in the spec. I also consider this issue resolved.

Tracking MBS implementation at fm-orchestrator#539.