#955 F33 Change: ELN compose and disttag
Opened 9 months ago by bookwar. Modified 7 months ago

Change: https://fedoraproject.org/wiki/Changes/ELN_Buildroot_and_Compose

For setting up the ELN compose we'd like to use the new disttag (eln) which will differentiate rpms built for the compose from official ones. It will also allow us to tweak spec files of packages in Fedora Rawhide without branching them.

We'd like to get FPC approval to use the disttag and conditionals in Fedora Rawhide spec files

Note: this change updates the request in https://pagure.io/packaging-committee/issue/941


What does ELN stands for?

Anyway, I don't think this needs any approval from FPC since nobody will be relying on it if I understand correctly.

tweak spec files of packages in Fedora Rawhide without branching them

Do you have an example snippet of such tweak? Who will tweak the spec files? Will it be package maintainers willingly, or Red Hat provenpackagers will force conditionals to the specs without a review as in this particular nasty example (no public shaming intended)?

Do you have an example snippet of such tweak?

One example might be disabling the build of a docs subpackage to reduce the tree of dependencies.

The other example is the kernel tree. But afaik they currently do it differently, instead of doing "if eln" conditional, they set it as "if fedora". This way the eln tweaks are going to be inherited in epel and centos without changes. CC @dzickus

Does FPC have specific opinion on which approach is better?

Who will tweak the spec files? Will it be package maintainers willingly

ELN SIG may send a pull-request with the change to the package maintainer. It is up to package maintainer to approve it.

We would like FPC to clarify that such conditionals are generally allowed, so that we can have a conversation on the content of the change rather than on policy matters.

One example might be disabling the build of a docs subpackage to reduce the tree of dependencies.

Could you please post an example specfile snippet?

@churchyard - I know you sited a 'doc' example, but I am pro-actively adding a fedora kernel example. The fedora kernel is trying to merge with pre-RHEL kernel (ARK) and this disttag allows us to build two different kernels from the same source code/spec file. This eln disttag is critical to our merge (we announced this merge at FUDCON over a year ago).

As @bookwar said, we use 'fedora' for conditionals to limit our risk of dealing with the changing non-fedora disttag. Also the fedora kernel changes were agreed to and added by the fedora kernel maintainers. RHEL kernel folks would never blindly add changes without those maintainers approval.

Let us know, what we can do to move this along.

Example select correct config files
https://gitlab.com/cki-project/kernel-ark/-/blob/internal/redhat/kernel.spec.template#L65

Example don't enable s390x dumping
https://gitlab.com/cki-project/kernel-ark/-/blob/internal/redhat/kernel.spec.template#L326

Compiling tools on arches RHEL does not support
https://gitlab.com/cki-project/kernel-ark/-/blob/internal/redhat/kernel.spec.template#L420

Build dep for tool on Fedora builds
https://gitlab.com/cki-project/kernel-ark/-/blob/internal/redhat/kernel.spec.template#L473

I hope this helps!

Unfortunately, neither of the provided examples does anything with the eln distag mentioned in this ticket. The reason I ask for a specific example is not to see what can be accomplished with an %if (I've already seen and wrote code like %if 0%{?fedora}) but to actually see the %if with this new disttag.

That said, @bookwar and @sgallagh said on today's FESCo meeting that this ticket was probably opened too soon and that there might be changes based on the discussion that happens after https://fedoraproject.org/wiki/Changes/ELN_Buildroot_and_Compose is announced on devel mailing list.

@bookwar Shall this be closed for now and re-opened later?

@churchyard - Hmm, I am in a chicken and egg situation. Until we can agree on the new disttag, it is hard to implement in the spec file. :-(

Currently the fedora kernel/ARK has a disttag of .elrdy to separate things out (a legacy internal disttag). The goal is to utilize something to separate the two builds.

It makes me sad, we have to delay the release of this tag, as the kernel needs something sooner than later to meet our internal deadlines. :-(

Let me start from the beginning and hopefully things will make more sense. First, I'll explain a little about ELN. ELN is an evolution of the request for an alternate buildroot for newer x86_64 processors. The reasoning behind that new buildroot was that we expected that the next major release of RHEL would likely drop support for older hardware and therefore could take advantage of enhancements and processor extensions available for newer hardware. As plans for this proceeded, they expanded into a desire to do more than just test out the processor architecture. Instead, we want to have a complete alternative compose of Fedora Rawhide that resembles the way that Red Hat and CentOS builds their packages. The idea being that Fedora developers and third-party vendors who rely on Red Hat Enterprise Linux have a place where they can directly contribute to what will eventually become the next RHEL.

ELN (Enterprise Linux Next) is going to be that place. What we want to do is establish a set of RPM conditionals that can be used for the set of packages that may need to be built differently for a RHEL-like target as compared to a Fedora one. (For example, we may want to skip regenerating documentation during package-build and instead use pre-built docs from the tarball or SRPM.)

We plan to have the ELN environment built with a few modifications to the existing macro for %{dist} (and relatives) macros. Specifically:

  • %{dist} will return .eln (no numerical suffix)
  • %{fedora} will return XX (where XX is the Fedora version represented by Rawhide).
  • %{rhel} will be set and will return a number one higher than the most recent major release of Red Hat Enterprise Linux (at present, that will be 9).
  • %{eln} will be set and will expand to %{rhel}

The reasoning behind this approach is so that we break as little as possible when we implement this new build and compose. Existing packages that are using conditionals such as if 0%{?rhel} > 7 will transition over to building "like RHEL" automatically. Any packages that do not have a shared Fedora/RHEL specfile will continue to build as normal. Some unknown number of packages that rely on if ! 0%{?fedora} will require manual intervention.

The %{eln} variable will be set primarily to allow for the possibility of needing conditionals that apply to ELN only and should not carry over to RHEL, but we expect its use to be exceptionally rare.

What we are asking for from FPC at this time is feedback on the plan I just described. In particular, what pitfalls do you see that we may not have accounted for? We want to get to a point where our team can submit patches for the Fedora Packaging Guidelines to move forward on this.

Edit: fixed a mistake describing the implementation of %{fedora}.

Hmm, I am in a chicken and egg situation. Until we can agree on the new disttag, it is hard to implement in the spec file. :-(

I don't want an implemented spec file, I just wanted an example snippet about how do you imagine to write a conditional based on dist tag. @sgallagh's answer kinda shows that you don't (you either use %{?rhel} or %{?eln}, but not the dist tag).

@sgallagh Thanks for more info.

%{fedora} normally returns a number. How will .fcXX compare to a number? Should it just return a number as normally?

what pitfalls do you see that we may not have accounted for?

I can imagine that some conditionals will break if both %{fedora} and %{rhel} are defined and > 0. For example:

rubygem-rake-compiler.spec
rubygem-locale.spec
%if %{?fedora:0%{fedora} >= 17}%{?rhel:0%{rhel} >= 7}

Macaulay2.spec
%global ISSUE %{?fedora:Fedora-%{fedora}}%{?rhel:RedHatEnterprise-%{rhel}}

mariadb.spec
-DRPM="%{?rhel:rhel%{rhel}}%{!?rhel:fedora%{fedora}}"

Also %if %fedora ... %else ... assume RHEL here ... %endif will no longer work in all cases.

What I worry about from RHEL point of view is that the packages will be built with %{fedora} set. Unless future RHEL also has this set, upon importing to RHEL, the packages will suddenly build without %{fedora} set and hence they will produce different results.

As a side note, the fact that .eln sorts lower to .fcXX might be a problem, although so does .el9. The fact that .el9 sorts lower than .eln might also be a problem in the future, depending on what you intent to do with the compose.

Hmm, I am in a chicken and egg situation. Until we can agree on the new disttag, it is hard to implement in the spec file. :-(

I don't want an implemented spec file, I just wanted an example snippet about how do you imagine to write a conditional based on dist tag. @sgallagh's answer kinda shows that you don't (you either use %{?rhel} or %{?eln}, but not the dist tag.
@sgallagh Thanks for more info.
%{fedora} normally returns a number. How will .fcXX compare to a number? Should it just return a number as normally?

That was a mistake when typing it up. It should just return the number. I'll edit the comment above to fix that.

what pitfalls do you see that we may not have accounted for?

I can imagine that some conditionals will break if both %{fedora} and %{rhel} are defined and > 0. For example:
rubygem-rake-compiler.spec
rubygem-locale.spec
%if %{?fedora:0%{fedora} >= 17}%{?rhel:0%{rhel} >= 7}

Macaulay2.spec
%global ISSUE %{?fedora:Fedora-%{fedora}}%{?rhel:RedHatEnterprise-%{rhel}}

mariadb.spec
-DRPM="%{?rhel:rhel%{rhel}}%{!?rhel:fedora%{fedora}}"

Yeah, I'm sure there will be some of those, but I can't come up with a better answer offhand. Suggestions welcome!

Also %if %fedora ... %else ... assume RHEL here ... %endif will no longer work in all cases.
What I worry about from RHEL point of view is that the packages will be built with %{fedora} set. Unless future RHEL also has this set, upon importing to RHEL, the packages will suddenly build without %{fedora} set and hence they will produce different results.

Yeah, we will need to do some analysis and figure out what packages are likely to have problems here. Not all of them will FTBFS, so they may be subtle.

As a side note, the fact that .eln sorts lower to .fcXX might be a problem, although so does .el9. The fact that .el9 sorts lower than .eln might also be a problem in the future, depending on what you intent to do with the compose.

The ELN compose won't be shipping to end-users, so I don't think that's much of an issue. We also will absolutely not be supporting upgrading from ELN to RHEL, so I doubt that will matter either.

One additional point: there is some debate around the %{fedora} variable. Some packages will be negatively affected by having both %{fedora} and %{rhel} set, so we are also considering the possibility that we might unset %{fedora} in ELN. That will also impact a different set of packages and we're not sure which is the larger group. Any insights you might have would be desirable.

Problematic cases with both present

  • if 0%{?fedora} else #assume RHEL will incorrectly behave as Fedora

Problematic cases with %{fedora} unset

  • if 0%{?fedora} < 34 || 0%{?rhel} >= 9

I can imagine that some conditionals will break if both %{fedora} and %{rhel} are defined and > 0. For example:
rubygem-rake-compiler.spec
rubygem-locale.spec
%if %{?fedora:0%{fedora} >= 17}%{?rhel:0%{rhel} >= 7}

With both set:
It will return %if 03309 which would resolve as true.

With only RHEL set:
It will return %if 09 which would resolve as true.

In both cases, this would pass the conditional. So I think we're fine here.

Macaulay2.spec
%global ISSUE %{?fedora:Fedora-%{fedora}}%{?rhel:RedHatEnterprise-%{rhel}}

With both set:
This one would report Fedora-33. That said, this is clearly a case where they'll want to change the conditional no matter which way we go.

With only RHEL set:
This one would report RedHatEnterprise-9.

This one changes behavior and I think we'd need to tweak it either way to make sure we got the results we want. (I don't know what that is for this package currently).

mariadb.spec
-DRPM="%{?rhel:rhel%{rhel}}%{!?rhel:fedora%{fedora}}"

With both set:
This one will return -DRPM=rhel, which I think is what we want here, but I don't know the packaging for this very well.

With only RHEL set:
Same as above

Also %if %fedora ... %else ... assume RHEL here ... %endif will no longer work in all cases.

Yeah, that's a definite problematic case if we have both set. If Fedora is unset, this continues to work.

My problem is that I don't know how common each of the forms of these conditionals are. Unfortunately we don't really have any kind of standardization for them. And I don't think we could realistically write macros to do it.

What I worry about from RHEL point of view is that the packages will be built with %{fedora} set. Unless future RHEL also has this set, upon importing to RHEL, the packages will suddenly build without %{fedora} set and hence they will produce different results.

Right, this is something we need to watch out for. Changes will have to happen in some subset of packages; I just don't know which set to pick.

The main problem are snippets like this (which I genuinely believe are hundreds in Fedora, albeit not always with bconds):

%if 0%{?fedora}
%bcond_without docs
%bcond_without tests
%else
# We don't have the dependencies
%bcond_with docs
%bcond_with tests
%endif

What happens there? In ELN the package is built with tests and docs. But later in RHEL, without.

What happens there? In ELN the package is built with tests and docs. But later in RHEL, without.

Yes, this is definitely a concern. However, we can't possibly know all of these cases ahead of time. The advantage we have is that most of the packages that eventually end up in RHEL are maintained by the same person in both Fedora and RHEL, so our hope is that the maintainers will, with appropriate guidance, notice these changes and update them appropriately.

Also, "later" in RHEL will not be months like in previous releases. We (@bookwar and I) are working on having internal builds starting within the next two months, so we should have plenty of runway to address those packages that end up building differently.

So with my FPC hat on, I think being able to do ifs for %{eln} should be allowed.

Speaking of guidelines, it means extending https://docs.fedoraproject.org/en-US/packaging-guidelines/DistTag/#_conditionals

So with my FPC hat on, I think being able to do ifs for %{eln} should be allowed.

Allowed, yes. I think it should be discouraged, however. I'd even go so far as to say any such usages should need approval from FPC.

Speaking of guidelines, it means extending https://docs.fedoraproject.org/en-US/packaging-guidelines/DistTag/#_conditionals

Yes. Once we nail down these open questions , I will propose changes to that page.

Do you have any insight into whether we should defining %{fedora} or unsetting it in ELN is the better approach? As noted above, both approaches will face some difficulties.

Defining %{fedora} will make it work in most cases, being close to rawhide.
Not defining %{fedora} will make it closer to next RHEL, and will require immediate changes in the conditionals.

What to do depends mostly on the use case of the ELN thing. If the idea is to make the rawhide specfiles conditionals more "ready" for next RHEL, not defining it makes more sense, however will require enormous work for packages that will eventually not even be part of RHEL.

I'm sorry I didn't follow this conversation until now. So my comments will go back quite a ways. If we implement what Stephen has above, we will have a mashup of neither Fedora, nor RHEL builds. The problem is these two lines.

  • %{fedora} will return XX (where XX is the Fedora version represented by Rawhide).
  • %{rhel} will be set and will return a number one higher than the most recent major release of Red Hat Enterprise Linux (at present, that will be 9).

If you set BOTH %{fedora} AND %{rhel} to be a number, you are asking for trouble. When you build in Fedora, %{rhel} is set to 0, and we get what we expect to be Fedora. When you build in RHEL or EPEL, %{fedora} is set, and you get what you expect to be RHEL or EPEL. When you set them both, then all bets are off for what you get.

In your example

  • if 0%{?fedora} < 34 || 0%{?rhel} >= 9

This is broken right now. This is an example of a broken %if statement that I've been working for years to get out of all the spec files. Why is it broken? If I build that in EPEL 6, where %{fedora} is 0, then it passes, and it's going to utilize whatever is in the %if statement.

It's also a rather crazy example.

Please don't set both fedora and rhel. You are going to regret it. It will cause much much more work than if you set one or the other.

If you want to argue for it, then please give some real world examples why you want it.

And I believe, for everyone one example you give, I can give you four examples (used in hundred of packages) of why it will cause things to behave different.

Defining %{fedora} will make it work in most cases, being close to rawhide.
Not defining %{fedora} will make it closer to next RHEL, and will require immediate changes in the conditionals.

I totally agree with you. What is it we want to do with ELN. If we want it to look like RHEL, we need to unset %{fedora}

What to do depends mostly on the use case of the ELN thing. If the idea is to make the rawhide specfiles conditionals more "ready" for next RHEL, not defining it makes more sense, however will require enormous work for packages that will eventually not even be part of RHEL.

I agree with everything, but the very last part, that it will require enormous work for packages that will not even be part of RHEL. Or maybe I'm misunderstanding.

If we use the example 0ad, which will never be part of RHEL, from what I see, that packager doesn't have to do anything.

I agree, defining both %{fedora} AND %{rhel} doesn't make any sense here and is only going to lead to trouble.

I can say from an ARK kernel point of view, we were very much expecting %if 0%{?fedora} as an identifier to make ELN build differently. I suppose that we can work around that, but ideally we need something that would work make ELN and rhel build one way, and fedora build another. We don't need to differentiate between RHEL and ELN, we only need to differentiate between those and Fedora.

Defining %{fedora} will make it work in most cases, being close to rawhide.
Not defining %{fedora} will make it closer to next RHEL, and will require immediate changes in the conditionals.
What to do depends mostly on the use case of the ELN thing. If the idea is to make the rawhide specfiles conditionals more "ready" for next RHEL, not defining it makes more sense, however will require enormous work for packages that will eventually not even be part of RHEL.

@churchyard Can you provide more detail on why you think this would require a lot more work? Also, (and I realize this was unclear at the time you wrote this), we don't intend for ELN to reproduce ALL of Fedora; it will be a subset that is much closer to what we want to eventually have in RHEL. Does that change the calculus here?

To be clear, the more I think on this, the more I'm coming down on the side of "We should just unset the %{fedora} variable in ELN", but I want to know what we're getting into if we do.

To be clear, the more I think on this, the more I'm coming down on the side of "We should just unset the %{fedora} variable in ELN", but I want to know what we're getting into if we do.

@churchyard Can you provide more detail on why you think this would require a lot more work? Also, (and I realize this was unclear at the time you wrote this), we don't intend for ELN to reproduce ALL of Fedora; it will be a subset that is much closer to what we want to eventually have in RHEL. Does that change the calculus here?

It certainly does.

But here is the detail:

  • if you keep %fedora defined, everything will most likely build as is (except not very common cases) in ELN, but later it will need to be adapted for RHEL 9
  • if you undefine %fedora, lot of existing conditionals will need to be adapted righ away

For example:

%if 0%{?fedora}

Will need to be adapted to:

%if 0%{?fedora} || 0%{?rhel} >= 9

My (no longer valid) concern was that the ELN bootstrap is much bigger than RHEL bootstrap and the number of affected packages that would require changes like this would be enormous. But with the recent info, I think it is better to adapt this during ELN bootstrap than RHEL9 bootstrap.

OK, while I share @churchyard's hesitation around breaking existing conditionals, I think that unsetting %{fedora} is probably the correct choice. We'll just have to fix any breakage it causes early (which is a good idea in general).

I will amend the Change Proposal.

I agree that unsetting %fedora is probably the correct choice.

I can say from an ARK kernel point of view, we were very much expecting %if 0%{?fedora} as an identifier to make ELN build differently.

Testing for the eln macro would do that, as would testing if rhel is set to the N+1 version.

Note that without fedora macro set I'd assume ARK would build as it does on RHEL, which...

I suppose that we can work around that, but ideally we need something that would work make ELN and rhel build one way, and fedora build another. We don't need to differentiate between RHEL and ELN, we only need to differentiate between those and Fedora.

So do you just want ARK to build differently than fedora? And do you think that won't happen automatically?

@bookwar is there anything to do here? I think ELN stuff has moved forward so I guess all problems were solved by now that are relevant to this ticket. Am I right? I guess the guidelines now are missing some examples of %{eln} but you are welcome to send a PR.

I think the guidelines should not actually mention %{eln}. It's there for the rare case when it's necessary to differentiate ELN from RHEL/CentOS, but in general we don't want people to use it.

In my experience, when we want people not to do something, we should explicitly say that in the guidelines, or they will eventually just start doing it.

In my experience, when we want people not to do something, we should explicitly say that in the guidelines, or they will eventually just start doing it.

I agree.

Additionally, if there's a legitimate reason to use it, then it would probably help to have this usage somewhat standardized, even if we surround the documentation with "do only use this if you really really need it, but if you do, then here's how".

OK, I can get behind that. I'll work something up next week.

OK, I actually did come up with a legitimate reason to use it that I hadn't considered previously. There are some packages that have different names in RHEL as compared to Fedora (for example freeipa is called idm in RHEL).

So using %{?eln} to establish dependencies that are different between ELN and RHEL makes sense (since we probably don't want to be doing the renaming in ELN).

Well, assuming RHEL 9 will move to the new package name, it might be worth using %if 0%{?el8} instead for such a scenario.

Well, assuming RHEL 9 will move to the new package name, it might be worth using %if 0%{?el8} instead for such a scenario.

Sorry, I don't understand how that follows? I'm talking about making the assumption that when the package moves into RHEL 9, it will be renamed to the "RHEL name" and we want to avoid having to change the conditional when that happens. So we would have something like:

%if 0%{fedora} || 0%{eln}
Requires: freeipa-server
%else
# Assume RHEL
Requires: idm-server
%endif

Or in the case of rpm-ostree:

%if 0%{?fedora} || 0%{?eln}
BuildRequires: cargo
BuildRequires: rust
%else
BuildRequires: rust-toolset
%endif

Which has the cargo and rust packages, but in RHEL we need to pull in the rust-toolset package because it also provides some RPM macros.

Why would not it be:

%if 0%{?rhel}
# Assume RHEL
%else
# Assume anything else that does not rename packages
%endif

Why would not it be:
%if 0%{?rhel}

Assume RHEL

%else

Assume anything else that does not rename packages

%endif

Because ELN sets %{rhel} equal to "one higher than the most recently released RHEL". So on ELN, 0%{?rhel} evaluates to 09 currently.

Shouldn't idm-server provide freeipa-server instead?

To clarify: when building for ELN, %{fedora} is unset and %{rhel} is set so that we can simulate building in Fedora close to what we would be building in RHEL. There are some edge cases to deal with such as the rust-toolset one where that package doesn't exist in Fedora.

Shouldn't idm-server provide freeipa-server instead?

That's another option for that specific case, sure. I conjured that one out of thin air, but the rpm-ostree situation is more subtle.

Hmm, I suppose that in Fedora, cargo (which Requires: rust) could Provides: rust-toolset and then we could just use the latter unconditionally. That seems... wrong somehow though.

I am confused. I thought the idea was to make Fedora and RHEL more similar. Anyway, bac on topic:

I think we should document:

  • packages MUST NOT use %eln conditionals
  • exceptions to this rule MUST be discussed with the ELN SIG at (where?)
  • packages with exceptions SHOULD use %eln in this way: ...

Alternatively, turn the MUSTs to SHOULDs.

I am confused. I thought the idea was to make Fedora and RHEL more similar.

It absolutely is. However, there are some places where the process is disjoint and I'm trying to find the path of least disruption to Fedora. I don't want to ask dependencies to start doing Provides: <rhelname> or I'm quickly going to turn people against this effort.

Anyway, bac on topic:
I think we should document:

packages MUST NOT use %eln conditionals
exceptions to this rule MUST be discussed with the ELN SIG at (where?)
packages with exceptions SHOULD use %eln in this way: ...

Yes, I think that makes sense. As for where, ELN SIG would prefer to just use fedora-devel for communication to stay visible. Maybe just ask for messages relevant for this to be tagged with [ELN] in the subject line?

In that case, IPA should do:

%if 0%{?rhel} && !%{?eln}

not the other way around.

In that case, IPA should do:
%if 0%{?rhel} && !%{?eln}
not the other way around.

I can work with that.

@sgallagh if you document what @churchyard proposed above with the rule that people should use the rhel/eln conditions for such specific cases and not other way around, that will make me happy and vote +1 :)

Login to comment on this ticket.

Metadata