This is a huge change, but RelEng acks the change.

Lets discuss this in next RelEng meeting about the things that we need to do.

To clarify the expectations:

• we'd like to have a tag and target to build packages
• and a new disttag for these packages

The compose configuration is a stretch goal: we are going to build periodic composes through the ODCS service. For that we need a pungi configuration, but it probably can live in a separate repository.

So, there's a bunch of questions I have about implementation. :)

There's several ways we could setup the tag:

1. The tag is completely seperate from any other tag. This means you will need to add packages to it, likely import some already built small subset to get a working buildroot, curate adding/removing packages from it, all packages for it would have to be built in it, so you would need to mass rebuild everything if you wanted all packages. This is the least appealing to me, but you might want it if the new packages are not compatible with existing rawhide packages (ie, no mixing).

2. The tag is cloned from existing f32 tag. This means it has all the packages that exist as of the cloning time. You already have a buildroot (the f32 one). You can rebuild whatever packages you like in the tag and they are then used over the f32 ones. However, you would need to curate adding/removing packages as they are added/retired from rawhide, and what you want to do at branching is unknown (ignore it, but then you are based on f32. Create a new tag, but then all your builds are gone and need redoing, etc)

3. The tag is created from f32 with inheritence. This means you don't need to curate adding/removing packages, you just get the rawhide changes as they happen. You get a builroot (the f32 one) and you can build whatever subset of packages you like. What happens at branching is also a question. Could restart against the new branch or could change the inheritence so you inherit from f33 instead and keep all your f32 packages.

signing:

• No signing at all is easiest.
• Signing with a fixed key should be pretty easy, or signing with the current rawhide key should be reasonably easy.
  If the 'seperate tag' is used, this will mean a lot of work for signing server if you mass rebuild everything. I'd prefer to avoid that if possible.

arches:

Do you want all arches? some subset? which ones?

macro changes:

You will want to have your changes for things you build, are those just rpm macros? Otherwise I guess you need a rpm-macros package thats in the srpm/rpm buildroot for the tag. I think to avoid confusion, this might be a seperate package from the normal ones? If the changes are also simple rpm macros you may be able to set them in koji and avoid any macro package, althought it might be less clear whats going on then.

To make sure, composes, image creation and any testing/gating are all external right? Will you need storage space or anything for that?

There's likely more questions, but those are the ones off the top of my head... :)

@bookwar Can you respond on the comment above.

thanks.

So, there's a bunch of questions I have about implementation. :)
There's several ways we could setup the tag:

The tag is created from f32 with inheritence. This means you don't need to curate adding/removing packages, you just get the rawhide changes as they happen. You get a builroot (the f32 one) and you can build whatever subset of packages you like. What happens at branching is also a question. Could restart against the new branch or could change the inheritence so you inherit from f33 instead and keep all your f32 packages.

It seems to me that this should be a tag created from rawhide with inheritance initially. Ideally individual packages could turn off inheritance at the time that they choose, without impacting other packages.

signing:

No signing at all is easiest.
Signing with a fixed key should be pretty easy, or signing with the current rawhide key should be reasonably easy.
If the 'seperate tag' is used, this will mean a lot of work for signing server if you mass rebuild everything. I'd prefer to avoid that if possible.

I think it needs to be signed, I do not have an opinion as to which key is used initially.

0) For the base name for the tag we would like to use prefix eln. We will also use the disttag eln for rpm macro.

1) On the tag structure, we'd like to use gating (it is going to be implemented externally on the tooling side), thus we need a chain of tags:

eln-gate: empty tag          # packages get here when build
eln-candidate: empty tag     # packages are moved to this tag from the gate tag

eln-build:                 # buildroot tag which inherits from Fedora Rawhide
-> eln-candidate
-> f33-build

and the target:

Target: eln-candidate
  Build: eln-build
  Dest: eln-gate

Note 1: The actual names for the tags can be changed to align better with Fedora naming schema, only the structure is important.
Note 2: Empty -candidate tag doesn't mean we are going to rebuild everything in the tag to get started. We might just tag the relevant Rawhide packages manually into it. But we want to choose which packages to tag.

2) On signing: we would like to have it the same way as for Fedora Rawhide if possible. But it is not a critical requirement.

If we go with signing, then I guess the tags structure needs to be adjusted, could you help here? How it should look like with both gating and signing step?

3) On arches:
we would like to be able to build for all arches as soon as possible, but we can start with x86_64, and maybe ARM. If adding more architectures takes time, we'd better get going with what we can get.

Few more questions

1. Is it only x86_64 and armhfp, aarch64 later or any plans on power and s390x?
2. How long do you want the builds to stay around? (for koji gc policy)
3. Do you need them for F32 or F33 or both? And do they continue in the future?
4. It seems like you are might run some composes out of this tag, are they published somewhere and do you need any help on that?
5. Whats the end state for this change? And when?

Few more questions

Is it only x86_64 and armhfp, aarch64 later or any plans on power and s390x?

We want to have it for all architectures, but we wouldn't block oon availability of the architecture, so it can be added later.

How long do you want the builds to stay around? (for koji gc policy)

The latest couple of builds should be available. For older builds we don't need a longterm storage as we will have periodic compose snapshots which we will be managing outside of Koji.

Do you need them for F32 or F33 or both? And do they continue in the future?

We need only Fedora Rawhide. No branching is needed or planned. As soon as F33 branches, we should switch to f34/rawhide buildroot and so on.

It seems like you are might run some composes out of this tag, are they published somewhere and do you need any help on that?

We will run composes through ODCS infrastructure. Details will be figured later.

Whats the end state for this change? And when?

There is no "end". We want this buildroot to exist forever as an alternative buildroot for Fedora Rawhide. The scope and the content of the buildroot might change over time, but the infrastructure will stay the same.

@mohanboddu reports that he will work on this ticket today or tomorrow.

@bookwar @sgallagh Is this still valid with the new change https://fedoraproject.org/wiki/Changes/ELN_Buildroot_and_Compose ?

Yes, while https://fedoraproject.org/wiki/Changes/ELN_Buildroot_and_Compose changes the scope as we extend the application of the buildroot to a wider set of tasks, the request itself stays the same.

The only thing I would like to change is reversing eln-candidate tag and eln-gate tag (I think we should just call it eln tag). In general, we use -candidate tag to tag the builds once the package is built. Just to keep it consistence with others. And we will use -candidate tag to singing tag, once the build is tagged to eln-candidate tag, robosig will sign the package and tag it back into eln-candidate tag.

Updated schema:

eln-candidate: empty tag          # packages get here when build
eln: empty tag             # packages are moved to this tag from the gate tag

eln-build:             # buildroot tag which inherits from Fedora Rawhide
-> eln
-> f33-build
and the target:

Target: eln
  Build: eln-build
  Dest: eln-candidate

Looks good for me. I'd also prefer to keep it consistent.

Sounds good, I will start working on it once the change request is approved.

Since Mohan is going to start working on this once the Change is approved, can we firm up the answer to which architectures it should build on? Is the approach "build on all Fedora architectures"?

Since Mohan is going to start working on this once the Change is approved, can we firm up the answer to which architectures it should build on? Is the approach "build on all Fedora architectures"?

I think we should

Yeah, lets add it to all arches.

I think I missed that we hadn't answered this explicitly, but yes: the perspective of the ELN SIG is that we want to build it for all architectures, even those not supported by RHEL.

:thumbsup:

We will be trying this in stg first as discussed with @sgallagh

Added the tags in staging for now:

stg-koji add-tag eln --parent f33    # This inheritance will help not to add packages to eln tag
stg-koji add-tag eln-candidate --parent eln    # This is where builds will land after signing
stg-koji add-tag eln-pending --parent eln    # This is where builds will land after building and robosig will sign the packages from this tag.
stg-koji add-tag eln-build --parent eln --arches 'armv7hl i686 x86_64 aarch64 ppc64le s390x'    # eln buildroot 
stg-koji add-tag-inheritance eln-build f33-build --priority 5    # eln buildroot will have builds from eln tag and f33-build tag but builds in eln will be prioritized
stg-koji add-target eln eln-build eln-pending    # eln build target

FBR sent to infra list for enabling auto signing.
Please wait until the robosig config is deployed. I will update this ticket once its deployed.

Please let me know if any changes are required.

The robosig config is now deployed to stg.

Changes made to make it look like rawhide:

stg-koji add-tag --parent eln eln-updates
stg-koji add-tag --parent eln eln-updates-candidate
stg-koji add-tag --parent eln eln-updates-testing
stg-koji add-tag --parent eln-updates-testing eln-updates-testing-pending
stg-koji add-tag --parent eln eln-updates-pending
stg-koji add-tag --parent eln eln-override
stg-koji add-tag --parent eln-updates-testing-pending eln-signing-pending

stg-koji edit-target eln --dest-tag eln-updates-candidate

bodhi releases create --name "ELN" --long-name "Fedora ELN" --id-prefix FEDORA --version eln --branch eln --dist-tag eln --stable-tag eln --testing-tag eln-updates-testing --candidate-tag eln-updates-candidate --pending-stable-tag eln-updates-pending --pending-testing-tag eln-updates-testing-pending --pending-signing-tag eln-signing-pending --state pending --override-tag eln-override --create-automatic-updates --not-composed-by-bodhi --staging

# This is required for prod
koji edit-tag --perm autosign eln-updates-testing-pending

And also removed the unnecessary tags:

stg-koji remove-tag eln-pending
stg-koji remove-tag eln-candidate

Added eln-candidate build target:

stg-koji add-target eln-candidate eln-build eln-updates-candidate

When --release is provided in fedpkg build command it will look for eln-candidate build target or else --target should be provided to match eln build target.

Now both works

Closing this ticket as the work seems to be done and also deployed to prod:

https://pagure.io/releng/issue/9423