#8294 coreos-pool and coreos-release koji tags request for FCOS
Closed: Fixed 2 years ago by mohanboddu. Opened 2 years ago by sinnykumari.

Describe the issue

Fedora CoreOS will have various streams. All streams will use packages which are built in Fedora koji. Mechanical streams will use packages available in Fedora repos from tags f$(release), f$(release)-updates, f$(release)-updates-testing. Development and production streams will get built from packages available under koji tag coreos-pool. Packages which make it to production will also get tagged to coreos-release tag.

These tags will ensure that (1) packages are not automatically garbage collected (2) stream builds are reproducible (up to the GC retention policy we agree upon), and (3) packages are added to the pool (and thus into the production streams) in a controlled manner.

To build FCOS for different streams we need:
- coreos-pool and coreos-release koji tags created (include arches aarch64, ppc64le, x86_64)
- Permission to add packages to requested tags (ACLs for bgilbert, dustymabe, jlebon and sinnykumari, until we have bot setup done)
- Tag builds needs to be signed
- Generate dist repo for requested tags with option --non-latest to koji dist-repo (to perform build with desired NVRA)

Question

As we have streams like next-devel and testing-devel, packages will be from multiple Fedora releases in coreos-pool and coreos-release tags. Since we will be using signed packages, does koji dist-repo allows to specify multiple keys for generating dist repos? This question is because, in infra ansible tag2distrepo, so far single key has been used.

When do you need this? (YYYY/MM/DD)

ASAP

When is this no longer needed or useful? (YYYY/MM/DD)

Always needed

If we cannot complete your request, what is the impact?

we won't be able to build FCOS for different streams


@ausil @rdossant Do we need to include s390x here?

@ausil @rdossant Do we need to include s390x here?

I'd prefer to start with the three architectures we're already targeting and succeed there before we add s390 personally. It will be hard enough just to pull it together on those three. Once we work out the kinks then it shouldn't be a big deal adding a 4th arch.

@sinnykumari yes we need to, we are starting tow ork on s390x now in addition to aarch64 and ppc64le

@mohanboddu Any additional info needed or we are good to proceed with this request?

This will be discussed in Fedora RelEng meeting tomorrow Apr 24 at 16:00 UTC

Metadata Update from @mohanboddu:
- Issue tagged with: meeting

2 years ago

Some questions:

  • You would tag all the packages for a compose into coreos-pool and all the ones in a released compose into coreos-release. Could you perhaps just use fedora GA/fedora-updates tags and only tag in the builds that are not in those and use those 2 repos on top of coreos-pool/coreos-release? That would make them much smaller and more manageable. Of course the fedora ones would be signed with fedora keys.

  • What keys should these tags builds be signed with? If with the fedora ones, since they aren't versioned how can we tell what key to use?

If we have to sign a large pile of packages with a new key often I am a bit worried at our signing bandwith.

Can we use fxx-coreos-release and fxx-coreos-pool tags which follow a Fedora release and can make use of same fedora release keys?

Another thought we had was to version the tags? ie, coreos-pool-30, coreos-pool-31, etc... and we could then just sign them with the fedora keys and just always make them when we branch.

But is everything being signed by a different key a requrement?

Can you re-use the existing fedora GA/updates repos signed with the fedora key in addition to these? or does it need to be one dist-repo?

We just discussed this ticket in the releng meeting today. Here is what I think we decided on:

  • we create a coreos-pool tag - this is an aggregate of all the packages used to build the various streams of FCOS. The package NVRAs for each build will be picked specifically using input to the rpm-ostree build tooling.
  • we create a coreos-release tag - this tag is simply for us to tag what rpms we have actually shipped to end users in FCOS. There are no consumers of this tag. It's purpose is to have a more conservative garbage collection policy on a smaller subset of packages than what is in coreos-pool
  • we create a fxx-coreos-signing-pending tag for each major version of fedora. When we build backports rpms will go into this tag and robosignatory will then sign the rpms and tag them into the coreos-pool tag and untag them from fxx-coreos-signing-pending.

Thanks all for the valuable input during releng meeting ! Unless we have any unanswered questions, can we get tags finalized in https://pagure.io/releng/issue/8294#comment-567284 and dist repo for coreos-pool tags with option --non-latest created?

@sinnykumari Can it wait until Wed?, since this requires some koji hub policy changes and robosig changes, since we are in infra freeze we can make the changes once the freeze lifts off.

Made the necessary changes to koji hub policy and robosignatory config and added the koji tags needed.

$ koji add-tag coreos-pool
$ koji add-tag coreos-pool --parent=coreos-release
$ koji add-tag f30-coreos-signing-pending --parent=coreos-pool
$ koji list-tag-inheritance coreos-release --reverse
coreos-release (8633)
  └─coreos-pool (8632)
     └─f30-coreos-signing-pending (8634)

hey @mohanboddu - thanks for working on this. Quick question related to the implementation:

From the releng meeting (last week, I think) we discussed this and I think we had ended up deciding to not use koji inheritance for any of these tags. My understanding is that we should have coreos-release, coreos-pool, and f30-coreos-signing-pending all be separate with no inheritance.

hey @mohanboddu - thanks for working on this. Quick question related to the implementation:
From the releng meeting (last week, I think) we discussed this and I think we had ended up deciding to not use koji inheritance for any of these tags. My understanding is that we should have coreos-release, coreos-pool, and f30-coreos-signing-pending all be separate with no inheritance.

@dustymabe The reason why I thought inheritance is, you dont need to add packages multiple times. Once you add the package to coreos-release they will be available for all other tags. And you can compose either from coreos-release or coreos-pool.

Now that I am thinking out loud, coreos-pool will pick all the builds in coreos-release, which I think might not be desirable. Can you confirm this and I can remove the inheritance between coreos-pool and coreos-release.

But I think we should leave the coreos-pool and fxx-coreos-signing-pending inheritance as it is, since you can add a package to coreos-pool and it will be available in fxx-core-signing-pending and if they are separate you need to add the package to both tags and also we need to change koji hub permissions as well. Since fxx-coreos-signing-pending is just a holding tag, it wont be of a problem.

@dustymabe The reason why I thought inheritance is, you dont need to add packages multiple times. Once you add the package to coreos-release they will be available for all other tags. And you can compose either from coreos-release or coreos-pool.

Now that I am thinking out loud, coreos-pool will pick all the builds in coreos-release, which I think might not be desirable. Can you confirm this and I can remove the inheritance between coreos-pool and coreos-release.

We aren't planning on composing from coreos-release. Just from coreos-pool. The rationale for the need for coreos-release is in my earlier comment. Yes, please remove the inheritance. Also note that we only need one distrepo set up and that is against coreos-pool.

But I think we should leave the coreos-pool and fxx-coreos-signing-pending inheritance as it is, since you can add a package to coreos-pool and it will be available in fxx-core-signing-pending and if they are separate you need to add the package to both tags and also we need to change koji hub permissions as well. Since fxx-coreos-signing-pending is just a holding tag, it wont be of a problem.

My understanding is that the package goes to fxx-coreos-signing-pending when it is built and then robosig moves it to coreos-pool when the package has been signed. I don't think we need inheritance there? What would be the benefit?

We aren't planning on composing from coreos-release. Just from coreos-pool. The rationale for the need for coreos-release is in my earlier comment. Yes, please remove the inheritance. Also note that we only need one distrepo set up and that is against coreos-pool.

I will remove the inheritance here.

My understanding is that the package goes to fxx-coreos-signing-pending when it is built and then robosig moves it to coreos-pool when the package has been signed. I don't think we need inheritance there? What would be the benefit?

Without the inheritance, you need to add the package to both the coreos-pool and fxx-coreos-signing-pending tags, just like you did here since both the tags have no idea about the packages. Using inheritance, thats not needed and once the build gets signed it will moved to coreos-pool tag from fxx-coreos-signing-pending tag.

Removed the tag inheritance between coreos-release and coreos-pool

$ koji remove-tag-inheritance coreos-pool coreos-release

Without the inheritance, you need to add the package to both the coreos-pool and fxx-coreos-signing-pending tags, just like you did here since both the tags have no idea about the packages. Using inheritance, thats not needed and once the build gets signed it will moved to coreos-pool tag from fxx-coreos-signing-pending tag.

OK. One or two questions here:

  • is there any chance for an unsigned build to get into the resulting dist-repo?
  • once we have a f31-coreos-signing-pending will this break? We need this setup to work equally for f30-coreos-signing-pending and f31-coreos-signing-pending at the same time.

OK. One or two questions here:

is there any chance for an unsigned build to get into the resulting dist-repo?

Nope

once we have a f31-coreos-signing-pending will this break? We need this setup to work equally for f30-coreos-signing-pending and f31-coreos-signing-pending at the same time.

I think this will make a perfect choice then, since we can add f31-coreos-signing-pending tag to the inheritance and both f30-c-s-p and f31-c-s-p will work.

@mohanboddu - can you link me to the distrepo for coreos-pool once you've set it up?

@mohanboddu - can you link me to the distrepo for coreos-pool once you've set it up?

@dustymabe What do you mean? You can tag a build to f30-coreos-signing-pending, and once signed, it will be available in coreos-pool tag and I think currently tag2distrepo is still not working, so after tagging a build please let me know and I can run the koji dist-repo for you.

@mohanboddu - can you link me to the distrepo for coreos-pool once you've set it up?

@dustymabe What do you mean? You can tag a build to f30-coreos-signing-pending, and once signed, it will be available in coreos-pool tag and I think currently tag2distrepo is still not working, so after tagging a build please let me know and I can run the koji dist-repo for you.

As per @mizdebsk FBR, automatic dist-repo generation with tag2distrepo doesn't work only in case of unsigned packages get tagged. Since, coreos-pool uses signed packages, it should be fine.

As per @mizdebsk FBR, automatic dist-repo generation with tag2distrepo doesn't work only in case of unsigned packages get tagged. Since, coreos-pool uses signed packages, it should be fine.

Ah nvm then, this patch should fix this and I completed running the playbook.

Just to clear up re. that patch,

+ 'coreos-pool': [cfc659b9],

When we're going to be sourcing from both f30 and f31 versions of the signing pending tags, we'd just have to list both keys here, right?

(Also shouldn't that keyid be in quotes?)

Just to clear up re. that patch,
+ 'coreos-pool': [cfc659b9],
When we're going to be sourcing from both f30 and f31 versions of the signing pending tags, we'd just have to list both keys here, right?

Yes

(Also shouldn't that keyid be in quotes?)

Thanks, I will rerun the playbook fixing it :disappointed:

To check necessary permissions, I did:

$ koji add-pkg coreos-pool rpm-ostree --owner sinnykumari
$ koji tag-build coreos-pool  rpm-ostree-2019.3-1.fc30

It went fine, but I don't see a coreos-pool dist-repo being created at https://kojipkgs.fedoraproject.org/repos-dist/ . What are we missing here?

I restarted fedmsg-hub-3 on bodhi-backend02, it looks like it might not have picked up the new config yet.

Added the arches to coreos-pool tag

$ koji edit-tag coreos-pool --arches='aarch64 ppc64le s390x x86_64'

I am guessing its not required for coreos-release since it will just hold the builds from the released FCOS. Please let me know if I am wrong.

Did few more checks to make sure we have everything inplace:

  • coreos-pool dist-repo allows to have multiple version of same package. Added openssh-7.9p1-5.fc30 and openssh-8.0p1-1.fc30 and we do have packages from both nvr available in latst generated dist-repo.
  • Can add packages to coreos-release tag (tested by adding rpm-ostree)
  • Can tag-build and untag-build on specific pakcages in coreos-release tag (tested with rpm-ostree-2019.3-1.fc30)

I think we have all requested items in this ticket done. Did we miss anything before we close this ticket?

Added the arches to coreos-pool tag
$ koji edit-tag coreos-pool --arches='aarch64 ppc64le s390x x86_64'

I am guessing its not required for coreos-release since it will just hold the builds from the released FCOS. Please let me know if I am wrong.

@sinnykumari Do you know answer for this ^?

Did we miss anything before we close this ticket?

One thing that is missing is special permissions surrounding tagging for the kernel. Only people who are allowed to build the kernel can tag it apparently. We need to be able to tag the kernel into our various tags using a bot. How can we achieve this goal?

Added the arches to coreos-pool tag
$ koji edit-tag coreos-pool --arches='aarch64 ppc64le s390x x86_64'

I am guessing its not required for coreos-release since it will just hold the builds from the released FCOS. Please let me know if I am wrong.

That's correct, we don't need this since coreos-release tag is only to tag specific nvr builds which will make it to the FCOS release. we won't creating a dist-repo out of it.

Did we miss anything before we close this ticket?

One thing that is missing is special permissions surrounding tagging for the kernel. Only people who are allowed to build the kernel can tag it apparently. We need to be able to tag the kernel into our various tags using a bot. How can we achieve this goal?

@dustymabe This commit should fix that issue.

It worked!

[dustymabe@hattop ~]$ koji tag-build coreos-pool kernel-5.0.11-300.fc30                                 
Created task 34767868
Watching tasks (this may be safely interrupted)...
34767868 tagBuild (noarch): free
34767868 tagBuild (noarch): free -> closed
  0 free  0 open  1 done  0 failed

34767868 tagBuild (noarch) completed successfully

We think we got everything we need here.

Closing the ticket.

Metadata Update from @mohanboddu:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

A couple additional requests:

  • can we get a few more signing-pending tags (maybe we can add this to a branching SOP somewhere) that have the same setup (inherits from coreos-pool) as the f30-coreos-signing-pending tag?
    • f29-coreos-signing-pending
    • f31-coreos-signing-pending
  • right now the distrepo config for coreos-pool has the f29 and f30 keys. Can we add the f31 key? Should be able to update with: koji edit-tag coreos-pool -x tag2distrepo.keys="429476b4 cfc659b9 3c3359c4"
$ koji taginfo coreos-pool
Tag: coreos-pool [8632]
Arches: aarch64 ppc64le s390x x86_64
Groups: 
Tag options:
  tag2distrepo.enabled : 'true'
  tag2distrepo.keys : '429476b4 cfc659b9'
Inheritance:

Done. This needs an update to robosig config

$ koji add-tag f29-coreos-signing-pending --parent=coreos-pool
$ koji add-tag f31-coreos-signing-pending --parent=coreos-pool
$ koji edit-tag coreos-pool -x tag2distrepo.keys="429476b4 cfc659b9 3c3359c4"
$ koji taginfo coreos-pool
Tag: coreos-pool [8632]
Arches: aarch64 ppc64le s390x x86_64
Groups: 
Tag options:
  tag2distrepo.enabled : 'true'
  tag2distrepo.keys : '429476b4 cfc659b9 3c3359c4'
Inheritance:

now that there is a key for f32 can we get the coreos-pool tag updated to support the new key:

$ koji edit-tag coreos-pool -x tag2distrepo.keys="429476b4 cfc659b9 3c3359c4 12c944d0"
$ koji taginfo coreos-pool 
Tag: coreos-pool [8632]
Arches: aarch64 ppc64le s390x x86_64
Groups: 
Tag options:
  tag2distrepo.enabled : 'true'
  tag2distrepo.keys : '429476b4 cfc659b9 3c3359c4 12c944d0'
Inheritance:

Thanks @kevin:

Login to comment on this ticket.

Metadata