Learn more about these different git repos.
Other Git URLs
Describe the issue
Fedora CoreOS will have various streams. All streams will use packages which are built in Fedora koji. Mechanical streams will use packages available in Fedora repos from tags f$(release), f$(release)-updates, f$(release)-updates-testing. Development and production streams will get built from packages available under koji tag coreos-pool. Packages which make it to production will also get tagged to coreos-release tag.
coreos-pool
coreos-release
These tags will ensure that (1) packages are not automatically garbage collected (2) stream builds are reproducible (up to the GC retention policy we agree upon), and (3) packages are added to the pool (and thus into the production streams) in a controlled manner.
To build FCOS for different streams we need: - coreos-pool and coreos-release koji tags created (include arches aarch64, ppc64le, x86_64) - Permission to add packages to requested tags (ACLs for bgilbert, dustymabe, jlebon and sinnykumari, until we have bot setup done) - Tag builds needs to be signed - Generate dist repo for requested tags with option --non-latest to koji dist-repo (to perform build with desired NVRA)
--non-latest
Question
As we have streams like next-devel and testing-devel, packages will be from multiple Fedora releases in coreos-pool and coreos-release tags. Since we will be using signed packages, does koji dist-repo allows to specify multiple keys for generating dist repos? This question is because, in infra ansible tag2distrepo, so far single key has been used.
When do you need this? (YYYY/MM/DD)
ASAP
When is this no longer needed or useful? (YYYY/MM/DD)
Always needed
If we cannot complete your request, what is the impact?
we won't be able to build FCOS for different streams
@ausil @rdossant Do we need to include s390x here?
I'd prefer to start with the three architectures we're already targeting and succeed there before we add s390 personally. It will be hard enough just to pull it together on those three. Once we work out the kinks then it shouldn't be a big deal adding a 4th arch.
@sinnykumari yes we need to, we are starting tow ork on s390x now in addition to aarch64 and ppc64le
@mohanboddu Any additional info needed or we are good to proceed with this request?
This will be discussed in Fedora RelEng meeting tomorrow Apr 24 at 16:00 UTC
Metadata Update from @mohanboddu: - Issue tagged with: meeting
Some questions:
You would tag all the packages for a compose into coreos-pool and all the ones in a released compose into coreos-release. Could you perhaps just use fedora GA/fedora-updates tags and only tag in the builds that are not in those and use those 2 repos on top of coreos-pool/coreos-release? That would make them much smaller and more manageable. Of course the fedora ones would be signed with fedora keys.
What keys should these tags builds be signed with? If with the fedora ones, since they aren't versioned how can we tell what key to use?
If we have to sign a large pile of packages with a new key often I am a bit worried at our signing bandwith.
Can we use fxx-coreos-release and fxx-coreos-pool tags which follow a Fedora release and can make use of same fedora release keys?
Another thought we had was to version the tags? ie, coreos-pool-30, coreos-pool-31, etc... and we could then just sign them with the fedora keys and just always make them when we branch.
But is everything being signed by a different key a requrement?
Can you re-use the existing fedora GA/updates repos signed with the fedora key in addition to these? or does it need to be one dist-repo?
We just discussed this ticket in the releng meeting today. Here is what I think we decided on:
fxx-coreos-signing-pending
Thanks all for the valuable input during releng meeting ! Unless we have any unanswered questions, can we get tags finalized in https://pagure.io/releng/issue/8294#comment-567284 and dist repo for coreos-pool tags with option --non-latest created?
@sinnykumari Can it wait until Wed?, since this requires some koji hub policy changes and robosig changes, since we are in infra freeze we can make the changes once the freeze lifts off.
Sure, thanks @mohanboddu
Made the necessary changes to koji hub policy and robosignatory config and added the koji tags needed.
$ koji add-tag coreos-pool $ koji add-tag coreos-pool --parent=coreos-release $ koji add-tag f30-coreos-signing-pending --parent=coreos-pool $ koji list-tag-inheritance coreos-release --reverse coreos-release (8633) └─coreos-pool (8632) └─f30-coreos-signing-pending (8634)
hey @mohanboddu - thanks for working on this. Quick question related to the implementation:
From the releng meeting (last week, I think) we discussed this and I think we had ended up deciding to not use koji inheritance for any of these tags. My understanding is that we should have coreos-release, coreos-pool, and f30-coreos-signing-pending all be separate with no inheritance.
f30-coreos-signing-pending
hey @mohanboddu - thanks for working on this. Quick question related to the implementation: From the releng meeting (last week, I think) we discussed this and I think we had ended up deciding to not use koji inheritance for any of these tags. My understanding is that we should have coreos-release, coreos-pool, and f30-coreos-signing-pending all be separate with no inheritance.
@dustymabe The reason why I thought inheritance is, you dont need to add packages multiple times. Once you add the package to coreos-release they will be available for all other tags. And you can compose either from coreos-release or coreos-pool.
Now that I am thinking out loud, coreos-pool will pick all the builds in coreos-release, which I think might not be desirable. Can you confirm this and I can remove the inheritance between coreos-pool and coreos-release.
But I think we should leave the coreos-pool and fxx-coreos-signing-pending inheritance as it is, since you can add a package to coreos-pool and it will be available in fxx-core-signing-pending and if they are separate you need to add the package to both tags and also we need to change koji hub permissions as well. Since fxx-coreos-signing-pending is just a holding tag, it wont be of a problem.
@dustymabe The reason why I thought inheritance is, you dont need to add packages multiple times. Once you add the package to coreos-release they will be available for all other tags. And you can compose either from coreos-release or coreos-pool. Now that I am thinking out loud, coreos-pool will pick all the builds in coreos-release, which I think might not be desirable. Can you confirm this and I can remove the inheritance between coreos-pool and coreos-release.
We aren't planning on composing from coreos-release. Just from coreos-pool. The rationale for the need for coreos-release is in my earlier comment. Yes, please remove the inheritance. Also note that we only need one distrepo set up and that is against coreos-pool.
My understanding is that the package goes to fxx-coreos-signing-pending when it is built and then robosig moves it to coreos-pool when the package has been signed. I don't think we need inheritance there? What would be the benefit?
I will remove the inheritance here.
Without the inheritance, you need to add the package to both the coreos-pool and fxx-coreos-signing-pending tags, just like you did here since both the tags have no idea about the packages. Using inheritance, thats not needed and once the build gets signed it will moved to coreos-pool tag from fxx-coreos-signing-pending tag.
Removed the tag inheritance between coreos-release and coreos-pool
$ koji remove-tag-inheritance coreos-pool coreos-release
OK. One or two questions here:
f31-coreos-signing-pending
OK. One or two questions here: is there any chance for an unsigned build to get into the resulting dist-repo?
is there any chance for an unsigned build to get into the resulting dist-repo?
Nope
once we have a f31-coreos-signing-pending will this break? We need this setup to work equally for f30-coreos-signing-pending and f31-coreos-signing-pending at the same time.
I think this will make a perfect choice then, since we can add f31-coreos-signing-pending tag to the inheritance and both f30-c-s-p and f31-c-s-p will work.
ok Thanks @mohanboddu !
@mohanboddu - can you link me to the distrepo for coreos-pool once you've set it up?
@dustymabe What do you mean? You can tag a build to f30-coreos-signing-pending, and once signed, it will be available in coreos-pool tag and I think currently tag2distrepo is still not working, so after tagging a build please let me know and I can run the koji dist-repo for you.
@mohanboddu - can you link me to the distrepo for coreos-pool once you've set it up? @dustymabe What do you mean? You can tag a build to f30-coreos-signing-pending, and once signed, it will be available in coreos-pool tag and I think currently tag2distrepo is still not working, so after tagging a build please let me know and I can run the koji dist-repo for you.
As per @mizdebsk FBR, automatic dist-repo generation with tag2distrepo doesn't work only in case of unsigned packages get tagged. Since, coreos-pool uses signed packages, it should be fine.
Ah nvm then, this patch should fix this and I completed running the playbook.
Just to clear up re. that patch,
+ 'coreos-pool': [cfc659b9],
When we're going to be sourcing from both f30 and f31 versions of the signing pending tags, we'd just have to list both keys here, right?
(Also shouldn't that keyid be in quotes?)
Just to clear up re. that patch, + 'coreos-pool': [cfc659b9], When we're going to be sourcing from both f30 and f31 versions of the signing pending tags, we'd just have to list both keys here, right?
Yes
Thanks, I will rerun the playbook fixing it :disappointed:
To check necessary permissions, I did:
$ koji add-pkg coreos-pool rpm-ostree --owner sinnykumari $ koji tag-build coreos-pool rpm-ostree-2019.3-1.fc30
It went fine, but I don't see a coreos-pool dist-repo being created at https://kojipkgs.fedoraproject.org/repos-dist/ . What are we missing here?
I restarted fedmsg-hub-3 on bodhi-backend02, it looks like it might not have picked up the new config yet.
Added the arches to coreos-pool tag
$ koji edit-tag coreos-pool --arches='aarch64 ppc64le s390x x86_64'
I am guessing its not required for coreos-release since it will just hold the builds from the released FCOS. Please let me know if I am wrong.
https://kojipkgs.fedoraproject.org/repos-dist/coreos-pool/latest/
thanks all who helped!
Awesome, thanks all!
Did few more checks to make sure we have everything inplace:
I think we have all requested items in this ticket done. Did we miss anything before we close this ticket?
Added the arches to coreos-pool tag $ koji edit-tag coreos-pool --arches='aarch64 ppc64le s390x x86_64' I am guessing its not required for coreos-release since it will just hold the builds from the released FCOS. Please let me know if I am wrong.
Added the arches to coreos-pool tag $ koji edit-tag coreos-pool --arches='aarch64 ppc64le s390x x86_64'
@sinnykumari Do you know answer for this ^?
Did we miss anything before we close this ticket?
One thing that is missing is special permissions surrounding tagging for the kernel. Only people who are allowed to build the kernel can tag it apparently. We need to be able to tag the kernel into our various tags using a bot. How can we achieve this goal?
That's correct, we don't need this since coreos-release tag is only to tag specific nvr builds which will make it to the FCOS release. we won't creating a dist-repo out of it.
Did we miss anything before we close this ticket? One thing that is missing is special permissions surrounding tagging for the kernel. Only people who are allowed to build the kernel can tag it apparently. We need to be able to tag the kernel into our various tags using a bot. How can we achieve this goal?
@dustymabe This commit should fix that issue.
It worked!
[dustymabe@hattop ~]$ koji tag-build coreos-pool kernel-5.0.11-300.fc30 Created task 34767868 Watching tasks (this may be safely interrupted)... 34767868 tagBuild (noarch): free 34767868 tagBuild (noarch): free -> closed 0 free 0 open 1 done 0 failed 34767868 tagBuild (noarch) completed successfully
We think we got everything we need here.
Closing the ticket.
Metadata Update from @mohanboddu: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
A couple additional requests:
f29-coreos-signing-pending
koji edit-tag coreos-pool -x tag2distrepo.keys="429476b4 cfc659b9 3c3359c4"
$ koji taginfo coreos-pool Tag: coreos-pool [8632] Arches: aarch64 ppc64le s390x x86_64 Groups: Tag options: tag2distrepo.enabled : 'true' tag2distrepo.keys : '429476b4 cfc659b9' Inheritance:
Done. This needs an update to robosig config
$ koji add-tag f29-coreos-signing-pending --parent=coreos-pool $ koji add-tag f31-coreos-signing-pending --parent=coreos-pool $ koji edit-tag coreos-pool -x tag2distrepo.keys="429476b4 cfc659b9 3c3359c4" $ koji taginfo coreos-pool Tag: coreos-pool [8632] Arches: aarch64 ppc64le s390x x86_64 Groups: Tag options: tag2distrepo.enabled : 'true' tag2distrepo.keys : '429476b4 cfc659b9 3c3359c4' Inheritance:
robosig change commit - https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?h=master&id=0b50324342d7eb933993db79d6563311c1c176f1
Thanks @mohanboddu. @kevin - can I get you to run the playbook that would get that change to apply?
Done.
❤️
now that there is a key for f32 can we get the coreos-pool tag updated to support the new key:
$ koji edit-tag coreos-pool -x tag2distrepo.keys="429476b4 cfc659b9 3c3359c4 12c944d0"
$ koji taginfo coreos-pool Tag: coreos-pool [8632] Arches: aarch64 ppc64le s390x x86_64 Groups: Tag options: tag2distrepo.enabled : 'true' tag2distrepo.keys : '429476b4 cfc659b9 3c3359c4 12c944d0' Inheritance:
Thanks @kevin:
Login to comment on this ticket.