#8165 new koji tag for Fedora CoreOS continuous builds
Closed: Fixed 4 years ago by mohanboddu. Opened 5 years ago by dustymabe.

  • Describe the issue

We are exploring the idea of building and tagging some rpms that we care about more continously. After discussing with releng we can start by having a f29-coreos-continuous tag and a distrepo set up off of that. We'll use f29 for now to experiment with and then apply it to f30 once it's working.

  • When do you need this? (YYYY/MM/DD)

This week would be nice.

  • When is this no longer needed or useful? (YYYY/MM/DD)

N/A

  • If we cannot complete your request, what is the impact?

We can't explore this option for building more continuously using koji and need to implement it in a separate system that is less like production.


ACLs for dustymabe and bgilbert for now until we automate via a bot.

Metadata Update from @mohanboddu:
- Issue tagged with: meeting

5 years ago

So, the plan here is add a tag f29-coreos-continuous without a parent

$ koji add-tag f29-coreos-continuous --arches='armv7hl i686 x86_64 aarch64 ppc64le s390x'

Then add the packages that they want, for ex kernel

$ koji add-pkg f29-coreos-continuous kernel --owner=releng

Then generate the dist-repo for this tag

koji dist-repo f29-coreos-continuous fedora-29

Am I right and is this all that needs to be done?

Regarding acls, we might have to look at koji hub policy.

Am I right and is this all that needs to be done?

that looks right i think. after koji dist-repo is run it will automatically recreate a repo after a new rpm is tagged in?

Regarding acls, we might have to look at koji hub policy.

ok - do you know how to do that?

Am I right and is this all that needs to be done?

that looks right i think. after koji dist-repo is run it will automatically recreate a repo after a new rpm is tagged in?

Its not automated, so either you have to manually regen the repo or use cron/fedmsg listener, it depends on how often you want to regen the repo, if its not many times, you can generate it when needed.

Regarding acls, we might have to look at koji hub policy.

ok - do you know how to do that?
@mizdebsk is testing this sorta thing in stg koji and its a koji hub policy without any changes or upgrades to koji. We can fix it with his help.

And, one more question, how are you going to handle signing? On the same note, how are you going to do a build and tag the rpms into this tag? if you can build using f29 build target, they will be in f29-updates-candidate tag, but they wont be signed until you submit it as an update. Not sure, if building that way and submitting updates and then tagging the build to this new tag... is that all feasible or add this tag to autosign config, which needs infra help.

Am I right and is this all that needs to be done?
that looks right i think. after koji dist-repo is run it will automatically recreate a repo after a new rpm is tagged in?

Its not automated, so either you have to manually regen the repo or use cron/fedmsg listener, it depends on how often you want to regen the repo, if its not many times, you can generate it when needed.

ok. for some reason I thought it was triggered today automatically. I guess for things (like buildroot overrides) that use this functionality today are automated at a level higher than koji?

Regarding acls, we might have to look at koji hub policy.
ok - do you know how to do that?
@mizdebsk is testing this sorta thing in stg koji and its a koji hub policy without any changes or upgrades to koji. We can fix it with his help.

How is this much different than what we already have for atomic host.

And, one more question, how are you going to handle signing? On the same note, how are you going to do a build and tag the rpms into this tag? if you can build using f29 build target, they will be in f29-updates-candidate tag, but they wont be signed until you submit it as an update. Not sure, if building that way and submitting updates and then tagging the build to this new tag... is that all feasible or add this tag to autosign config, which needs infra help.

Since the purpose of this is for development signed builds isn't really an issue but I would like to explore a build target maybe as a follow up step. If we had our own build target I think something like this would work:

  • we create new dist-git branch in a repo f29-coreos-continuous.
  • We then commit to that branch and can do fedpkg-build and it gets tagged into that tag. Is that correct?

ok. for some reason I thought it was triggered today automatically. I guess for things (like buildroot overrides) that use this functionality today are automated at a level higher than koji?

Nope, repos generated from infra tags use fedmsg listener.

@mizdebsk is testing this sorta thing in stg koji and its a koji hub policy without any changes or upgrades to koji. We can fix it with his help.

How is this much different than what we already have for atomic host.

Over there, we added a new permission for koji and given that permission to the tag and the people so that they can do tagging and untagging operations, we can do the same here if needed. But with koji hub policy changes, we dont need to keep adding new permissions, we directly set the permissions at tag/user level.

Since the purpose of this is for development signed builds isn't really an issue but I would like to explore a build target maybe as a follow up step. If we had our own build target I think something like this would work:

we create new dist-git branch in a repo f29-coreos-continuous.

How many repos are we talking here and what are they?

We then commit to that branch and can do fedpkg-build and it gets tagged into that tag. Is that correct?

Yes, that is correct.

ok. for some reason I thought it was triggered today automatically. I guess for things (like buildroot overrides) that use this functionality today are automated at a level higher than koji?

Nope, repos generated from infra tags use fedmsg listener.

ok so we would use our own listener then.

@mizdebsk is testing this sorta thing in stg koji and its a koji hub policy without any changes or upgrades to koji. We can fix it with his help.
How is this much different than what we already have for atomic host.

Over there, we added a new permission for koji and given that permission to the tag and the people so that they can do tagging and untagging operations, we can do the same here if needed. But with koji hub policy changes, we dont need to keep adding new permissions, we directly set the permissions at tag/user level.

Let's go with whatever we can implement today. There will be more iterations of this in the future (i.e. f30-coreos-continuous) so we can improve as we go if there are better solutions in the future.

Since the purpose of this is for development signed builds isn't really an issue but I would like to explore a build target maybe as a follow up step. If we had our own build target I think something like this would work:
we create new dist-git branch in a repo f29-coreos-continuous.

How many repos are we talking here and what are they?

Right now: one build-target, one tag, one dist-repo (yum repo)

We then commit to that branch and can do fedpkg-build and it gets tagged into that tag. Is that correct?

Yes, that is correct.

perfect

Right now: one build-target, one tag, one dist-repo (yum repo)

Again, signing, build-target will use f29-build tag for buildroot and the destination tag has to understand signing.

Again, signing, build-target will use f29-build tag for buildroot and the destination tag has to understand signing.

I thought I addressed this concern with:

@dustymabe
Since the purpose of this is for development signed builds isn't really an issue

Again, signing, build-target will use f29-build tag for buildroot and the destination tag has to understand signing.

I thought I addressed this concern with:

@dustymabe
Since the purpose of this is for development signed builds isn't really an issue

Well, koji dist-repo requires it. Sorry, I didn't answer that before.

$ koji dist-repo --help
Usage: koji dist-repo [options] tag keyID [keyID...]
(Specify the --help option for a list of other options)
...

From RelEng meeting on Mar 6th 2019:

RelEng will create the f29-coreos-continuous tag and add a build target which uses buildroot as f29-build and dest tag as f29-coreos-continuous tag. mboddu will work with mizdebsk and nirik to get the permissions setup in koji hub policy. Dist repos can be generated either manually, cron, fedmsg listener or by the bot/pipeline. Thats up to the coreos folks.

From irc chat:

[12:02:30] <ksinny> mboddu: Have a follow-up question from coreos continuous build discussion.
[12:03:05] <ksinny> Suppose we build a package p1 which gets tagged to f29-coreos-continuous, now package p2 depends upon p1 build available in tag f29-coreos-continuous . Will it be possible to build p2 with using using dependency p1 which we just built ? I bllieve it won't be avilable in f29-build buildroot, right?
[12:04:11] <+mboddu> ksinny: Yes, it wont be, but is it something you want?
[12:05:50] <+mboddu> ksinny: If you want it, we can create a side-tag

From RelEng meeting on Mar 6th 2019:

RelEng will create the f29-coreos-continuous tag and add a build target which uses buildroot as f29-build and dest tag as f29-coreos-continuous tag. mboddu will work with mizdebsk and nirik to get the permissions setup in koji hub policy. Dist repos can be generated either manually, cron, fedmsg listener or by the bot/pipeline. Thats up to the coreos folks.

Thanks - let me know when this is done and I'll test it out!

From irc chat:

[12:02:30] <ksinny> mboddu: Have a follow-up question from coreos continuous build discussion.
[12:03:05] <ksinny> Suppose we build a package p1 which gets tagged to f29-coreos-continuous, now package p2 depends upon p1 build available in tag f29-coreos-continuous . Will it be possible to build p2 with using using dependency p1 which we just built ? I bllieve it won't be avilable in f29-build buildroot, right?
[12:04:11] <+mboddu> ksinny: Yes, it wont be, but is it something you want?
[12:05:50] <+mboddu> ksinny: If you want it, we can create a side-tag

@kevin If we add side tag/buildroot, is it going to affect dist-repo by any chance, as the build repo gets generated once a build gets tagged into it?

You can make a dist-repo of any tag I think, it doesn't care which. side-tags don't update the buildroot tho.

Making these builds update the regular buildroot could be dangerous though. Is this really going to be needed? We could manually tag such builds into fNN-override, but then they apply to ALL fNN builds.

yeah maybe let's just get this going and we'll worry about some of the corner cases later when we hit them.

I've just read meeting logs that lead me here. I've set up a PoC of Koji hub policy that allows users with coreos-continuous permission to add/block/unblock packages in active f$N-coreos-continuous tags. I believe it meets requirements described in this ticket and discussed on the meeting.

I've set up a PoC of Koji hub policy that allows users with coreos-continuous permission to add/block/unblock packages in active f$N-coreos-continuous tags

Thanks @mizdebsk! That is one part of this ticket, correct? I think we still need to create the tags:

[dustymabe@hattop ~]$ koji list-tags  | grep coreos
[dustymabe@hattop ~]$

bump.. can we make this happen?

@dustymabe Finally we got the stg koji working again and here's the build target

$ stg-koji add-tag f29-coreos-continuous
$ stg-koji add-target f29-coreos-continuous f29-build f29-coreos-continuous

I am keeping the ticket open as I sense that there might be more things needed.

@mohanboddu - can you give me to commands to run to test the ACLs ?

I assume something like:

koji tag-build f29-coreos-continuous rpm-ostree-V-R-A
koji dist-repo f29-coreos-continuous 429476B4 --skip-missing-signatures

Added the necessary permissions

$ stg-koji grant-permission --new coreos-continuous mohanboddu
$ stg-koji grant-permission coreos-continuous dustymabe bgilbert

From irc:

<dustymabe> mboddu: actually - let's add ksinny to the list of people with acls

$ stg-koji grant-permission coreos-continuous sinnykumari

looks like I'm hitting some issue:

$ stg-koji tag-build f29-coreos-continuous ignition-0.31.0-2.gitf59a653.fc29
2019-03-13 13:11:01,782 [ERROR] koji: TagError: Package ignition not in list for f29-coreos-continuous

here is that build: https://koji.fedoraproject.org/koji/buildinfo?buildID=1221479

Added the x86_64 aarch64 ppc64le arches:

$ stg-koji edit-tag f29-coreos-continuous --arches='x86_64 aarch64 ppc64le'

Here are the steps I used to test this out:

stg-koji add-pkg f29-coreos-continuous ignition --owner dustymabe
stg-koji tag-build f29-coreos-continuous ignition-0.31.0-2.gitf59a653.fc29
stg-koji dist-repo f29-coreos-continuous 429476B4 --allow-missing-signatures

Then the output showed up under https://kojipkgs.stg.fedoraproject.org/repos-dist/f29-coreos-continuous/

So it looks like everything is working. Can we put this in place in real koji now?

@dustymabe Can it wait until freeze? The tag creation and other work should be fine, but for adding the permissions it needs an FBR. I guess I should rather ask @kevin and @mizdebsk if they are okay with the changes to koji hub policy change for adding permissions during this freeze.

@mizdebsk If you are making changes, then please take a look at https://pagure.io/releng/issue/8142 as well.

Thanks.

This is a big change in a critical system for freeze, that said if we are confident in stg testing we could submit a freeze break.
@dustymabe would you like us to do that? or can we wait?

@mohanboddu
@dustymabe Can it wait until freeze?

@kevin
@dustymabe would you like us to do that? or can we wait?

Considering the slip I'd really prefer not to wait another week since it's already some time since we originally made the request and we'd like to start using it. I'm sorry about making your life more difficult :(

@mohanboddu Infra freeze has been lifted today. Unless there is any other concern, can we get FBR patch applied ?

@sinnykumari I was planning to work on this one first thing in the morning and will work with @kevin to make the necessary changes.

@dustymabe @sinnykumari
Applied the patch and ran the following commands, so you should be set now.

$ koji add-tag f29-coreos-continuous
$ koji add-target f29-coreos-continuous f29-build f29-coreos-continuous
$ koji grant-permission --new coreos-continuous mohanboddu
$ koji grant-permission coreos-continuous dustymabe bgilbert sinnykumari
$ koji edit-tag f29-coreos-continuous --arches='x86_64 aarch64 ppc64le'

Thanks @mohanboddu - first roadblock:

[dustymabe@media ~]$ koji add-pkg f29-coreos-continuous ignition --owner dustymabe
Adding 1 packages to tag f29-coreos-continuous
2019-04-04 22:57:33,933 [ERROR] koji: ActionNotAllowed: policy violation (package_list)

cc @mizdebsk

This is expected - in production only Koji admins can add packages to tags. The policy allowing selected non-admin users to manipulate package lists is enabled in staging Koji only. To enable it in production someone needs to remove {% if env == 'staging' %} conditional surrounding package_list policy from roles/koji_hub/templates/hub.conf.j2 in ansible.git and run appropriate Koji hub playbook.

@dustymabe I think https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=913a8f8 might have fixed it. Could you please give it another try and let us know?

Thanks.

As I said in a comment above, there is a different for the policy violation error @dustymabe is getting. Custom policy is simply not deployed in production Koji.

Can we also have a corresponding f30 version of the tag and build target? We're in the process of rebasing FCOS to f30 and it's likely coreos-assembler will soon follow. Could you also add jlebon to the ACL list?

OK, so I just tried building rpm-ostree into the existing target, and got:

$ fedpkg build --target=f29-coreos-continuous
Building rpm-ostree-2019.3.5.g0da9f997-1.fc29 for f29-coreos-continuous
Created task: 34077477
Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=34077477
...
34077477 build (f29-coreos-continuous, /rpms/rpm-ostree.git:ea0b871bb667585af563284ef3f5329f3b9226c5): open (buildvm-armv7-05.arm.fedoraproject.org) -> FAILED: BuildError: package rpm-ostree not in list for tag f29-coreos-continuous

Do we need to provide upfront the packages we intend to build in the tag, or (preferably) can we extend the coreos-continuous permission for this?

@mohanboddu Please extend coreos-continuous permission to jlebon as well

@jlebon

Do we need to provide upfront the packages we intend to build in the tag, or (preferably) can we extend the coreos-continuous permission for this?

Yes, you need that and here are the steps to follow:
https://pagure.io/releng/issue/8165#comment-560145

Also, I have given you the coreos-continuous permission.

Gotcha, thanks! I think I had misunderstood how Koji works in that regard. So, I'm now hitting the same thing Dusty did in https://pagure.io/releng/issue/8165#comment-564534 :

$ koji add-pkg f29-coreos-continuous rpm-ostree --owner jlebon
Adding 1 packages to tag f29-coreos-continuous
2019-04-10 14:46:26,318 [ERROR] koji: ActionNotAllowed: policy violation (package_list)

As per @mizdebsk's comment, can we relax this restriction in prod specifically for coreos-continuous users targeting the f*-coreos-continuous tags?

Koji policy has been updated in production. Users with coreos-continuous should now be able to manipulate package lists in respective tags in production.

Thanks @mizdebsk for updating the koji policy in production!

I gave it a shot and I can successfully add package to f29-coreos-continuous and tag corresponding pacakge NVR to this tag.

$ koji add-pkg f29-coreos-continuous ignition --owner sinnykumari
Adding 1 packages to tag f29-coreos-continuous
🔹[skumari@toolbox ~]$ koji tag-build f29-coreos-continuous ignition-2.0.0-alpha.3.git906cf04.fc29
Created task 34107870

Looks like permission is needed to create dist-repo:

🔹[skumari@toolbox ~]$ koji dist-repo f29-coreos-continuous 429476B4 --allow-missing-signatures
2019-04-11 08:48:58,863 [ERROR] koji: ActionNotAllowed: dist-repo permission required

From irc:

[10:25:14] <ksinny> mizdebsk: Hi! Any idea what permissions are missing when I try to create koji dist-repo with packages in f29-coreos-continuous tag
[10:26:13] <mizdebsk> ksinny, hi, the error message you posted in the ticket says it
[10:26:34] <mizdebsk> ActionNotAllowed: dist-repo permission required
[10:26:45] <mizdebsk> that means you need permission called "dist-repo"
[10:28:07] <mizdebsk> koji admins can grant it to you (and others) with: koji grant-permission dist-repo sinnykumari
[10:32:26] <ksinny> mizdebsk: yeah, that's seem obvious by error message. I didn't see in the ticket comments that dist-repo permission was granted for stg-koji which made me unsure of what is missing
[10:33:15] <ksinny> mboddu: Are you the right person to grant dist-repo permission or need to ask someone else?
[10:35:17] <+mboddu> mizdebsk: Is there a permission called 'dist-repo'?
[10:35:31] »» +mboddu checking the ticket
[10:36:50] <+mboddu> There is no dist-repo permission
[10:37:54] <mizdebsk> mboddu, but you need to create one if you want to allow non-admin users to create dist repos
[10:38:06] <mizdebsk> (there is such permission in stg koji)
[10:40:06] <+mboddu> mizdebsk: Ah okay, but is there a way that we can limit the permission to certain users and to certain tags?
[10:40:19] <mizdebsk> no, not that i am aware of
[10:40:25] »» mizdebsk will check
[10:40:58] <+mboddu> mizdebsk: Thanks
[10:41:08] <mizdebsk> mboddu, do, there is no way to do that with current koji code
[10:41:11] <mizdebsk> no*
[10:41:31] <mizdebsk> by granting dist-repo permission you allow the user to create dist repos from any tags
[10:42:47] <+mboddu> mizdebsk: Okay, then, need to vote and give permission for now and create a ticket in koji to regulate the permission to user+tag level

So, I would like to call out for votes to give dist-repo permission to the users along with coreos-continuous permission.

So, I would like to call out for votes to give dist-repo permission to the users along with coreos-continuous permission.

I vote 👍 - but my vote doesn't count :)

Alternatively, is there a way for the repo to just be auto-regenerated whenever the package set changes? (Trigger based, not poll-based ideally). Then we wouldn't need to dist-repo at all :)

Yes, there is an existing mechanism to have dist-repos generated and updated as needed for selected tags.

Alternatively, is there a way for the repo to just be auto-regenerated whenever the package set changes? (Trigger based, not poll-based ideally). Then we wouldn't need to dist-repo at all :)

Which we discussed, but for some reasons we decided to go manual but I forgot why.

Alternatively, is there a way for the repo to just be auto-regenerated whenever the package set changes? (Trigger based, not poll-based ideally). Then we wouldn't need to dist-repo at all :)

Which we discussed, but for some reasons we decided to go manual but I forgot why.

https://pagure.io/releng/issue/8165#comment-557999 and some follow-up comments seems related.
From the discussion it appears koji dist-repo is not automatically generated. Existing dist-repos which we have in Fedora uses cron, femdsg listener etc to automate it.

Thanks Sinny, I had missed that earlier part of the convo. Yeah, a fedmsg listener sounds like what we want. How can we set that up?

@jlebon By extending list of tags in tag2distrepo config file in ansible.git

f30 coreos continuous setup is done

$ koji add-tag f30-coreos-continuous
$ koji add-target f30-coreos-continuous f30-build f30-coreos-continuous
$ koji edit-tag f30-coreos-continuous --arches='x86_64 aarch64 ppc64le'

Cron job to regenerate dist repos is deployed in production. You can find generated dist repos in kojipkgs, eg: latest f29-coreos-continuous dist repo.
fedmsg-based regeneration currently won't work due to unsigned packages, but today I'll try to make it allow missing signatures too.

Cron job to regenerate dist repos is deployed in production. You can find generated dist repos in kojipkgs, eg: latest f29-coreos-continuous dist repo.

yay! thanks @mizdebsk , How frquently this cron job runs?

fedmsg-based regeneration currently won't work due to unsigned packages, but today I'll try to make it allow missing signatures too.

+1

The purpose of the cron job is to generate repos that are missing or about to expire. Fedmsg-based regeneration is not enough by itself - without the cron job dist repos would expire after 7 days of inactivity (no builds tagged) and would be deleted.

Although the cron job runs once per hour, it won't generate repos unless they are older than 5 days. Until fedmsg-based regeneration is fixed you can ping me when you need the repo generated and I will trigger it manually.

Interesting, thanks for the explanation!
Can you please regenerate dist-repo for f30-coreos-continuous? I added ignition-2.0.0-alpha.3.git906cf04.fc30 package which will be useful when FCOS switches to F30.

f30 coreos continuous setup is done
$ koji add-tag f30-coreos-continuous
$ koji add-target f30-coreos-continuous f30-build f30-coreos-continuous
$ koji edit-tag f30-coreos-continuous --arches='x86_64 aarch64 ppc64le'

Thanks, can tag build packages to f30-coreos-continuous

Great, thanks @mizdebsk and @mohanboddu for all the help!

fedmsg-based regeneration currently won't work due to unsigned packages, but today I'll try to make it allow missing signatures too.

OK gotcha. Let's keep this ticket open to track this last remaining item then.

@mohanboddu Sorry for late request, can we add s390x arch to the f*-coreos-continuous ?

Although the cron job runs once per hour, it won't generate repos unless they are older than 5 days. Until fedmsg-based regeneration is fixed you can ping me when you need the repo generated and I will trigger it manually.

@mizdebsk Did fedmsg-based regeneration got fixed?

Not yet - I have a fix ready, but I can't deploy it due to Fedora infrastructure freeze - I submitted a freze break request and I'm waiting for ACKs that are required to deploy the fix.

@mohanboddu Sorry for late request, can we add s390x arch to the f*-coreos-continuous ?

Done.

$ koji edit-tag --arches='x86_64 aarch64 ppc64le s390x' f30-coreos-continuous
$ koji edit-tag --arches='x86_64 aarch64 ppc64le s390x' f29-coreos-continuous

Just tagged ignition-2.0.0-beta.1.git910e6c6.fc30 to f30-coreos-continuous tag and I see that dist-repo regeneration happened automatically. Also, s39ox packages are included in dist-repo as well https://kojipkgs.fedoraproject.org/repos-dist/f30-coreos-continuous/1154739/

I think all the items from this ticket is resolved now.

We think we resolved everything here.

Closing the ticket.

Metadata Update from @mohanboddu:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

4 years ago

Login to comment on this ticket.

Metadata