#8421 allow coreos team to regen coreos distrepos
Closed: Fixed 2 years ago by dustymabe. Opened 2 years ago by dustymabe.

  • Describe the issue

Today we had an issue where the distrepo yum repos for our coreos tags went away. @mohanboddu regenerated them for us. We're not sure why they went away (this should probably also be investigated), but we'd like the ability to regen them on our own in case this happens again in the future (i.e. we'd like to be able to unblock ourselves in case it happens in a scenario where we need a build and time is critical)

the commands were:

  • koji dist-repo f29-coreos-continuous 429476b4 --allow-missing-signatures
  • koji dist-repo f30-coreos-continuous cfc659b9--allow-missing-signatures
  • koji dist-repo coreos-pool 429476b4 cfc659b9

  • When do you need this? (YYYY/MM/DD)

would be nice to have it in the nxt week

  • When is this no longer needed or useful? (YYYY/MM/DD)

never

  • If we cannot complete your request, what is the impact?

we wait until releng can complete our request


Today we had an issue where the distrepo yum repos for our coreos tags went away

One thing that i just remembered is that tag2distrepo was just converted to a koji plugin by @puiterwijk . Maybe a bug there?

From RelEng meeting on Jun 19th 2019

[12:18:32] <mboddu> #info mizdebsk is working on fixing the dist repo generation (probably adding it to kojira)
[12:19:15] <mboddu> #info For the mean time, we will renable cron job that generates the dist repos

Ignore my previous comment

To answer the original issue, the repos disappeared. Its because the timeout to cleanup the dist-repos is set for 7 days.

Now, we increased the timeout to 6 months for now, which will be set to indefinite for latest rpm generated dist-repos and for non-latest rpm generated dist-repos we will set a lower retention time. This need a koji fix.

But, you dont need the permissions to generate the repo as its event based and once a build is tagged, it will generate the repo for you.

Metadata Update from @mohanboddu:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Ignore my previous comment

done :)

To answer the original issue, the repos disappeared. Its because the timeout to cleanup the dist-repos is set for 7 days.
Now, we increased the timeout to 6 months for now, which will be set to indefinite for latest rpm generated dist-repos and for non-latest rpm generated dist-repos we will set a lower retention time. This need a koji fix.

How do we see what it is currently set to for each type of repo?

Metadata Update from @dustymabe:
- Issue status updated to: Open (was: Closed)

2 years ago

i'm going to re-open this issue and re-request permissions for this to be given to users who have coreos-continuous permissions.

The main reason for the need for this is that there is a race condition we sometimes need to resolve.

There is a race condition in the tag2distrepo functionality where if we tag many packages at once (let's say 10) then many distrepo tasks (10) get kicked off at the same time. If the 2nd distrepo task that got created is the last one to finish then the resulting distrepo in

in the releng meeting today they decided to grant access for members of coreos-continuous to do a dist-repo. In a hack session right after the meeting we couldn't figure out how to get the policy set up exactly the way we want it so we settled on a two pronged solutions:

  • give dist-repo access to @dustymabe and @jlebon for now
  • work with koji team to set up dist-repo capability for anyone with coreos-continuous permissions on the coreos-pool and fxx-coreos-continuous tags in the future

Metadata Update from @dustymabe:
- Issue assigned to kevin

2 years ago

Additionally, we found that there's no 'dist-repo' permission in production right now. I wonder if this could be a missed item in upgrade schemas... ?
will investigate.

@kevin can we work with the koji team this week to figure this one out?

Can someone run this for me today since I still can't?

koji dist-repo --non-latest coreos-pool 429476b4 cfc659b9 3c3359c4 12c944d0

Can someone run this for me today since I still can't?

@smooge ran this for me (thanks @smooge). Maybe we can get the rest of this ticket sorted out before it happens the next time.

I did file a koji ticket on the missing perm:

https://pagure.io/koji/issue/1637

From RelEng meeting:

[13:05:09] <mboddu> #info grant dustymabe the permission for now, file another koji upstream ticket to add policy around dist-repos, close ticket and wait for koji upstream to implement, then fix our policy
[13:06:05] <mboddu> #info mohanboddu can file the ticket in upstream koji for the dist-repo policy

I have granted the perms.

@mohanboddu can you close this one once you file the koji upstream one?

seems to work!

$ koji dist-repo --non-latest coreos-pool 429476b4 cfc659b9 3c3359c4 12c944d0
Creating dist repo for tag coreos-pool
Watching tasks (this may be safely interrupted)...
37861786 distRepo (coreos-pool): free

I did file a koji ticket on the missing perm:
https://pagure.io/koji/issue/1637

Looks like this issue is fixed now?

From earlier:

  • work with koji team to set up dist-repo capability for anyone with coreos-continuous permissions on the coreos-pool and fxx-coreos-continuous tags in the future

We do still need this as it's blocking us working around the tag2distrepo race condition issue.

well, we can now grant people that permission just fine... but as far as I know, we can't add policy to grant it to anyone with another permission.

Are there more folks we should add to the permission now?

To further clairfy, the ticket asking for the policy for this @mohanboddu filed is https://pagure.io/koji/issue/1660 (and not fixed yet).

So, IMHO, we should:

  1. See if there are others we should add dist-repo perms to for now to work around this.

  2. close this and wait for a fix in upstream issue, once it exists, add policy.

What we are blocked on is the ability for our coreos-koji-tagger to request a distrepo to happen.

The definition for that application is here: https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/playbooks/openshift-apps/coreos-koji-tagger.yml

If we can assign a permission to the keytab that is used for that application then we'll be unblocked.

@kevin did the work here:

$ koji grant-permission dist-repo coreos-koji-tagger/coreos-koji-tagger.fedoraproject.org@FEDORAPROJECT.ORG
$ koji list-permissions --user coreos-koji-tagger/coreos-koji-tagger.fedoraproject.org@FEDORAPROJECT.ORG
coreos-continuous
dist-repo

@kevin and @mohanboddu helped set this up today:

For the prod keytab:

koji grant-permission dist-repo coreos-koji-tagger/coreos-koji-tagger.fedoraproject.org@FEDORAPROJECT.ORG

We didn't apply this to staging right now because the staging keytab isn't in use (I don't have the app running there right now) but here is how we would apply the permission there:

stg-koji grant-permission dist-repo --user=coreos-koji-tagger/coreos-koji-tagger.phx2.fedoraproject.org@STG.FEDORAPROJECT.ORG

ok I was able to run the distrepo command using the keytab for koji tagger in prod. I'll close this for now.

Metadata Update from @dustymabe:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

We didn't apply this to staging right now because the staging keytab isn't in use (I don't have the app running there right now) but here is how we would apply the permission there:
stg-koji grant-permission dist-repo --user=coreos-koji-tagger/coreos-koji-tagger.phx2.fedoraproject.org@STG.FEDORAPROJECT.ORG

Ok, now I need this for staging since I'm executing tests there. I have the app set up there now. Can we get that command run?

Also, any new ideas on how we can codify this policy in https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/koji_hub/templates/hub.conf.j2 ?

Metadata Update from @dustymabe:
- Issue status updated to: Open (was: Closed)

2 years ago

I've granted dist-repo permission to coreos-koji-tagger/coreos-koji-tagger.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG user in staging Koji.

Thanks @mizdebsk - I'm still getting a permission denied:

sh-5.0$ stg-koji hello
g'day, coreos-koji-tagger/coreos-koji-tagger.stg.fedoraproject.org!

You are using the hub at https://koji.stg.fedoraproject.org/kojihub
Authenticated via GSSAPI
sh-5.0$ stg-koji dist-repo --non-latest coreos-pool d300e724
2019-12-06 13:14:03,660 [ERROR] koji: ActionNotAllowed: dist-repo permission required

Any ideas?

it's odd because querying the user's permissions seems to be right:

$ stg-koji list-permissions --user=coreos-koji-tagger/coreos-koji-tagger.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG
coreos-continuous
dist-repo

ok @mizdebsk fixed the problem for me. Apparently koji changed how it maps users and he needed to twiddle the database:

[root@db-koji01 ~][STG]# sudo -u postgres psql koji
could not change directory to "/root": Permission denied
psql (10.10)
Type "help" for help.

koji=# select * from users where name like '%coreos%';
  id  |                                       name                                        | password | status | usertype
------+-----------------------------------------------------------------------------------+----------+--------+----------
 4350 | coreos-koji-tagger/coreos-koji-tagger.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG |          |      0 |        0
 4351 | coreosbot                                                                         |          |      0 |        0
 4357 | coreos-koji-tagger/coreos-koji-tagger.stg.fedoraproject.org                       |          |      0 |        0
(3 rows)

koji=# select * from user_krb_principals where user_id in (4350, 4357);
 user_id |                                   krb_principal                                  
---------+-----------------------------------------------------------------------------------
    4357 | coreos-koji-tagger/coreos-koji-tagger.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG
(1 row)

koji=# update user_krb_principals set user_id=4350 where user_id=4357;
UPDATE 1
koji=# select * from user_krb_principals where user_id in (4350, 4357);
 user_id |                                   krb_principal                                  
---------+-----------------------------------------------------------------------------------
    4350 | coreos-koji-tagger/coreos-koji-tagger.stg.fedoraproject.org@STG.FEDORAPROJECT.ORG
(1 row)

Metadata Update from @dustymabe:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

2 years ago

Login to comment on this ticket.

Metadata