#6422 request to access waiverdb and greenwave in fedora openshift
Closed: Fixed 6 years ago Opened 6 years ago by mjia.

We've deployed greenwave to stage, but it did not work. To debug the deplyoment problems, we need to have the access to the console and the rsh access to the containers. So could you please give the access to @dcallagh and I so that we can access to waiverdb and greenwave projects in fedora openshift?


Maybe we should extend the openshift/project role to take multiple owners, something like the following... But first we would have to solve #6404 which is making that role unusable.

diff --git a/playbooks/openshift-apps/greenwave.yml b/playbooks/openshift-apps/greenwave.yml
index 6f3ed7e..5ddb9b7 100644
--- a/playbooks/openshift-apps/greenwave.yml
+++ b/playbooks/openshift-apps/greenwave.yml
@@ -10,5 +10,5 @@

   roles:
-  - { role: openshift/project, app: greenwave, description: greenwave, fas_owner: codeblock }
+  - { role: openshift/project, app: greenwave, description: greenwave, fas_owner: [codeblock, mjia, dcallagh] }
   - { role: openshift/object, app: greenwave, file: imagestream.yml }
   - { role: openshift/object, app: greenwave, template: buildconfig.yml }
diff --git a/playbooks/openshift-apps/waiverdb.yml b/playbooks/openshift-apps/waiverdb.yml
index 11a9587..73235e1 100644
--- a/playbooks/openshift-apps/waiverdb.yml
+++ b/playbooks/openshift-apps/waiverdb.yml
@@ -12,5 +12,5 @@
   # The openshift/project role breaks if the project already exists:
   # https://pagure.io/fedora-infrastructure/issue/6404
-  #- { role: openshift/project, app: waiverdb, description: waiverdb, fas_owner: codeblock }
+  #- { role: openshift/project, app: waiverdb, description: waiverdb, fas_owners: [codeblock, mjia, dcallagh] }
   - { role: openshift/object, app: waiverdb, template: secret.yml }
   #- { role: openshift/secret-file
diff --git a/roles/openshift/project/templates/admin.yml b/roles/openshift/project/templates/admin.yml
index 1602468..5e3055b 100644
--- a/roles/openshift/project/templates/admin.yml
+++ b/roles/openshift/project/templates/admin.yml
@@ -9,6 +9,10 @@ roleRef:
   name: admin
 subjects:
+{% for fas_owner in fas_owners %}
 - kind: User
   name: "{{fas_owner}}"
+{% endfor %}
 userNames:
+{% for fas_owner in fas_owners %}
 - "{{fas_owner}}"
+{% endfor %}

ping, @puiterwijk, this is related to the ACLs we were talking about this evening.

This should now be resolved.
openshift/project now has a variable "appowners", which is a list of users that get basically the "view" role minus the listing of ConfigMaps (which likely contain secrets).
I have added @mjia and @dcallagh as "appowner" to greenwave.

Metadata Update from @puiterwijk:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

6 years ago

Now I can see the greenwave project in https://os.stg.fedoraproject.org, but I can't run the playbook.
Here is the log:

https://paste.fedoraproject.org/paste/lrDrCYbpCLmKi-AvNIbzfQ

Metadata Update from @mjia:
- Issue status updated to: Open (was: Closed)

6 years ago

That was caused by some old remnants of previous attempts, and is now fixed.

Metadata Update from @puiterwijk:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

6 years ago

Okay, I think I know why. I am not a member of the greewave project in the stage server. @puiterwijk, could you please add @dcallagh and me as the owner of greenwave on the stage server as well?

Metadata Update from @mjia:
- Issue status updated to: Open (was: Closed)

6 years ago

@mjia no, the problem was that I had to move to using ClusterRole, but on the staging server there was still a Role object left from that attempt. (which worked fine in staging, but not in prod. Project-local Roles don't work until Openshift 3.6.).

Metadata Update from @puiterwijk:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

6 years ago

So I think as the appowner, I should have the rsh access to the containers, but it seems not.

oc rsh greenwave-web-24-zk5qc
Error from server (Forbidden): User "mjia" cannot create pods/exec in project "greenwave"

May I please ask for this access which is quite handy for us to debug the problems?

Metadata Update from @mjia:
- Issue status updated to: Open (was: Closed)

6 years ago

No, you were not supposed to have that access.
@ralph agreed with me that the access that "appowners" should have in production is limited to view all info, including logs, except for configmaps.
I would be okay with granting "oc debug" access, since the pods started that way do not get traffic routed to them, but still get a pretty much identical pod shell.
Could you try running "oc debug" and report what access that needs?

oc debug greenwave-web-24-zk5qc
Debugging with pod/greenwave-web-24-zk5qc-debug, original command: /bin/sh -c gunicorn --bind 0.0.0.0:8080 --access-logfile=- greenwave.wsgi:app
Error from server (Forbidden): User "mjia" cannot create pods in project "greenwave"

After checking this, this is not going to work anyway in production right now due to http2 not being available.
Would staging be enough? Since I'd be fine with granting appowners more permissions in staging.

Yes please @puiterwijk , access in staging will at least let us get all the deploymentconfig stuff down pat. Then there should be no issues when we roll it into production.

@puiterwijk , could you please take a look at waiverdb? It seems both stg and prod are down.

https://waiverdb-web-waiverdb.app.os.stg.fedoraproject.org/api/v1.0
https://waiverdb-web-waiverdb.app.os.fedoraproject.org/api/v1.0

And I can only see Greenwave in stg web console, so could you please also add @dcallagh and me as "appowner" to waiverdb?

@mjia Please note that waiverdb does not have any endpoints registered to /api/v1.0.
This does work: https://waiverdb-web-waiverdb.app.os.stg.fedoraproject.org/api/v1.0/waivers/ and https://waiverdb-web-waiverdb.app.os.fedoraproject.org/api/v1.0/waivers/ :-).

Nonetheless, you're absolutely welcome as appowners to waiverdb, and I've now added you.
Just rerun the playbook and it should set everything up.

Oh, yeah, you are absolutely right, thanks.

@mjia also, note that I just added create, delete and update permissions for the "appowner" role template in openshift staging.
So after you next rerun the playbook, you should have those permissions.

Do note that unfortunately the shell does not yet work over the internet, I am still working on fixing that.

Do note that unfortunately the shell does not yet work over the internet, I am still working on fixing that.

FWIW, @mjia and @dcallagh should be able to get shell with oc commands on batcave01, no? I guess we would need to install the client commands there first. The limitation as I understand it has been http/2 support in the proxy layer. Accessing the cluster from inside the prod environment should work.

Added clients here: https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=dc585a70269c64c471f89f01c2d71d99a63c9dd5

It seems to work. Try:

oc login https://os.fedoraproject.org

on batcave01.

I lied.

You need to oc login https://os-master01.phx2.fedoraproject.org or oc login https://os-master01.stg.phx2.fedoraproject.org.

Is there anything more here for now? Or should we close this out and open new issues as we hit them?

The only thing left here is to be able to oc rsh <pod>, but it seems this is still not working.

[mjia@batcave01 ~][PROD]$ oc rsh greenwave-web-69-57p69
Error from server (Forbidden): User "mjia" cannot create pods/exec in project "greenwave"

The only thing left here is to be able to...

Yup. @puiterwijk and I talked about it briefly yesterday. I think he had a fix, but maybe it didn't take. @mjia - oc rsh <pod> still doesn't work for you, correct?

@mjia: I think you might have been using the production environment.
In there, you indeed to not have access to change things.

In staging, you do:

[puiterwijk@batcave01 ~][PROD]$ oc login --token=........ --server=https://os-master01.stg.phx2.fedoraproject.org

Logged into "https://os-master01.stg.phx2.fedoraproject.org:443" as "mjia" using the token provided.

You have one project on this server: "greenwave"

Using project "greenwave".
[puiterwijk@batcave01 ~][PROD]$ oc rsh greenwave-web-81-ftt6c
sh-4.4$ 

[puiterwijk@batcave01 ~][PROD]$ oc logout
Logged "mjia" out on "https://os-master01.stg.phx2.fedoraproject.org:443"

For those who find this later and want to debug permissions, this is the neater way to test things:

[puiterwijk@batcave01 ~][PROD]$ oc --as=mjia rsh greenwave-web-81-ftt6c
sh-4.4$ exit
[puiterwijk@batcave01 ~][PROD]$ 

This is how I did and the error I received.

[mjia@batcave01 ~][PROD]$ oc login --token=xxxxx --server=https://os.stg.fedoraproject.org
Logged into "https://os.stg.fedoraproject.org:443" as "mjia" using the token provided.

You have one project on this server: "greenwave"

Using project "greenwave".
[mjia@batcave01 ~][PROD]$ oc --as=mjia rsh greenwave-web-97-69c7t
Error from server (Forbidden): User "mjia" cannot "impersonate" "users.user.openshift.io" with name "mjia" in project ""

@mjia: 1. note that you need the os-master01.stg.phx2.fedoraproject.org server, and 2. you are unable to use --as. That was just for if another admin wanted to debug.

@puiterwijk , cool, it works. It would be nice if I can use the os.stg.fedoraproject.org server to login so that I do not need to manually replace the server when getting an API token.

@mjia, unfortunately, that requires significant architecture changes.
I am working on those changes, but they will take a while.

Metadata Update from @ralph:
- Issue status updated to: Closed (was: Open)

6 years ago

@puiterwijk , that's okay, thank you for your help. I think we can close this issue.

Metadata Update from @mjia:
- Issue status updated to: Open (was: Closed)

6 years ago

Metadata Update from @mjia:
- Issue status updated to: Closed (was: Open)

6 years ago

Nonetheless, you're absolutely welcome as appowners to waiverdb, and I've now added you.
Just rerun the playbook and it should set everything up.

@puiterwijk I've rerun the playbook but I still can't see waiverdb in the web console. Could you please have a look at it? Thanks.

Metadata Update from @mjia:
- Issue status updated to: Open (was: Closed)

6 years ago

This is fixed now, also for future projects.

Metadata Update from @puiterwijk:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

6 years ago

Thank you very much!

Login to comment on this ticket.

Metadata