So, I made a user for fedora-ci-testing-farm and gave it to @mvadkert

I assume this is for a different need?

What is a descriptive name for it?

Can you email me a gpg key to use to encrypt the tokens back to you? (assuming you can't share the testing-farm ones).

@kevin hi!
Let's create a user with name: fedora-ci-run-tests
I do not have access to private testing-farm GPG key .

My public GPG key is:

http://keys.gnupg.net/pks/lookup?op=vindex&fingerprint=on&search=0xFFDFB4A69A82842F
https://pgp.mit.edu/pks/lookup?op=get&search=0xFFDFB4A69A82842F

If there any chance to get some info, I would be very grateful, I sent email : https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/4AVA4P2KK2IYNSBRKWH4CWJS4RUYS2JP/

It is a bit related to this ticket.

That seems a bit weird name. I would go with what we have, fedora-ci-TEAM-NAME, so that is fedora-ci-osci in this case :)

BTW if you want the same access as we have (to create EKS cluster), I believe it can be copied over from our account, you are right though that you will need 2 separate subnets and your own VPC.

fedora-ci-osci is perfect name.

I also replied to the conversation at: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/4AVA4P2KK2IYNSBRKWH4CWJS4RUYS2JP/

Friendly ping =)

Sorry for the delay. I will try and process this today for you.

ok, token sent via encrypted email. Can you confirm you got it and it works and then close this? Unless you need anything further.

@kevin thank you, I am checking, I need a few mins.

I can login with the secrets keys, but that is all.
I cannot use/create any resources.
I sent email with additional info.
Thank you for your help and patience :-D

Huh. I thought I gave you the same perms as fedora-ci-testing-farm, but I guess not.

Can you try again now?

The same: I cannot access the testing-farm cluster.
I cannot create new AWS resources.

https://aws.amazon.com/premiumsupport/knowledge-center/eks-api-server-unauthorized-error/

(python3ve) ➜  aws eks update-kubeconfig --name testing-farm --region=us-east-1
Updated context arn:aws:eks:us-east-1:xxx:cluster/testing-farm in /home/andrei/.kube/config
(python3ve) ➜  aws eks update-kubeconfig --name eks-cluster-name --region aws-region --role-arn arn:aws:iam::XXXXXXXXXXXX:role/testrole
(python3ve) ➜  kubectl config view --minify\
> 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://xxx.gr7.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:xxx:cluster/testing-farm
contexts:
- context:
    cluster: arn:aws:eks:us-east-1:xxx:cluster/testing-farm
    user: arn:aws:eks:us-east-1:xxx:cluster/testing-farm
  name: arn:aws:eks:us-east-1:xx:cluster/testing-farm
current-context: arn:aws:eks:us-east-1:xxx:cluster/testing-farm
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxx:cluster/testing-farm
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - testing-farm
      command: aws
      env: null
(python3ve) ➜  kubectl get svc
error: You must be logged in to the server (Unauthorized)

@kevin please give me a hint. Am I trying to access testing-farm EKS cluster or creating a new VPC/EKS? Thank you.

You should try and do the things you need to do, don't access testing-farm... I just meant that your account/user had the same access and they were able to make a cluster.

So, concentrate on the things you need to do and what errors that gives you and we can try and sort out permissions.

Note that if you need a new vpc, you have to ask us to make you one, we don't normally grant that perm to every account, we simply make them for you as needed.
What region will you be working in?

Q: What region will you be working in?
A: The same as Testing-farm, I believe it is us-east-1

I think we need a new VPC. However we can share the same VPC with testing-farm.
I do not know what is better.

So far it fails on:

[✖]  AWS::EC2::EIP/NATIP: CREATE_FAILED – "API: ec2:allocateAddress You are not authorized to perform this operation."
[✖]  AWS::EC2::InternetGateway/InternetGateway: CREATE_FAILED – "API: ec2:CreateInternetGateway You are not authorized to perform this operation."
[✖]  AWS::IAM::Role/ServiceRole: CREATE_FAILED – "API: iam:CreateRole User: arn:aws:iam::125523088429:user/fedora-ci-osci is not authorized to perform: iam:CreateRole on resource: arn:aws:iam::xxx:role/eksctl-osci-fargate-cluster-ServiceRole-xxx"

OR

eksctl create cluster --name osci-1                   
[ℹ]  eksctl version 0.20.0
[ℹ]  using region us-east-1
[ℹ]  setting availability zones to [us-east-1c us-east-1d]
[ℹ]  subnets for us-east-1c - public:192.168.0.0/19 private:192.168.64.0/19
[ℹ]  subnets for us-east-1d - public:192.168.32.0/19 private:192.168.96.0/19
Error: unable to determine AMI to use: error getting AMI from SSM Parameter Store: AccessDeniedException: User: arn:aws:iam::xxx:user/fedora-ci-osci is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:us-east-1:xxx:parameter/aws/service/eks/optimized-ami/1.16/amazon-linux-2/recommended/image_id
    status code: 400, request id: 59b1a024-1b44-4b10-a27e-60ce041d726f

Thank you for helping me.

If I try to use testing-farm VPC: it fails in the same way:

eksctl create cluster --name osci2 --vpc-private-subnets='subnet-03089904253762f32,subnet-0b84fdcd88b5803c2'
[ℹ]  eksctl version 0.20.0
[ℹ]  using region us-east-1
[✔]  using existing VPC (vpc-0896aedab4753e76f) and subnets (private:[subnet-0b84fdcd88b5803c2 subnet-03089904253762f32] public:[])
[!]  custom VPC/subnets will be used; if resulting cluster doesn't function as expected, make sure to review the configuration of VPC/subnets
Error: unable to determine AMI to use: error getting AMI from SSM Parameter Store: AccessDeniedException: User: arn:aws:iam::125523088429:user/fedora-ci-osci is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:us-east-1:125523088429:parameter/aws/service/eks/optimized-ami/1.16/amazon-linux-2/recommended/image_id
    status code: 400, request id: eaae4bbe-3cba-488b-9e0d-3eeeb327c786

Testing farm VPC has in totals 253 addresses. This means they will be able spin in total no more then ~230 pods at the same time. I doubt that it will be enough 230 pods for their needs, not to say to share this cluster with others.

Any way, command for creating cluster still fails no matter what.

Also, I cannot delete cluster, that one that I created with my web-account:

Could not delete cluster
User: arn:aws:sts::125523088429:assumed-role/aws-fedora-ci/astepano is not authorized to perform: eks:DeleteCluster on resource: arn:aws:eks:us-east-1:125523088429:cluster/astepano

The cluster was created with my web-account. But my web-login do not have rights to add nodes to it. Empty EKS cluster (without nodes) is also paid resource .

There are to many issues, I am not sure what the best approach. Each new issue/deny hides unresolved previous one.

(python3ve) ➜  ~ aws eks create-nodegroup --cluster-name osci2 --nodegroup-name ngroup-for-osci2 --disk-size 50 --subnets 'subnet-03089904253762f32' 'subnet-0b84fdcd88b5803c2' --remote-access 'ec2SshKey=astepano' --scaling-config 'minSize=2,maxSize=15,desiredSize=2' --ami-type AL2_x86_64 --instance-types t3.xlarge --node-role "arn:aws:iam::125523088429:role/fedora-ci-node-instance-role"  

An error occurred (AccessDeniedException) when calling the CreateNodegroup operation: User: arn:aws:iam::125523088429:user/fedora-ci-osci is not authorized to perform: eks:CreateNodegroup on resource: arn:aws:eks:us-east-1:125523088429:cluster/osci2

"Arn": "arn:aws:iam::125523088429:user/fedora-ci-osci" cannot create nodes for its own cluster.

@kevin I understand that Fedora infra team wants to have fine grained control over its users. However, could you please allow me configure resources from web-console? For my aws-fedora-ci/astepano account?

Crafting such long commands, and where each parameter has its own syntax, and to find its value it is necessary to run 5 more additional aws cli commands its very exhausting and time consuming. More frustrating that command results in another policy-deny. I tried different approaches, with aws and with eksctl commands. Both bumped into deny. I posted examples above. Hope for your understanding. Thank you.

PS
During my experiments I created some resources, that cannot be used. And my account cannot destroy them too. Exampe: EKS cluster, and CloudFormation stacks. There are a few of them. My point that approach : bump-into-deny-ask-for-permit is very time consuming for both sides. Plus creates a lot of unmanaged aws-resources. It would be fine if I can create necessary resources with web-console + access them from cli with secrets-account.

I'm sorry you are having issues. I am not able to work on this due to our datacenter move happening... well, NOW.

Perhaps @mobrien could look and see what iam perms we need to add? I can try and get him the current iam permissions.

So, I pondered some and one possible issue was that I had the policy locked down to specifically testing-farm for the cluster name.

I've relaxed that and the policy should allow any eks cluster now (but of course you and testing-farm have to make sure not to mess up each others resources.

I've mailed the current iam policy to @mobrien for a look...

HI astepano,

I'm sorry this is a slow process for you but as Kevin mentioned the DC move is happening at the moment which reduces bandwidth massively. It can be frustrating having to ask for permissions but security is important.

is it possible that you could provide us with a list of resources that you will be using? If you are using a cloudformation template as mentioned you can send me that and I will be able to work from there.

As Kevin mentioned you should use an existing VPC as that will mean you will not need any of the following permissions issues you mentioned earlier.

So far it fails on:

[✖] AWS::EC2::EIP/NATIP: CREATE_FAILED – "API: ec2:allocateAddress You are not authorized to >perform this operation."
[✖] AWS::EC2::InternetGateway/InternetGateway: CREATE_FAILED – "API: >ec2:CreateInternetGateway You are not authorized to perform this operation."
[✖] AWS::IAM::Role/ServiceRole: CREATE_FAILED – "API: iam:CreateRole User: >arn:aws:iam::125523088429:user/fedora-ci-osci is not authorized to perform: iam:CreateRole on >resource: arn:aws:iam::xxx:role/eksctl-osci-fargate-cluster-ServiceRole-xxx"

I have had a look at the IAM policy that Kevin sent me and it will need to be updated slightly to allow you access to the SSM parameter store to get the latest EKS AMI id however in the meantime you should be able to explicitly specify the AMI id (ami-011b077a6cc247f40)

if you are using eksctl you can use config files as outlined here https://eksctl.io/usage/creating-and-managing-clusters/ so you won't need to keep specifying cli arguments.

@mobrien hi, would it be a problem once you are doing it for astepano, also enable the same for the testing-farm related accounts please, if we could use eksctl that would be awesome. I gave up on that as I was at least able to work with awscli, but eksctl would be definitely a plus.

Not at all a priority for me, just wanted to state it here, seems doing those changes together makes most sense ...

@mobrien hello, it was not my intend to use cloudformation template.
It turned out that eksctl create cluster implicitly uses cloudformation.
For the start it would be great to have:

1. VPC, because testing farm VPC has only 250 IPs, including for their running containers.
2. It would be great to have: EKS backed with storage + access from the Internet to hosted apps on that EKS (I confirm that I can create EKS + node groups).
3. I want that *.ci.fedoraproject.org points to some of these running apps. (I have access to mange ci.fp.o zone.)

Thank you!

Thanks @astepano I will have to work with @kevin for the VPC work as I don't have the full access to create this. I can write some scripts for him to run, we can create both private and public subnets which would allow the eks cluster be contacted from the internet, then hopefully grant you the required permissions to create the cluster.

There is some rules to follow here when creating the VPC outside of eksctl that we will follow in the VPC creation that will also need to be addressed when creating the cluster.

Once the VPC is created and the permissions are updated you should be able to use eksctl to create the new cluster.

@mobrien @kevin thank you for your help.
It seems I created EKS cluster + nodegroups + I can access it from the Internet.
It is bound to testing-farm VPC. Let's close this ticket. Thank you both for help.

Using factory-team VPC seems is fine.
Pods use IP from internal EKS address space.
Services are exposed with real IPs.

(python3ve) ➜  help git:(master) ✗ kubectl get nodes -o wide
NAME                           STATUS   ROLES    AGE     VERSION              INTERNAL-IP    EXTERNAL-IP      OS-IMAGE         KERNEL-VERSION                  CONTAINER-RUNTIME
ip-10-123-0-184.ec2.internal   Ready    <none>   5h33m   v1.16.8-eks-e16311   10.123.0.184   54.162.151.195   Amazon Linux 2   4.14.177-139.254.amzn2.x86_64   docker://18.9.9
ip-10-123-0-96.ec2.internal    Ready    <none>   5h33m   v1.16.8-eks-e16311   10.123.0.96    54.172.98.51     Amazon Linux 2   4.14.177-139.254.amzn2.x86_64   docker://18.9.9
(python3ve) ➜  help git:(master) ✗ kubectl get svc          
NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)          AGE
example-service-ext4   LoadBalancer   172.20.250.99    a8c452a97e10c475f939e1c4eadd5125-2100752490.us-east-1.elb.amazonaws.com   80:30231/TCP     4h9m
example-service-int    NodePort       172.20.150.93    <none>                                                                    8080:32678/TCP   162m
hello-int              NodePort       172.20.180.160   <none>                                                                    8080:30714/TCP   53m
hello2                 LoadBalancer   172.20.188.178   aeccf11381c2f458490a860a98dbec6b-1158136003.us-east-1.elb.amazonaws.com   80:31652/TCP     4h14m
(python3ve) ➜  help git:(master) ✗ 

10.123.0.184/10.123.0.184 -- are from test-farming VPC. This I understand.

Real IPs: 54.172.98.51 / 54.162.151.195 for nodes --- this is magic for me. I didn't specify. However that is what I want.

 dig +short a8c452a97e10c475f939e1c4eadd5125-2100752490.us-east-1.elb.amazonaws.com
3.218.111.203
35.172.65.196

This is also magic for me. I am not quite sure understand how EKS-loadbalacer can propagate services to real-ip. But, it works.

Also I removed stuck cloudformations stacks:

(python3ve) ➜  help git:(master) ✗ aws cloudformation delete-stack  --stack-name eksctl-osci-cluster
(python3ve) ➜  help git:(master) ✗ aws cloudformation delete-stack  --stack-name eksctl-osci-fargate-cluster
+ I removed cluster a created from webconsole.

We are good. Let's close ticket. I will open a new one for other cases.
I appreciate your time and help.

Great, I'm glad it worked.

Wrt to the IPs eksctl adds the IPs to the nodes automatically. These are elastic IPs which are a free resource as long as they are attached.

The load balancer IPs are from a pool of IPs that aws has specifically for this purpose. They are not permanent and can change so don't use them for dns. If you want to point dns to a load balancer use a cname to the dns name you did the dig on.

@mobrien this is excellent explanation, thank you! Ticket can be closed.

@mobrien hi, would it be a problem once you are doing it for astepano, also enable the same for the testing-farm related accounts please, if we could use eksctl that would be awesome. I gave up on that as I was at least able to work with awscli, but eksctl would be definitely a plus.
Not at all a priority for me, just wanted to state it here, seems doing those changes together makes most sense ...

@mvadkert would you mind opening a separate ticket for this just in case there are any issues with it, it would be easier to track in a new ticket and it would allow us to close out this one. Thanks