Documentation: https://docs.aws.amazon.com/eks/latest/best-practices/cas.html

Done. Can you please test?

@kevin hmm, I cannot see it?

https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-2#/roles/details/fedora-ci-eks-autoscaler?section=permissions

Oh, I set it up as a policy instead of a role.

https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-2#/policies/details/arn%3Aaws%3Aiam%3A%3A125523088429%3Apolicy%2Ffedora-ci-testing-farm-autoscaler?section=permissions

I'm not sure how to add a role for this. I guess I'll read over the docs more.

it should be simple:

1. https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/roles/create
2. select Custom trust policy
3. Paste in the json I gave you, next, next, give it the name I provided

Should be done

ah, I see...

well, it doesn't like it. ;)

Ln 4, Col 8

Missing Principal: Add a Principal element to the policy statement. Learn more 
Ln 10, Col 24

Role Trust Policy Syntax Error Resource: Role trust policies apply to the role that they are attached to. You cannot specify a resource. Remove the Resource or NotResource element. Learn more 
Ln 18, Col 8

Missing Principal: Add a Principal element to the policy statement. Learn more 
Ln 32, Col 24

Role Trust Policy Syntax Error Resource: Role trust policies apply to the role that they are attached to. You cannot specify a resource. Remove the Resource or NotResource element. Learn more 

Any ideas? I can't get past the syntaxt errors with the proposed one (even from their docs)... so I am surely doing something wrong. ;(

Any ideas what to try here?

Asked @mwinters if he can try to help here.

Thanks for ringing me in. Happy to help! I've included the above details in this post so you should be able to just follow it top to bottom.

Preface: Use Karpenter

First, I highly recommend using Karpenter to autoscale your clusters instead of the vanilla Cluster Autoscaler. It is slightly more complex and another tool to learn, but it simplifies many other admin tasks including cluster upgrades, cost optimization, and security.

But, I'll assume here that we want to use Cluster Autoscaler anyway for simplicity.

Pod IAM Credentials with PIA

There are several ways to get AWS credentials into a pod. I'm operating entirely blind here -- I have no access to Fedora AWS and no idea what our preferences / policies / norms are. But the modern recommended approach is to use the "Pod Identity Agent", aka PIA, which is both a Kubernetes add-on and a set of AWS IAM features. This allows you to explicitly and easily create a chain of IAM Policy -> IAM Role -> k8s Service Account -> My Pod.

My assumptions

Based on what I've gathered from random convos, I'll assume:
- Most setup is happening via the AWS Console, not IAC or CLI.
- We do not need cross-cluster or cross-account (AWS account) permissions.
- We want to use IAM best practices for isolation.

Step 1: Setup PIA in your cluster

With EKS Auto Mode

If you're using EKS Auto Mode then you are probably already done, unless your cluster is very old. PIA is installed by default into Auto Mode clusters. You can check this with kubectl get pods -n kube-system | grep 'eks-pod-identity-agent'.

Without Auto Mode

Step 1.1: Grant the worker node's IAM role permissions

• If the worker nodes are using an IAM role which includes the AWS-managed policy AmazonEKSWorkerNodePolicy then you are already done.
• If not, add this policy to the IAM role:

{
    "Version":"2012-10-17",              
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "eks-auth:AssumeRoleForPodIdentity"
            ],
            "Resource": "*"
        }
    ]
}

Step 1.2: Enable PIA in the EKS console

1. AWS Console -> EKS -> Clusters -> My Cluster
2. Add-ons -> Get more add-ons -> EKS Pod Identity Agent -> Next
3. If necessary, fill out any additional config that a k8s Service Account in your cluster would need, e.g. ImagePullSecrets.
4. Next. Wait a few minutes and you should see PIA running with kubectl get pods -n kube-system | grep 'eks-pod-identity-agent'.

Step 2: Create an IAM Policy with the permissions needed by your pod

1. AWS Console -> IAM -> Policies -> Create Policy -> JSON mode
2. Paste the following and click Next:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:ResourceTag/k8s.io/cluster-autoscaler/enabled": "true"
                },
                "StringLike": {
                    "aws:ResourceTag/k8s.io/cluster-autoscaler/testing-farm-*": "owned"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeScalingActivities",
                "autoscaling:DescribeTags",
                "ec2:DescribeImages",
                "ec2:DescribeInstanceTypes",
                "ec2:DescribeLaunchTemplateVersions",
                "ec2:GetInstanceTypesFromInstanceRequirements",
                "eks:DescribeNodegroup"
            ],
            "Resource": "*"
        }
    ]
}

3. Name the policy something like foocluster_autoscaling -> set Tags (because you do this on everything, right? right??!!!) -> Create Policy.

Step 3: Attach the Policy to a new IAM Role

1. AWS Console -> IAM -> Roles -> Create Role -> Custom Trust Policy
2. Paste the following trust policy and click Next:

{
    "Version":"2012-10-17",              
    "Statement": [
        {
            "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}

3. Search for the foocluster_autoscaling policy we just created -> click the checkbox next to it -> Next
4. Name the role something like foocluster_autoscaling (yes, the role name is the same as the policy name) -> set tags -> Create role.

Step 4. Create a Service Account for this app in your Kubernetes cluster

Note: You don't have to do this just yet. But if you do, then it will show up in the GUI in the next step instead of requiring you to type it.

kubectl -n kube-system create serviceaccount cluster-autoscaler

Step 5. Associate the k8s Service Account with the AWS IAM Role

1. AWS Console -> EKS -> Clusters -> My Cluster
2. Access tab -> Pod identity associations -> Create
3. Select the foocluster_autoscaling role, the Kubernetes namespace (kube-system in this case, I think), and the cluster-autoscaler Service Account from that namespace.
- If step 4 above isn't done yet, you can just hardcode these values and they'll apply whenever someone does that step.

Step 6. Make sure the pod isn't using the default Service Account

spec:
  serviceAccountName: cluster-autoscaler
  automountServiceAccountToken: false

#HMU

You know where to find me if you have questions! And what FAS groups to add me to if you don't any more! ;)

I can do the IAM parts of this... but I don't have access to the eks cluster at all, so I can't see those parts of it.

@mvadkert Thoughts? Do you want to look at Karpenter? it looks interesting... but of course up to you.

I can do the IAM parts of this... but I don't have access to the eks cluster at all, so I can't see those parts of it.

@mvadkert Thoughts? Do you want to look at Karpenter? it looks interesting... but of course up to you.

My $0.02: if you have time to apply the change, this shouldn't break anything since Cluster Autoscaler already can't work. So I'd say: go for it. You'll just have to type the namespace and serviceaccount in Step 5 instead of selecting them in the GUI.