#11848 AWS: unable to provision spot instances for Fedora CI
Closed: Fixed 10 months ago by mvadkert. Opened 10 months ago by mvadkert.

Hi,

seems our user arn:aws:iam::125523088429:user/fedora-ci-testing-farm does not have permissions to provision spot instances. I believe copr folks have this on their users.

At least RequestSpotInstances is missing.

Thank you!

│     poolname: fedora-aws-x86_64-metal                                                                                                                                                                                                     │
│     commandname: aws.ec2-request-spot-instances                                                                                                                                                                                           │
│     scrubbed_command: "aws ec2 request-spot-instances --spot-price=1.4516250000000002\                                                                                                                                                    │
│         \ '--launch-specification={ \"ImageId\": \"ami-0f453779d521e1e88\", \"KeyName\"\                                                                                                                                                  │
│         : \"testing-farm-worker\", \"InstanceType\": \"c7i.metal-24xl\", \"Placement\"\                                                                                                                                                   │
│         : { \"AvailabilityZone\": \"us-east-2c\" }, \"NetworkInterfaces\": [{\"DeviceIndex\"\                                                                                                                                             │
│         : 0, \"SubnetId\": \"subnet-4f971734\", \"DeleteOnTermination\": true, \"\                                                                                                                                                        │
│         Groups\": [\"sg-09babf993e181a81f\"], \"AssociatePublicIpAddress\": true}],\                                                                                                                                                      │
│         \ \"BlockDeviceMappings\": [{\"DeviceName\": \"/dev/sda1\", \"Ebs\": {\"DeleteOnTermination\"\                                                                                                                                    │
│         : true, \"SnapshotId\": \"snap-0970e16f3cbae0b98\", \"VolumeSize\": 100, \"\                                                                                                                                                      │
│         VolumeType\": \"gp3\", \"Encrypted\": false}}], \"UserData\": \"\" }' --tag-specifications\                                                                                                                                       │
│         \ 'ResourceType=spot-instances-request,Tags=[{Key=FedoraGroup,Value=ci},{Key=ServiceName,Value=Artemis},{Key=ServiceOwner,Value=TFT},{Key=ServicePhase,Value=Dev},{Key=ArtemisGuestName,Value=8cf61c63-7a8b-42b6-bf1d-45eb3d02d6b │
│     command_output:                                                                                                                                                                                                                       │
│         stdout: ''                                                                                                                                                                                                                        │
│         stderr: |4                                                                                                                                                                                                                        │
│                                                                                                                                                                                                                                           │
│             An error occurred (UnauthorizedOperation) when calling the RequestSpotInstances operation: You are not authorized to perform this operation. User: arn:aws:iam::125523088429:user/fedora-ci-testing-farm is not authorized to │
│     message: error running CLI command   

Metadata Update from @phsmoura:
- Issue assigned to kevin
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble, ops

10 months ago

Strange. You should have the RequestSpotInstances permssion, I see it in your policy as allowed.

I don't see anything obvious here. ;(

I also checked against the copr policy and I don't see anything there off hand thats not in the fedora-ci ec2 policy...

@praiskup any ideas here?

I noticed we had RunInstances, but the perm should be Runinstance, so I changed it to that... can you try now?

yeah sorry, after looking again as it is failing, I did not see the whole output as it was wrapped in k9s.

│             An error occurred (UnauthorizedOperation) when calling the RequestSpotInstances operation: You are not authorized to perform this operation. User: arn:aws:iam::125523088429:user/fedora-ci-testing-farm is not authorized to │
│  perform: ec2:CreateTags on resource: arn:aws:ec2:us-east-2:125523088429:spot-instances-request/* because no identity-based policy allows the ec2:CreateTags action. Encoded authorization failure message: tC-KZxvB_fUq4QJvslbk5ijMbm13h │
│ 8Ztcepv0lx6ClVBFwthMB87FZbYvtvDRgAZxpu4e1Bbyl-vbooyEM_QKv0M9lgqb8zKlxRWyzq_Q41LtN5DiQBC0g53ZkaH0J6RHlGHhembqaXq1YH3sIKZrtASfz2x723cLrph9x6ai3ZbZAXf8c2PHLUyQtqSrimDvJdFHYMSxNsJlSJEiyid8AB96mZnGbWekOnDo2ZdGHpsInD1LMA62kCQK4s0cT2Y7Ot88A │
│ bmRlMpKGdWM6kHI7AYgT820rvdKD7TrM6ymARF9h6ffef8_WJp8kmRUTt2dl-6jSogatQFg5DD6hU-1Jj-_lytlqgvapoDcMjPa_kLBMjVE5UeFh61MdjWUWd-PCN0wzadY_Ei3V6QXaO5UMo3pG8i48G-HZC4jMCnuGfj7PzMNTljAlWSVLx-45GF4CE80AZkQr-xlfg84jFD3a7KMBebYsKaJ3x5sFnSo1lMccC │
│ GrtsJ-ABHMTHwliApamhyDZ0mdOJSsfvqaEqOraWAUOI-GofyyA   
An error occurred (UnauthorizedOperation) when calling the RequestSpotInstances operation: You are not authorized to perform this operation. User: arn:aws:iam::125523088429:user/fedora-ci-testing-farm is not authorized to perform: ec2:CreateTags on resource: arn:aws:ec2:us-east-2:125523088429:spot-instances-request/* because no identity-based policy allows the ec2:CreateTags action

ok. Added ec2:CreateTag

Try again?

looks better, but I am still seeing some issues when cancelling spot requests

An error occurred (UnauthorizedOperation) when calling the CancelSpotInstanceRequests operation: You are not authorized to perform this operation. User: arn:aws:iam::25523088429:user/fedora-ci-testing-farm is not authorized to perform: ec2:CancelSpotInstanceRequests on resource: arn:aws:ec2:us-east-2:125523088429:spot-instances-request/sir-w1aevc6q because no identity-based policy allows the ec2:CancelSpotInstanceRequests action. Encoded authorization failure message: 

All good, closing, and ty for quick response!

Metadata Update from @mvadkert:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

10 months ago

Log in to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog