Good day.
In context of fedora-ci initiative may we ask to have an access to s3 bucket please? Currently we are deploying pipelines on AWS EKS: https://osci-jenkins-1.ci.fedoraproject.org/ We need to store artifacts and logs with easy way of maintenance.
I see there are two ways:
Allow to user/fedora-ci-osci token create s3 buckets on demand. This would be preferable solution. From our side I promise to follow resource-tagging conventions. This approach would allow us to define necessary access rules to the resources.
user/fedora-ci-osci
Create a bucket, with name fedora-ci-osci-01, than can be written from our aws-fedora-ci-resources, and RO for public.
fedora-ci-osci-01
aws-fedora-ci
I tried to create a s3 bucket:
aws-fedora-ci/astepano
aws sts get-caller-identity "Arn": "arn:aws:iam::125523088429:user/fedora-ci-osci"
aws s3 mb s3://osci-01 make_bucket failed: s3://osci-01 An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied
or
aws s3api create-bucket --bucket osci-01 --region us-east-1 An error occurred (AccessDenied) when calling the CreateBucket operation: Access Denied
No strict timeline. The earlier is the better. This would help us improve pipelines for fedora-ci testing.
This is related to the previous ticket and comment: https://pagure.io/fedora-infrastructure/issue/8958#comment-656531
Metadata Update from @kevin: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-gain, medium-trouble, ops
So, we already set this up a while back with a bucket named 'fedora-ci-bucket'
Can you just use that one? or would you like a fedora-ci-osci-01 one?
It seems the bucket is empty:
aws s3 ls s3://fedora-ci-bucket --recursive aws s3 cp readme s3://fedora-ci-bucket/ upload: ./readme to s3://fedora-ci-bucket/readme
I cannot find who exactly requested that bucket. Google search doesn't help
site:pagure.io 'fedora-ci-bucket'
If it is OK to use it sure, we can use it.
I tried to upload file with public read access. But it fails:
~ aws s3api put-object-acl --bucket fedora-ci-bucket --key readme --acl public-read An error occurred (AccessDenied) when calling the PutObjectAcl operation: Access Denied ~ aws s3 rm s3://fedora-ci-bucket/readme delete: s3://fedora-ci-bucket/readme ~ aws s3 cp readme s3://fedora-ci-bucket/ --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers upload failed: ./readme to s3://fedora-ci-bucket/readme An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
It seems we cannot put for public data.
Also, I am not very confident AWS user, how do we upload files to fedora-ci-bucket from fedora-ci resources? Is there way to do this without token? The point is: the token I have has power to modify all AWS resources, which is not safe.
fedora-ci-bucket
Thank you for help.
I can add those perms to it. I thought it had s3* set on it for your role...
We could make another user/token with just s3 access if you like?
If this is the right way to upload files from ci-pipelines, than yes, could you please create a token with limited access to the s3 bucket. So we can use that token for managing s3 files.
I can add those perms to it. I thought it had s3* set on it for your role... Please, if possible.
Thank you very much for helping, I appreciate you help.
Friendly ping :-) Thank you!
ok, sorry for the delay here. ;(
IN your homedirectory on batcave01 is: fedora-ci-s3.csv that has the info in it.
Please delete the file once you have copied it off and do keep the token as secure as you are able.
It should have perms to that s3 bucket.
Feel free to re-open if you run into problems or there's more to do.
Metadata Update from @kevin: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
@kevin, hello,
I checked the token. It works. However, ... Could you please tweak permissions please, that we can use this token to store files with public access? Here is some logs:
➜ aws sts get-caller-identity --profile s3 { "UserId": "AIDAR2OOCKQWSX6Y2WMU2", "Account": "125523088429", "Arn": "arn:aws:iam::125523088429:user/fedora-ci-s3" }
Next OK:
➜ export AWS_PROFILE=s3 ➜ aws s3 cp readme s3://fedora-ci-bucket/ upload: ./readme to s3://fedora-ci-bucket/readme ➜ aws s3 ls s3://fedora-ci-bucket --recursive 2020-10-08 10:18:53 51 readme ➜ aws s3 rm s3://fedora-ci-bucket/readme
Next fails:
➜ aws s3api put-object-acl --bucket fedora-ci-bucket --key readme --acl public-read An error occurred (AccessDenied) when calling the PutObjectAcl operation: Access Denied
➜ aws s3 cp readme s3://fedora-ci-bucket/ --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers upload failed: ./readme to s3://fedora-ci-bucket/readme An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
Could you please change config, that we could store files with RO for all?
Thank you!!!
Metadata Update from @astepano: - Issue status updated to: Open (was: Closed)
Issue status updated to: Closed (was: Open) Issue close_status updated to: Fixed
@pingou hi,
I just checked, the token doesn't have permission to upload files to s3 for public access. I am not sure why you moved ticket to Fixed.
Sorry for the long delay here. ;(
Can you try again now? I think we were missing a * at the end of the policy, so it wasn't allowing things for anything but the bucket name/top level.
ok, we got it figured out on IRC.
Turns out there is a default set 'block all public access' checkbox. You have to uncheck this for that bucket in order for it to allow you to make things public. :)
I am not sure why you moved ticket to Fixed.
My apologies, I do not know how I did this, must have made a mistake somewhere
Login to comment on this ticket.