I will be the sponsor for this request.

I will be the sponsor for this request.

So, can you explain how this will work?

Perhaps it would be good for a overview in the next meeting?
Or just a high level overview here.

What ssh keys would these hosts have? What ansible would they use?

Initial draft of the docs are here: https://pagure.io/fork/maxamillion/releng/blob/docs/f/docs/source/automation_engine.rst

I'm ssh keys would theoretically be a new releng-automation user with sudo permissions assigned to it so that it can execute the specific releng utils necessary. We would want to use rbac-playbook (loopabull can be configured to use that, Patrick submitted the patch to enable that).

Chat logs from our recent discussion are below

11:18 < nirik> basically how about this: can you walk me through what happens when a fedmsg comes in that you want to act on?
11:19 < puiterwijk> Me? Or Adam?
11:20 < nirik> well, either way. ;) I just want to know what runs where...
11:21 < nirik> or will rather
11:21 < puiterwijk> Okay, so the plan was that we have a relengauto01 box (for lack of a better name) that'll run loopabull. It also has a git clone (or some other method) of the playbooks, and has flr installed
11:22 < puiterwijk> So everything runs on relengauto01, as a loopabull user. That user can use rbac-playbook to execute ansible playbooks as another user (loopabull-runner or something) that has a private key with access to some machines
11:23 < puiterwijk> maxamillion: ^ please let me know if I'm remembering something incorrectly
11:23 < nirik> and it has sudo on those machines ? or ssh as root?
11:24 < maxamillion> nirik: sudo
11:24 < maxamillion> puiterwijk: I don't care what we call the various account names, but yes ... that's the gist
11:24 < maxamillion> jist*
11:24 < puiterwijk> It's limited sudo rules, to just the commands it needs
11:24 < maxamillion> nirik: and it would only have permissions to run whitelisted commands, which we can write up a review process for
11:24 < maxamillion> +1
11:25 < nirik> so to add something we have to add to sudoers on relengauto01 (so it can run the playbook) and sudoers on the target machine to make the user running have sudo to do whatever?
11:26 < puiterwijk> well, "whatever" is "the commands it needs to run on that machine", but yes
11:26 < nirik> so, does this need to be releng specific?
11:27 < nirik> and is there an example of something you want to add right away?
11:31 < puiterwijk> I don't think it needs to be releng specific, but it should only run trusted playbooks and programs
11:32 < puiterwijk> As for right away stuff, I'll leave that to Adam
11:33 < nirik> I can think it might be handy for other places in infrastructure too... that want to do something on a fedmsg.
11:33 < nirik> we have the ones on batcave... and granted those might go away, but I bet we can think of more.
11:34 < puiterwijk> That sounds reasonable to me. We just need to be careful about the auditing of the code and playbooks.
11:35 < nirik> true.
11:35 < puiterwijk> For the releng automation stuff, we've said that before going into master, it needs to pass security review by "the Security Officer or any appointed persons"
11:35 < nirik> so perhaps we should just call the instances loopabull01 or something...
11:35 < puiterwijk> (which for releng is currently Adam)
11:35 < nirik> thats completely fair/good
11:35 < puiterwijk> And sure, loopabull01 works for me
11:36 < puiterwijk> maxamillion: any opinion?
11:37 < maxamillion> puiterwijk: sorry, multitasking badly ... let me catchup
11:38 < maxamillion> yeah, if this is something that could be useful beyond releng I'd be happy to make it more generic
11:40 < nirik> maxamillion: do you have an initial thing/things you want to add to it?
11:41 < maxamillion> nirik: just one, 'flr-docker remotecopy' and 'flr-docker removeimage' (the latter doesn't exist yet) -> https://pagure.io/flr/blob/master/f/flr-docker
11:42 < maxamillion> nirik: package review was approved this morning for the 0.0.1 release that I put out so we could test all this stuff
11:42 < maxamillion> nirik: 0.0.1 release of flr *
11:43 < nirik> so, it listens for a docker build complete, then runs a playbook that copies it somewhere?
11:43 < nirik> puiterwijk: so this box/these boxes will need a ssh agent loading the key on reboot right?
11:43 < puiterwijk> nirik: yes
11:48 < maxamillion> nirik: n, there's not a playbook yet for it yet but those will live over here --> https://pagure.io/releng-automation (or whatever, if we want to move it to something more generic) but it will wait for a completed successful container test from taskotron and then copy it to a "stable" registry (some details on time schedule and the like are still to be sorted, we might have a special fedmsg to
11:48 < maxamillion> kick off the playbook and the playbook will query datagrepper or something)
11:49 < nirik> I think we can leave it at that for the releng stuff, if we want more other generic things using it we can put them in another repo or whatever. I imagine it would pull this over periodically
11:50 < maxamillion> nirik: alright, sounds good
11:51 < nirik> ok, so sounds ok to me... it's going to be a bit fniky I predict to get all the sudo rules right for things, but it does provide a nice additional security measure, so thats worth it.
11:55 < maxamillion> nirik: agreed, I'd like to try and document the sudo stuff so there's a reference/process for adding new commands to be whitelisted by the sudo rules as we add for functionality to the releng automation

Is there anything else needed on my end?

The final question I'm seeing in the RFR SOP that needs answering for the current step: * Who all is involved in maintaining and deploying the instance?

At the beginning it would just be puiterwijk and myself. From there my plan is to have documented procedures for how to allow new playbooks/tools added to the automation tooling as well as how new community members can apply to join the work effort.

If we are going with the more generic version that we can use for automation around infrastructure, I'd also be happy to help maintain/deploy and write docs.

:thumbsup:

Okay, then I think we have everything to move on to the Development Instance phase.

Needed results to continue from here:

• MUST have RFR sponsor sign off that the resource is ready to move to
  the next step.

• MUST have answered any outstanding questions on the infrastructure
  list about the resource. Decisions about caching, load balancing and
  how the resource would be best deployed should be determined.

• MUST add any needed SOP's for the service. Should there be an Update
  SOP? A troubleshooting SOP? Any other tasks that might need to be done
  to the instance when those who know it well are not available?

So, Adam, let me know if you need/want a development instance, and please start writing SOPs for the Fedora Infra Docs repo (https://infrastructure.fedoraproject.org/infra/docs/).

I don't think a development instance is needed, only staging and production. Thank you.

SOP started, will update as we get things deployed.

https://infrastructure.fedoraproject.org/infra/docs/loopabull.rst

Sorry for the big delay here.
I am signing off on moving towards the Staging instance phase.

Please start writing the playbook for this, the staging instance should be defined soon.

The staging host and empty playbook is created. Please let me know when you have a working playbook and want to move forward or if you have any further questions.

ok, whats the status here now. We have the instances, but they aren't yet running?

This is waiting on docs/SOPs and monitoring and then it should be done.

So, should we keep this open? We are using loopabull for a few things.

There's a prod instance now also, and a very short doc. Should we just consider it done?

This is all done and in prod. Thanks!

:bullettrain_side: