#6182 [RFR] On Demand Compose Service VMs
Closed: Fixed 4 years ago Opened 4 years ago by sochotni.

This is a request for resources for new Factory 2 service: On Demand Compose Service (ODCS).

Focus doc for ODCS for more background information:
Sources: https://pagure.io/odcs

Requested 4 VMs: stg and prod, a frontend and a backend for each.
- 2 vCPUs
- 4GB ram for stg/prod
- 20 GB HDD

Related setup requests:
* We'll need /mnt/koji read-only access.
* We need a gluster/nfs share between the frontend and backend nodes in both stg and prod.

@ralph is the infra sponsor.

See #6167 for the associated security audit.

Like in #6183 (which was for freshmaker), here can I get a :+1: from another sysadmin to start provisioning these VMS even though the security audit of the app code isn't complete? Of course, we'll be around to fix any issues found before we go live. Getting a head start on deployment configuration work would be helpful at this point.

Metadata Update from @pingou:
- Issue tagged with: security

4 years ago

Metadata Update from @pingou:
- Issue tagged with: request-for-resources

4 years ago

OK - I created stg nodes and I claimed IPs for the prod node, although they're commented out in the ansible inventory so they won't be created.

FYI - a missing piece here is that we need a nfs share served from the backend to the frontend (in staging and eventually prod).

@ralph You could look at the docker-registry.yml or ask.yml playbooks as to how we usually set up gluster shared storage.

OK - first attempt at gluster shared storage to be found here.

After a few tweaks, it works. /srv/odcs is the currently mounted shared location.

@qwan, @cqi, @jkaluza - can you guys move forwards with setting up roles for the app?

@ralph: We will need database and keytab to contact Koji before deploying. I will also try writing the ODCS SOP today.

The openidc staging client and scopes have been added here: https://id.stg.fedoraproject.org/openidc/.well-known/openid-configuration

The staging database is created, with corresponding secrets in the secrets repo.

After talking in IRC, we see now that we don't actually need a keytab to talk to koji. All requests can be done anonymously.

$ curl https://odcs.stg.fedoraproject.org/odcs/1/composes/
  "items": [],
  "meta": {
    "first": "http://odcs.stg.fedoraproject.org/odcs/1/composes/?per_page=10&page=1",
    "last": "http://odcs.stg.fedoraproject.org/odcs/1/composes/?per_page=10&page=0",
    "next": null,
    "page": 1,
    "pages": 0,
    "per_page": 10,
    "prev": null,
    "total": 0

OK - I missed a crucial piece here.

The odcs-backend01.stg and odcs-backend01.phx2 VMs need read-only access to /mnt/koji.

  • The odcs-backend01.phx2 (prod) VM needs it mounted in the standard place - /mnt/koji.
  • The odcs-backend01.stg (stg) VM is expecting to find the content in /mnt/koji/vol/prod/, but it can be mounted somewhere else like in /mnt/fedora_koji_prod just like on koji01.stg.

    [koji01 ~][STG]# ls -alh /mnt/koji/vol/prod
    lrwxrwxrwx. 1 root root 26 Jun 16 00:05 /mnt/koji/vol/prod -> /mnt/fedora_koji_prod/koji

I guess this needs a ticket to change the network ACLs to allow the read-only mount from these hosts?

For the record, @puiterwijk has asked us to fill out the details here a bit more formally... so I'm going to use his template from infra-docs#64

Phase I

  • Software: On Demand Compose Service (ODCS)
  • Advantage for Fedora: Provide an easy way for non-releng people to request temporary/throwaway composes, primarily for testing.
  • Sponsor: @ralph

Phase II

Phase III

Phase IV

the fedora_koji netapp volume is exported ro to ie, the entire storage network.

Just adding a eth1 (installing with a two_nics option and a eth1 ip set) should allow you to mount it.

Metadata Update from @puiterwijk:
- Issue untagged with: security

4 years ago

Mounting /mnt/koji is now set in staging.
As the other systems, it has /mnt/koji which is the staging koji (/mnt/fedora_koji/koji), and it has /mnt/koji/vol/prod, which is a symlink to the prod NFS mount on /mnt/fedora_koji_prod.

One headsup: all of these NFS mounts (including /mnt/koji in staging) are readonly.
For production, an NFS IP will need to be assigned before the role can be enabled.

@ralph: I will also need rights to do "sudo journalctl -u odcs-backend.service" on the backend nodes, would it be possible to set it up?

@jkaluza, yes. It should be set now in ansible.

OK - ODCS should be up here: https://odcs.fedoraproject.org/api/1/composes/

What else do we need before closing this out?

  • [X] SOP
  • [ ] nagios service-level checks

Anything else?

I think we're good to close this now?

Metadata Update from @kevin:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

4 years ago

Login to comment on this ticket.