#1350 websites: deploy new fedoraproject.org in prod
Merged a year ago by kevin. Opened a year ago by darknao.
fedora-infra/ darknao/ansible websites  into  main

@@ -277,28 +277,6 @@ 

      path: /index.html

      target: https://flocktofedora.org/

  

-   # Redirect specific websites from fedoraproject.org to getfedora.org

-   - role: httpd/redirect

-     shortname: main-fedoraproject

-     website: fedoraproject.org

-     path: /index.html

-     target: https://getfedora.org/

-     when: env != "staging"

- 

-   - role: httpd/redirect

-     shortname: get-fedora-old

-     website: fedoraproject.org

-     path: /get-fedora

-     target: https://getfedora.org/

-     when: env != "staging"

- 

-   - role: httpd/redirect

-     shortname: sponsors

-     website: fedoraproject.org

-     path: /sponsors

-     target: https://getfedora.org/sponsors

-     when: env != "staging"

- 

    - role: httpd/redirect

      shortname: code-of-conduct

      website: fedoraproject.org
@@ -317,26 +295,6 @@ 

      path: /code-of-conduct.html

      target: https://docs.fedoraproject.org/en-US/project/code-of-conduct/

  

-   - role: httpd/redirect

-     shortname: verify

-     website: fedoraproject.org

-     path: /verify

-     target: https://getfedora.org/verify

-     when: env != "staging"

- 

-   - role: httpd/redirect

-     shortname: keys

-     website: fedoraproject.org

-     path: /keys

-     target: https://getfedora.org/keys

-     when: env != "staging"

- 

-   - role: httpd/redirect

-     shortname: release-banner

-     website: fedoraproject.org

-     path: /static/js/release-counter-ext.js

-     target: https://getfedora.org/static/js/release-counter-ext.js

-     when: env != "staging"

  

  #

  # When there is no prerelease we redirect the prerelease urls

@@ -89,15 +89,12 @@ 

      objectname: fedora-websites-cron.yml

  

    # New websites 3.0

-   # STAGING ONLY

    - role: openshift/object

      app: websites

      file: obc.yml

      objectname: obc.yml

-     when: env == "staging"

  

    - role: openshift/object

      app: websites

      template: deployment.yml

      objectname: deployment.yml

-     when: env == "staging"

roles/fedora-web/main/files/cron-sync-fedora-web-v3 roles/fedora-web/main/files/cron-sync-fedora-web-stg
file renamed
file was moved with no change to the file
@@ -3,36 +3,26 @@ 

      state: present

      name:

      - s3cmd

-   when: env == "staging"

    tags:

    - fedora-web

    - fedora-web/main

  

- - name: Copy in the sync-fedora-web cronjob

-   copy: src=cron-sync-fedora-web dest=/etc/cron.d/sync-fedora-web

-   tags:

-   - fedora-web

-   - fedora-web/main

-   when: env != "staging"

- 

  - name: Load s3 credentials

    ansible.builtin.include_vars:

      file: "{{ private }}/files/websites/s3_fedoraproject_{{ env_short }}.yml"

    ignore_errors: true

-   when: env == "staging"

    tags:

    - fedora-web

    - fedora-web/main

  

- - name: Copy in the sync-fedora-web-stg cronjob

+ - name: Copy in the sync-fedora-web-v3 cronjob

    copy:

-     src: cron-sync-fedora-web-stg

+     src: cron-sync-fedora-web-v3

      dest: /etc/cron.d/sync-fedora-web

    tags:

    - fedora-web

    - fedora-web/main

    when:

-   - env == "staging"

    - fedoraproject_s3_bucket_name is defined

  

  - name: Create fedoraproject-sync script
@@ -44,7 +34,6 @@ 

    - fedora-web

    - fedora-web/main

    when:

-   - env == "staging"

    - fedoraproject_s3_bucket_name is defined

  

  - name: Make directory for the config files for {{website}} we are about to copy

@@ -8,48 +8,12 @@ 

    failedBuildsHistoryLimit: 2

    successfulBuildsHistoryLimit: 1

    source:

- {% if env == 'staging' %}

      git:

        uri: "https://gitlab.com/fedora/websites-apps/fedora-websites/fedora-websites-3.0.git"

-       ref: main

-     dockerfile: |-

-       FROM docker.io/library/node:18 as build

-       RUN apt-get update && apt-get install -y translate-toolkit && rm -rf /var/lib/apt/lists/*

-       ADD . /websites

-       WORKDIR /websites

-       RUN npm install

-       RUN npm run generate

- 

-       FROM quay.io/fedora/fedora:37

-       RUN dnf -y install s3cmd && dnf clean all

-       COPY --from=build /websites/.output/public /output

- 

+ {% if env == 'staging' %}

+       ref: staging

  {% else %}

-     dockerfile: |-

-       FROM fedora:34

-       RUN dnf -y install \

-         git \

-         nss_wrapper \

-         python-unversioned-command \

-         python3-flask \

-         python3-frozen-flask \

-         python3-flask-assets \

-         python3-rjsmin \

-         python3-cssmin \

-         python3-flask-babel \

-         python3-flask-htmlmin \

-         python3-cssutils \

-         python3-gnupg \

-         rubygem-sass \

-         babel \

-         python3-jinja2 \

-         python3-pyyaml \

-         python3-dateutil \

-         python3-dogpile-cache \

-         python3-requests \

-         python3-zanata-client && \

-           dnf clean all

-       CMD bash /etc/websites/build.sh

+       ref: main

  {% endif %}

    strategy:

      type: Docker

That's where all the fun begins :)

stg part, for references: https://pagure.io/fedora-infra/ansible/pull-request/1264

This is the first step of the new getfedora / fedoraproject replacement in prod. Here's the plan:
- removes fedoraproject.org redirects to getfedora.org.
- Provisions s3 storage on OCP.
- Deploys buildconfig / deployment on OCP to build the new website and push to s3.
- Installs s3cmd on proxies (used to sync s3 storage)
- Deploys the sync script with previously created s3 credentials.
- Profit

Then, on release day, we will create a redirect from getfedora.org (old) to fedoraproject.org (new).
I'll send a PR for that part later.

This is a 3 steps deployment:

Openshift part

First, we need to run the Openshift playbook (playbooks/openshift-apps/websites.yml) and wait for the s3 bucket to be created (usually takes ~2 minutes)

Then, we can retrieve the created S3 credentials and save them in /srv/private.
There are 3 values to fetch:
- AWS_SECRET_ACCESS_KEY
- AWS_ACCESS_KEY_ID
- BUCKET_NAME

The first two are stored in a secret and the third one in a configmap. They are both named fedoraproject-s3.

oc -n websites extract secrets/fedoraproject-s3 --to=-
oc -n websites extract cm/fedoraproject-s3 --to=-

The expected private vars file name is /srv/private/ansible/files/websites/s3_fedoraproject_prod.yml with the following variables in it:

fedoraproject_s3_access_key: <access_key>
fedoraproject_s3_access_key_id: <access_key_id>
fedoraproject_s3_bucket_name: <bucket_name>

Cleaning old redirects

A few files need to be manually removed on proxies to take care of the previous fedoraproject -> getfedora redirects:

 /etc/httpd/conf.d/fedoraproject.org/get-fedora-old-redirect.conf
 /etc/httpd/conf.d/fedoraproject.org/main-fedoraproject-redirect.conf
 /etc/httpd/conf.d/fedoraproject.org/sponsors-redirect.conf
 /etc/httpd/conf.d/fedoraproject.org/verify-redirect.conf

or you could just drop the whole /etc/httpd/conf.d/fedoraproject.org directory and the playbook will recreate everything.

Proxies part

Now that everything is in place, we can run the proxies playbook (playbooks/groups/proxies.yml).

As usual, I'll be around on irc if anything goes south :)

rebased onto 6b596e8

a year ago

rebased onto 6b596e8

a year ago

Pull-Request has been merged by kevin

a year ago