pagure

#5497 [WIP] Containerfiles and manifests to build images and run pagure on k8s

Opened 9 months ago by wombelix. Modified 7 months ago

wombelix/pagure feat_containers_and_kubernetes into master

chore: allow alembic.ini outside root project folder.

Dominik Wombacher • 9 months ago

f00b337

feat(containerfiles): First version of definitions and entrypoints

Dominik Wombacher • 9 months ago

7edc269

.gitignore

file modified

+1 -1

		`@@ -41,7 +41,7 @@`
		`tests/*.git`

		`# Don't track alembic.ini`
		`- alembic.ini`
		`+ /alembic.ini`

		`# Don't track the virtualenv we tell developers to use`
		`pagure_env/`

files/containerfiles/alembic.ini

file added

+37

		`@@ -0,0 +1,37 @@`
		`+ [alembic]`
		`+ script_location = /pagure/alembic`
		`+ sqlalchemy.url = postgresql://pagure:pagure@postgresql/pagure`
		`+`
		`+ [loggers]`
		`+ keys = root,sqlalchemy,alembic`
		`+`
		`+ [handlers]`
		`+ keys = console`
		`+`
		`+ [formatters]`
		`+ keys = generic`
		`+`
		`+ [logger_root]`
		`+ level = WARN`
		`+ handlers = console`
		`+ qualname =`
		`+`
		`+ [logger_sqlalchemy]`
		`+ level = WARN`
		`+ handlers =`
		`+ qualname = sqlalchemy.engine`
		`+`
		`+ [logger_alembic]`
		`+ level = INFO`
		`+ handlers =`
		`+ qualname = alembic`
		`+`
		`+ [handler_console]`
		`+ class = StreamHandler`
		`+ args = (sys.stderr,)`
		`+ level = NOTSET`
		`+ formatter = generic`
		`+`
		`+ [formatter_generic]`
		`+ format = %(levelname)-5.5s [%(name)s] %(message)s`
		`+ datefmt = %H:%M:%S`

files/containerfiles/entrypoint-pagure-docs.sh

file added

		`@@ -0,0 +1,5 @@`
		`+ #!/bin/bash`
		`+`
		`+ source /venv/bin/activate`
		`+`
		`+ exec gunicorn "pagure.docs_server:APP" --bind 0.0.0.0 --access-logfile - --error-logfile - $@`

files/containerfiles/entrypoint-pagure-ev.sh

file added

		`@@ -0,0 +1,5 @@`
		`+ #!/bin/bash`
		`+`
		`+ source /venv/bin/activate`
		`+`
		`+ exec python /pagure/pagure-ev/pagure_stream_server.py`

files/containerfiles/entrypoint-pagure-server.sh

file added

+13

		`@@ -0,0 +1,13 @@`
		`+ #!/bin/bash`
		`+`
		`+ source /venv/bin/activate`
		`+`
		`+ if [ ! -f /attachments/inited ];`
		`+ then`
		`+ python /pagure/createdb.py`
		`+ && touch /attachments/inited`
		`+ else`
		`+ alembic upgrade head`
		`+ fi`
		`+`
		`+ exec gunicorn "pagure.flask_app:create_app()" --bind 0.0.0.0 --access-logfile - --error-logfile - $@`

files/containerfiles/entrypoint-pagure-worker.sh

file added

		`@@ -0,0 +1,7 @@`
		`+ #!/bin/bash`
		`+`
		`+ source /venv/bin/activate`
		`+`
		`+ # for logcom pass as ARGS/CMD:`
		`+ # --tasks "pagure.lib.tasks_services" --queue "pagure_logcom"`
		`+ exec python /pagure/runworker.py $@`

files/containerfiles/pagure-base

file added

+45

		`@@ -0,0 +1,45 @@`
		`+ FROM quay.io/almalinuxorg/9-base`
		`+`
		`+ LABEL org.opencontainers.image.authors="pagure community"`
		`+ LABEL org.opencontainers.image.url="https://pagure.io/pagure"`
		`+ LABEL org.opencontainers.image.source="https://pagure.io/pagure"`
		`+ LABEL org.opencontainers.image.documentation="https://docs.pagure.org/pagure/index.html"`
		`+`
		`+ ARG repo=https://pagure.io/pagure.git`
		`+ ARG branch=master`
		`+`
		`+ ENV REPO=$repo`
		`+ ENV BRANCH=$branch`
		`+ ENV ALEMBIC_CONFIG=/alembic.ini`
		`+ ENV PAGURE_CONFIG=/pagure.cfg`
		`+`
		`+ RUN dnf -y install \`
		`+ python3 \`
		`+ git \`
		`+ python3-devel \`
		`+ gcc \`
		`+ zlib-devel \`
		`+ libjpeg-devel \`
		`+ && git clone $REPO -b $BRANCH \`
		`+ && python3 -m venv /venv \`
		`+ && source /venv/bin/activate \`
		`+ && python -m pip install --upgrade pip build \`
		`+ && python -m pip install -r /pagure/requirements.txt \`
		`+ -r /pagure/requirements-ci.txt \`
		`+ -r /pagure/requirements-optional.txt \`
		`+ -r /pagure/requirements-testing.txt \`
		`+ gunicorn \`
		`+ && cd /pagure \`
		`+ && python setup.py build \`
		`+ && python setup.py install \`
		`+ && rm -rf build/ \`
		`+ && python -m pip cache purge \`
		`+ && dnf -y remove \`
		`+ python3-devel \`
		`+ gcc \`
		`+ zlib-devel \`
		`+ libjpeg-devel \`
		`+ && dnf clean all`
		`+`
		`+ COPY alembic.ini /alembic.ini`
		`+ COPY pagure.cfg /pagure.cfg`

files/containerfiles/pagure-docs

file added

+12

		`@@ -0,0 +1,12 @@`
		`+ FROM pagure-base:latest`
		`+`
		`+ LABEL org.opencontainers.image.authors="pagure community"`
		`+ LABEL org.opencontainers.image.url="https://pagure.io/pagure"`
		`+ LABEL org.opencontainers.image.source="https://pagure.io/pagure"`
		`+ LABEL org.opencontainers.image.documentation="https://docs.pagure.org/pagure/index.html"`
		`+`
		`+ VOLUME ["/data"]`
		`+`
		`+ COPY entrypoint-pagure-docs.sh /entrypoint.sh`
		`+`
		`+ ENTRYPOINT ["/entrypoint.sh"]`

files/containerfiles/pagure-ev

file added

+12

		`@@ -0,0 +1,12 @@`
		`+ FROM pagure-base:latest`
		`+`
		`+ LABEL org.opencontainers.image.authors="pagure community"`
		`+ LABEL org.opencontainers.image.url="https://pagure.io/pagure"`
		`+ LABEL org.opencontainers.image.source="https://pagure.io/pagure"`
		`+ LABEL org.opencontainers.image.documentation="https://docs.pagure.org/pagure/index.html"`
		`+`
		`+ VOLUME ["/data"]`
		`+`
		`+ COPY entrypoint-pagure-ev.sh /entrypoint.sh`
		`+`
		`+ ENTRYPOINT ["/entrypoint.sh"]`

files/containerfiles/pagure-server

file added

+12

		`@@ -0,0 +1,12 @@`
		`+ FROM pagure-base:latest`
		`+`
		`+ LABEL org.opencontainers.image.authors="pagure community"`
		`+ LABEL org.opencontainers.image.url="https://pagure.io/pagure"`
		`+ LABEL org.opencontainers.image.source="https://pagure.io/pagure"`
		`+ LABEL org.opencontainers.image.documentation="https://docs.pagure.org/pagure/index.html"`
		`+`
		`+ VOLUME ["/data"]`
		`+`
		`+ COPY entrypoint-pagure-server.sh /entrypoint.sh`
		`+`
		`+ ENTRYPOINT ["/entrypoint.sh"]`

files/containerfiles/pagure-worker

file added

+15

		`@@ -0,0 +1,15 @@`
		`+ FROM pagure-base:latest`
		`+`
		`+ LABEL org.opencontainers.image.authors="pagure community"`
		`+ LABEL org.opencontainers.image.url="https://pagure.io/pagure"`
		`+ LABEL org.opencontainers.image.source="https://pagure.io/pagure"`
		`+ LABEL org.opencontainers.image.documentation="https://docs.pagure.org/pagure/index.html"`
		`+`
		`+ VOLUME ["/data"]`
		`+`
		`+ COPY entrypoint-pagure-worker.sh /entrypoint.sh`
		`+`
		`+ # Run celery as root, required to access /repos`
		`+ ENV C_FORCE_ROOT=true`
		`+`
		`+ ENTRYPOINT ["/entrypoint.sh"]`

files/containerfiles/pagure.cfg

file added

+10

		`@@ -0,0 +1,10 @@`
		`+ ENABLE_DOCS = True`
		`+ SECRET_KEY = 'ReplaceMeWithASecretPassphrase'`
		`+ DB_URL = 'postgresql://pagure:pagure@postgresql/pagure'`
		`+ APP_URL = 'http://pagure:8000/'`
		`+ REDIS_HOST = 'redis'`
		`+ GIT_FOLDER = '/data/repos'`
		`+ REMOTE_GIT_FOLDER = '/data/remote'`
		`+ ATTACHMENTS_FOLDER = '/data/attachments'`
		`+ EVENTSOURCE_SOURCE = 'http://pagure-ev:8080'`
		`+ PAGURE_CI_SERVICES = ['jenkins']`
		`\ No newline at end of file`

wombelix commented 9 months ago

Work in progress / under development

container

Currently there are no pre-build pagure container images.
When the work on this PR is completed, the next step is to regularly build images and publish them under the pagure org on quay.io.

Why new container files? The current files are located under dev/ and tailored to the requirements to run local unit / integration tests.

pagure-base

FROM quay.io/almalinuxorg/9-base
- At this moment pagure works best with pyton 3.9

pip based venv
- Gives us the most control about the module versions, necessary at this point, the goal is to switch to rpm at a later point

dev tools and caches
- installed dev tools (e.g.) are removed at the end of the RUN step. The dnf and pip cache is cleaned as well to reduce image size.

pagure.cfg and alembic.ini
- The base container ships a default version which is supposed to be replaced with a customized version during deployment. On k8s for example by mounting a ConfigMap.

pagure-server

The entrypoint script contains logic to trigger either a createdb or upgrade action via alembic. Gunicorn is then used to start a pagure application instance.

pagure-worker

The entrypoint script activates the python venv and executes runworker.py. Default values to work as task handler for standard pagure tasks. Parameters can be customized to run another work instance for logcom

pagure-docs

Behavior similar to pagure-server, gunicorn starts and instance of the pagure.docs_server application instead.

pagure-ev

Entrypoint activates the venv and then executes pagure-ev/pagure_stream_server.py.

kubernetes

The first iteration is a deployment manifest that spins up a working pagure instance on a k8s cluster.
In future this should then probably become a more majure Helm Chart that's available via quay.io next to pagure container images.