#4815 Upgrade the Docker-based dev environment
Closed 3 years ago by sudoman. Opened 3 years ago by sudoman.
sudoman/pagure docker-improvements-0  into  master

file modified
+6 -16
@@ -47,30 +47,20 @@ 

  

  For more information about docker-compose cli, see: https://docs.docker.com/compose/reference/.

  

- Once installed, create the folder that will receive the projects, forks, docs,

- requests and tickets' git repo. Run this command exactly as it appears from 

- within the cloned git repo folder::

+ To build and run the containers, use the following command::

  

-     $ mkdir -p lcl/{repos,remotes,attachments,releases}

- 

- A docker compose environment is available to run pagure. First use the following

- command to build the containers. ::

- 

-     $ docker-compose -f dev/docker-compose.yml build

- 

- Once all the containers are built, run the following command to start the containers. ::

- 

-     $ docker-compose -f dev/docker-compose.yml up

+     $ ./dev/docker-start.sh

  

  Once all the containers have started, you can access pagure on http://localhost:5000.

  To stop the containers, press Ctrl+C.

  

- Once the containers are up and running after running the previous command, run 

- this command to populate the container with test data and create a new account ::

+ Once the containers are up and running, run this command to populate the

+ container with test data and create a new account ::

  

-     $ docker-compose -f dev/docker-compose.yml exec web python dev-data.py --all

+     $ docker-compose -f dev/docker-compose.yml exec web python3 dev-data.py --all

  

  You can then login with any of the created users, by example:

+ 

  - username: pingou

  - password: testing123

  

file added
+29
@@ -0,0 +1,29 @@ 

+ FROM registry.fedoraproject.org/fedora:30

Can we please use fedora:32? It should be available from registry.fedoraproject.org...

+ LABEL maintainers="Patrick Uiterwijk <patrick@puiterwijk.org>, Andrew Engelbrecht <andrew@engelbrecht.io>"

+ 

+ # using distro packages instead of pip, for a stable base image

+ 

+ RUN dnf -y update && \

+     dnf -y install findutils gcc git libgit2-devel python3-alembic \

+                    python3-arrow python3-bcrypt python3-beautifulsoup4 \

+                    python3-binaryornot python3-black python3-bleach \

+                    python3-blinker python3-celery python3-chardet \

+                    python3-coverage python3-cryptography python3-devel \

+                    python3-docutils python3-eventlet python3-fedmsg \

+                    python3-fedora python3-fedora-flask python3-filelock \

+                    python3-flake8 python3-flask python3-flask-oidc \

+                    python3-flask-wtf python3-funcsigs python3-jinja2 \

+                    python3-kitchen python3-markdown python3-mock \

+                    python3-munch python3-nose python3-openid \

+                    python3-openid-cla python3-openid-teams python3-pillow \

+                    python3-psutil python3-psycopg2 python3-pygit2 \

+                    python3-redis python3-requests python3-setuptools \

+                    python3-six python3-sqlalchemy python3-straight-plugin \

+                    python3-trololio python3-wtforms which && \

+     dnf clean all

+ 

+ ### not in Fedora yet

+ RUN pip3 install nosexcover

This is present in Fedora 31+ as python3-nose-xcover.

+ 

+ RUN ln -s /usr/bin/python3 /usr/bin/python

This will not be necessary in Fedora 31+, just install python-unversioned-command, which adds /usr/bin/python pointing to Python 3.

+ 

file modified
+5 -14
@@ -1,21 +1,12 @@ 

- FROM registry.fedoraproject.org/fedora:28

- MAINTAINER Patrick Uiterwijk <patrick@puiterwijk.org>

+ FROM pagure-base:latest

+ LABEL maintainers="Patrick Uiterwijk <patrick@puiterwijk.org>, Andrew Engelbrecht <andrew@engelbrecht.io>"

  

  VOLUME ["/repos"]

- RUN mkdir /code

- 

- RUN dnf install -y python2-devel python-setuptools python-nose py-bcrypt python-alembic \

-                    python-arrow python-binaryornot python-bleach python-blinker \

-                    python-chardet python-cryptography python-docutils python-flask \

-                    python-flask-wtf python-markdown python-psutil \

-                    python-pygit2 python-fedora python-openid python-openid-cla \

-                    python-openid-teams python-straight-plugin python-wtforms python-munch \

-                    python-enum34 python-redis python-sqlalchemy systemd gitolite3 python-filelock \

-                    python-fedora-flask python2-pillow python2-psycopg2 python-trololio \

-                    python-celery

  

+ RUN mkdir /code

  WORKDIR /code

- ENTRYPOINT ["/usr/bin/python", "/code/pagure-ev/pagure_stream_server.py"]

+ 

+ ENTRYPOINT ["/usr/bin/python3", "/code/pagure-ev/pagure_stream_server.py"]

  

  # Code injection is last to make optimal use of caches

  VOLUME ["/code"]

file modified
+4 -16
@@ -1,23 +1,11 @@ 

- FROM registry.fedoraproject.org/fedora:28

- MAINTAINER Patrick Uiterwijk <patrick@puiterwijk.org>

+ FROM pagure-base:latest

+ LABEL maintainers="Patrick Uiterwijk <patrick@puiterwijk.org>, Andrew Engelbrecht <andrew@engelbrecht.io>"

  

  VOLUME ["/repos"]

- RUN mkdir /code

- 

- RUN dnf install -y python3-devel python3-setuptools python3-nose python3-bcrypt python3-alembic \

-                    python3-arrow python3-binaryornot python3-bleach python3-blinker \

-                    python3-chardet python3-cryptography python3-docutils python3-flask \

-                    python3-flask-wtf python3-markdown python3-psutil \

-                    python3-pygit2 python3-fedora python3-openid python3-openid-cla \

-                    python3-openid-teams python3-straight-plugin python3-wtforms python3-munch \

-                    python3-enum34 python3-redis python3-sqlalchemy systemd gitolite3 \

-                    python3-filelock \

-                    python3-fedora-flask python3-pillow python3-psycopg2 python3-requests \

-                    python3-kitchen

- 

- RUN dnf install -y python3-celery

  

+ RUN mkdir /code

  WORKDIR /code

+ 

  ENTRYPOINT ["/usr/bin/celery-3", "-A", "pagure.lib.tasks_services", "worker", "--loglevel", "info", "-Q", "pagure_logcom"]

  

  # Code injection is last to make optimal use of caches

file modified
+4 -12
@@ -1,23 +1,15 @@ 

- FROM registry.fedoraproject.org/fedora:28

- MAINTAINER Patrick Uiterwijk <patrick@puiterwijk.org>

+ FROM pagure-base:latest

+ LABEL maintainers="Patrick Uiterwijk <patrick@puiterwijk.org>, Andrew Engelbrecht <andrew@engelbrecht.io>"

  

  VOLUME ["/repos"]

  RUN mkdir /code

+ WORKDIR /code

  

- RUN dnf install -y python2-devel python-setuptools python-nose python2-bcrypt python-alembic \

-                    python-arrow python-binaryornot python-bleach python-blinker \

-                    python-chardet python-cryptography python-docutils python-flask \

-                    python-flask-wtf python-markdown python-psutil \

-                    python-pygit2 python-fedora python-openid python-openid-cla \

-                    python-openid-teams python-straight-plugin python-wtforms python-munch \

-                    python-enum34 python-redis python-sqlalchemy systemd gitolite3 python-filelock \

-                    python-fedora-flask python2-pillow python2-psycopg2 python2-celery \

-                    findutils

  COPY web-run /run.sh

  

- WORKDIR /code

  # Openshift: --no-debug

  ENTRYPOINT ["/usr/bin/bash", "/run.sh"]

+ 

  EXPOSE 5000

  

  # Code injection is last to make optimal use of caches

file modified
+6 -5
@@ -1,14 +1,15 @@ 

  #!/bin/bash -xe

- pip install -r requirements-testing.txt

- python setup.py build

+ 

+ python3 setup.py build

+ 

  if [ ! -f /attachments/inited ];

  then

      echo "Giving Postgres time to start"

      sleep 10

      touch /attachments/inited

-     PAGURE_CONFIG=/code/dev/openshift.cfg python createdb.py --initial /code/dev/openshift_alembic.ini

+     PAGURE_CONFIG=/code/dev/openshift.cfg python3 createdb.py --initial /code/dev/openshift_alembic.ini

  else

-     alembic --config /code/dev/openshift_alembic.ini upgrade head

+     alembic-3 --config /code/dev/openshift_alembic.ini upgrade head

  fi

  

- exec /usr/bin/python /code/runserver.py --host 0.0.0.0 --config /code/dev/openshift.cfg

+ exec /usr/bin/python3 /code/runserver.py --host 0.0.0.0 --config /code/dev/openshift.cfg

file modified
+4 -18
@@ -1,25 +1,11 @@ 

- FROM registry.fedoraproject.org/fedora:28

- MAINTAINER Patrick Uiterwijk <patrick@puiterwijk.org>

+ FROM pagure-base:latest

+ LABEL maintainers="Patrick Uiterwijk <patrick@puiterwijk.org>, Andrew Engelbrecht <andrew@engelbrecht.io>"

  

  VOLUME ["/repos"]

- RUN mkdir /code

- 

- RUN dnf install -y python3-devel python3-setuptools python3-nose python3-bcrypt python3-alembic \

-                    python3-arrow python3-binaryornot python3-bleach python3-blinker \

-                    python3-chardet python3-cryptography python3-docutils python3-flask \

-                    python3-flask-wtf python3-markdown python3-psutil \

-                    python3-pygit2 python3-fedora python3-openid python3-openid-cla \

-                    python3-openid-teams python3-straight-plugin python3-wtforms python3-munch \

-                    python3-enum34 python3-redis python3-sqlalchemy systemd gitolite3 \

-                    python3-filelock python3-bleach python3-cryptography \

-                    python3-fedora-flask python3-pillow python3-psycopg2 python3-requests \

-                    python3-blinker

- 

- RUN dnf install -y python3-celery

- 

- RUN ln -sf /usr/bin/python3 /usr/bin/python

  

+ RUN mkdir /code

  WORKDIR /code

+ 

  ENTRYPOINT ["/usr/bin/celery-3", "-A", "pagure.lib.tasks", "worker", "--loglevel", "info"]

  

  # Code injection is last to make optimal use of caches

file added
+14
@@ -0,0 +1,14 @@ 

+ #! /bin/bash

+ 

+ set -e

+ 

+ [[ -f dev/docker-compose.yml ]] || (echo "please run this command from the root of the pagure git repo" && exit 1)

+ 

+ mkdir -p lcl/{repos,remotes,attachments,releases}

+ 

+ docker image build -f dev/containers/base -t pagure-base .

+ 

+ docker-compose -f dev/docker-compose.yml build

+ 

+ docker-compose -f dev/docker-compose.yml up

+ 

The containers now all inherit from a common Fedora 30 image, with only Python 3 packages installed. This saves disk space, and creates a consistent testing environment.

When possible, the base Dockerfile installs Python packages from the distro repository instead of PyPi. This is creates a stable installation that isn't dependent on the latest version of any package, so it should continue to work with less adjustment in the future.

Both the old and new Docker images produce a lot of failures when running the test suite from within. In the latter case, it's because redis-server is not installed on the dev_web_1 container (it's currently shipped in the redis container). Until the runtests.py script is fixed to handle a network socket, a workaround is possible by installing the redis package on dev_web_1 before running the tests. After doing that, most of the tests pass.

I've also updated the documentation about how to build / use the Docker images.

I plan to continue improving the Docker setup for Pagure with more commits.

Can we please use fedora:32? It should be available from registry.fedoraproject.org...

This is present in Fedora 31+ as python3-nose-xcover.

This will not be necessary in Fedora 31+, just install python-unversioned-command, which adds /usr/bin/python pointing to Python 3.

@sudoman Thank you for this work, it looks great! I just had some specific feedback which I've left above. :smile:

Thanks for the great feedback. : )

I created a branch with the suggested changes, with Fedora 32 as the source image:

https://pagure.io/fork/sudoman/pagure/tree/docker-improvements-1

It turns out that Fedora 30 also supports those two packages, so I created another branch with your changes, but for fc30:

https://pagure.io/fork/sudoman/pagure/tree/docker-improvements-2

While using new software is great, I think there's an advantage to using older software for testing. If these Docker images are used for testing code by developers, then by using the latest release of Fedora means that incompatibility with older packages could go unnoticed for a while, and required dependencies would need to be at higher versions than are available on some distros.

I like to think that this is similar to the idea of testing code on old and slow systems that also have spotty, low bandwidth networking, and little memory. This ensures that the code will run on just about anywhere.

That being said, it's true that Fedora 30 won't be supported much longer, so there are pros and cons. The nice thing about it is that updating the Docker image only requires changing one digit. : )

@sudoman python-unversioned-command is python2 in Fedora 30, so a good compromise would be to use Fedora 31, then. :)

I think I like the f31 middle ground.

Our CI tests on CentOS7 with py2 so it should prevent us from merging broken code for older system.

Should we go with F31?

I also like the idea of going with Fedora 31 for now. I'm fine with upgrading to Fedora 32 at any point given that the CI is also testing with Python2. If someone needs a newer dependency that isn't in their distro, they can still use pip.

I opened a new pull request here:

https://pagure.io/pagure/pull-request/4818

Pull-Request has been closed by sudoman

3 years ago