PR#252: docs: Combine dev setup docs, refactor organization - fedora-commops/fedora-happiness-packets

fedora-commops / fedora-happiness-packets

#252 docs: Combine dev setup docs, refactor organization

Merged 4 years ago by jflory7. Opened 4 years ago by jflory7.

change/refactor-devel-guide into master

docs: Combine dev setup docs, refactor organization

Justin W. Flory • 4 years ago

3aa2eb1

docs/setup/development.rst

file modified

+152 -70

		`@@ -1,114 +1,196 @@`
		`- =========================================`
		`- Setting up a development environment`
		`- =========================================`
		`+ #######################################`
		`+ How to set up a development environment`
		`+ #######################################`

		`- This guide assumes that you are using a Unix-like system e.g a (GNU/)Linux distribution or macOS.`
		`+ This guide includes instructions for Linux / macOS and Windows.`

		`- Using Docker`
		`- ===============`

		`- The project comes with a Dockerfile that allows easy deployment of a web server.`
		`+ **************`
		`+ Pre-requisites`
		`+ **************`

		- #. Install Docker (refer `this <https://docs.docker.com/install/>`_) and Docker Compose (refer `this <https://docs.docker.com/compose/install/>`_) on your machine.
		`+ You need the following programs installed before proceeding (on all operating systems).`

		`- For platform specific installation guidelines:`
		+ - `Git <https://git-scm.com/>`__
		+ - `Python 3.6.x <https://www.python.org/downloads/>`__

		- :macOS: `Docker Desktop <https://docs.docker.com/docker-for-mac/install/>`_ (Docker Compose is included as part of desktop installs.)
		- :Windows: `Docker Desktop <https://docs.docker.com/docker-for-windows/install/>`_ (Docker Compose is included as part of desktop installs.)
		- :CentOS: `Docker CE <https://docs.docker.com/install/linux/docker-ce/centos/>`_
		- :Debian: `Docker CE <https://docs.docker.com/install/linux/docker-ce/debian/>`_
		- :Fedora: `Fedora Developer Portal <https://developer.fedoraproject.org/tools/docker/docker-installation.html>`_
		- :Ubuntu: `Docker CE <https://docs.docker.com/install/linux/docker-ce/ubuntu/>`_

		- #. Fork the repository, then clone using ``ssh``. For steps to setup ``ssh``, refer `this <https://docs.pagure.org/pagure/usage/first_steps.html>`_ ::
		`+ *****************`
		`+ How to use Docker`
		`+ *****************`

		`- git clone "ssh://git@pagure.io/forks/<user_name>/fedora-commops/fedora-happiness-packets.git"`
		`+ No matter what operating system you use, you need Docker.`
		`+ Docker is a tool to run applications inside what are known as "containers".`
		`+ Containers are similar to lightweight virtual machines (VMs).`
		`+ They make it easier to develop, test, and deploy a web application.`
		`+ Fedora Happiness Packets uses Docker for local development and to deploy to the production website.`

		`- #. Change into the repository directory::`
		`+ The project comes with a Dockerfile.`
		`+ A Dockerfile is the instructions for a container build tool (like Docker) to build a container.`
		+ Look at the `Fedora Happiness Packets Dockerfile <https://pagure.io/fedora-commops/fedora-happiness-packets/blob/master/f/Dockerfile>`__ for an example.

		`- cd fedora-happiness-packets`
		`+ How to install Docker`
		`+ =====================`
		`+`
		+ Install Docker as described in the `installation docs <https://docs.docker.com/install/>`__.
		+ You also need to install `Docker Compose <https://docs.docker.com/compose/install/>`__.
		`+ Docker Compose is used to run multiple containers side-by-side.`
		`+ While developing Fedora Happiness Packets, there are a few different services that run in multiple containers, like the Django web app, the Postgres database, and more.`
		`+`
		`+ See below for platform specific installation guidelines:`
		`+`
		+ - `Docker Desktop for Mac <https://docs.docker.com/docker-for-mac/install/>`__ (Docker Compose included)
		+ - `Docker Desktop for Windows <https://docs.docker.com/docker-for-windows/install/>`__ (Docker Compose included)
		+ - `CentOS <https://docs.docker.com/install/linux/docker-ce/centos/>`__
		+ - `Debian <https://docs.docker.com/install/linux/docker-ce/debian/>`__
		+ - `Fedora <https://developer.fedoraproject.org/tools/docker/docker-installation.html>`__
		+ - `Ubuntu <https://docs.docker.com/install/linux/docker-ce/ubuntu/>`__
		`+`
		`+`
		`+ ******************`
		`+ Run initial set-up`
		`+ ******************`
		`+`
		`+ This section explains how to get started developing for the first time.`
		`+`
		`+ Fork and clone`
		`+ ==============`

		`- #. Run the client secret generation script::`
		`+ First, you need to fork the Fedora Happiness Packets repo on Pagure.io to your Pagure account.`
		`+ Then, clone your fork to your workstation using git.`
		+ For extra help, see the `Pagure first steps <https://docs.pagure.org/pagure/usage/first_steps.html>`__ help page.
		`+ Once you clone your fork, you need to run a script to generate authentication tokens (more on this later).`

		`+ ::`
		`+`
		`+ git clone "ssh://git@pagure.io/forks/<username>/fedora-commops/fedora-happiness-packets.git"`
		`+ cd fedora-happiness-packets`
		`./generate_client_secrets.sh`

		`- Although the Dockerfile runs the script to check if a client_secrets.json file is present, please generate it before starting the Docker container, so that client secrets are not being constantly generated every time the image is rebuilt.`
		`+ Although Docker runs this script during container build-time, please generate a local copy first.`
		`+ This way, new client keys are not being generated each time the container is rebuilt.`
		`+ This avoids rate-limiting by the authentication service.`

		`- #. Create a fas-admin-details.json file and add a json object with your FAS-Username and FAS-Password. See fas-admin-details.json.example.`
		`+ Add FAS account login info`
		`+ ==========================`

		- #. Create a config.yml file and populate it with the user's details and usernames for `ADMINS <https://docs.djangoproject.com/en/2.1/ref/settings/#admins>`_ and superuser privileges respectively.
		`+ Next, you need to configure Fedora Happiness Packets with your Fedora Account System (FAS) username and password.`
		`+ This is used to authenticate with Fedora APIs for username search.`
		+ Copy the example file ``fas-admin-details.json.example`` as a new file named ``fas-admin-details.json``.
		`+ Add your username and password into the quotes.`

		- In order to run the web server, alongside the Redis queue and celery worker instance, simply run ``docker-compose up``.
		- After this, you can access the local development server at ``http://localhost:8000/`` in your web browser.
		`+ Create a project config file`
		`+ ============================`

		`- Auto reload of code changes is configured, so you shouldn't have to rebuild containers every time you make changes to the code base.`
		- If you have made any changes to the environment of the containers (eg - adding a new dependency), the Docker image must be rebuilt; this can be done my running ``docker-compose up --build``.
		`+ Next, create a configuration file to add admin users to Fedora Happiness Packets.`
		+ Like before, copy ``config.yml.example`` to a new file named ``config.yml``.
		+ Add your name and ``@fedoraproject.org`` email address for `ADMINS <https://docs.djangoproject.com/en/2.1/ref/settings/#admins>`_.
		`+ For superuser privileges, add your FAS username to the list.`

		`- Since the code is being run in a container, we must enter the shell of the container in order to run tests.`
		- Access the shell of the Docker container by running ``docker-compose exec web sh``.
		`+ How to test sending emails`
		`+ ==========================`

		- The test suite can be run by running the ``t`` script, which runs the tests with the appropriate testing settings and provides a coverage report.
		- In order to run the script, simply type ``./t`` in the Docker container's shell.
		`+ In the development environment, sending emails is tested in one of two ways:`

		- Integration tests are run via the following command: ``docker-compose exec web ./manage.py test -v 2 -p integration_test*.py --settings=happinesspackets.settings.tsting``
		`+ * Using console`
		`+ * Using third-party mail provider (e.g. Gmail`

		`- Alternative methods to test sending email`
		`- -----------------------------------------`
		`+ Using console`
		`+ -------------`

		`- In the development environment sending emails is setup in two ways:`
		`+ The default setup is to send emails on the console.`
		+ The full content of emails will appear in the ``docker-compose`` console output (explained below).
		`+ To see this in action, no changes are needed.`

		- * The default setup is to send emails on the console. The settings for the same can be found under the comment ``Configurations for sending email on console``. To see this in action, no changes to the present code base is needed.
		`+ Using Gmail`
		`+ -----------`

		`- * Emails are also configured to send using Gmail SMTP server. To test this functionality:`
		`+ Sending real, actual emails can be tested with a third-party mail provider, like Gmail.`
		`+ There are other mail services you can use, but this guide explains using Gmail.`
		`+ To test this functionality:`

		- 1. In ``settings/dev.py`` un-comment the setting for ``Configurations to test sending emails using Gmail SMTP`` and comment out the setting under ``Configurations for sending email on console`` and in ``docker-compose.yml`` un-comment the ports setting in ``celery`` service.
		- 2. Enable `less secure apps <https://myaccount.google.com/lesssecureapps>`_ in the Gmail account which you want to use as the host email.
		`- (It is strongly recommended to not allow less secure apps in your primary Gmail account. A separate account for testing is recommended with this setting enabled.)`
		- 3. Replace ``<HOST@EMAIL.COM>`` and ``<HOST_EMAIL_PASSWORD>`` with the email address of the above account and its password.
		+ #. In ``settings/dev.py``, un-comment the setting for ``Configurations to test sending emails using Gmail SMTP``.
		+ Comment out the setting under ``Configurations for sending email on console``.
		+ In ``docker-compose.yml``, un-comment the ports setting in ``celery`` service.
		+ #. Enable `less secure apps <https://myaccount.google.com/lesssecureapps>`_ in the Gmail account which you want to use as the host email.
		`+ (It is strongly recommended to not allow less secure apps in your primary Gmail account.`
		`+ A separate account for testing is recommended with this setting enabled.)`
		+ #. Replace ``<HOST@EMAIL.COM>`` and ``<HOST_EMAIL_PASSWORD>`` with the email address of the above account and its password.

		`- Testing Fedora Messaging Integration`
		`- --------------------------------------`

		- To test if messages are being sent to the RabbitMQ broker, open a new terminal and run the following command inside the shell of the Docker container `web`::
		`+ **************************************************`
		`+ Start Fedora Happiness Packets with Docker Compose`
		`+ **************************************************`

		`- fedora-messaging consume --callback=fedora_messaging.example:printer`
		`+ Now you are ready to start Fedora Happiness Packets!`
		`+ You will use Docker Compose to start all the containers used at the same time (like Redis, Celery, and others).`
		`+ Run this command to start up the project::`

		`- The messages sent to the RabbitMQ broker, when a sender confirms sending a happiness packet, will be printed in this terminal.`
		`+ docker-compose up`

		`- Alternatives to Docker`
		`- ======================`
		+ Once it finishes starting up, open `localhost:8000 <http://localhost:8000/>`__ in your browser
		`+ You should see the Fedora Happiness Packets home page.`

		`- Should you be unable to run Docker, or prefer to not use it, there is an alternative way of setting up a development environment.`
		+ Thanks to `PR #235 <https://pagure.io/fedora-commops/fedora-happiness-packets/pull-request/235>`__ from `@ShraddhaAg <https://twitter.com/ShraddhaAg>`__, changes to Django code, HTML templates, and CSS/JavaScript are automatically reloaded while ``docker-compose`` is running.
		`+ You should not need to rebuild the containers every time you make a change.`
		`+ However, sometimes you will need to rebuild the containers (e.g. adding a new dependency).`
		`+ This can be done with the following command::`

		`- 0. (Optional) Use a virtual environment solution, like virtualenv or Pipenv, in order to prevent dependency conflicts with other projects.`
		- 1. Install the development packages: ``pip install -r /requirements/dev.txt`` (Pipenv users can use pipenv install)
		- 2. Export the development settings module as an environment variable: ``export DJANGO_SETTINGS_MODULE=happinesspackets.settings.dev``
		- 3. Generate the client_secrets.json file (This is needed in order for login functionality to work): ``./generate_client_secrets.sh``.
		- If you get a permission denied error, change the file to an executable: ``chmod +x generate_client_secrets.sh``
		- 4. Collect static resources: ``python manage.py collectstatic``
		- 5. Ensure the database is up to date by running all migrations: ``python manage.py migrate``
		`+ docker-compose up --build`

		`- In order to ensure the server is fully functional, Redis and Celery must both be configured.`
		`+ Run integration tests`
		`+ =====================`

		`- Redis`
		`- ------`
		`+ Integration tests are tests that ensure an application works fully from beginning to end.`
		`+ Fedora Happiness Packets is not fully tested, but there are some integration tests.`
		`+ To run integration tests, you need to enter the container while it is running and run the test suite.`
		`+ Open a new window and run this command to open a shell prompt inside the Django web app container::`

		`- 1. Install redis from your package manager, or follow the instructions on the Redis website.`
		- 2. Ensure that the redis server has been started: ``redis-server``
		`+ docker-compose exec web bash`

		`- Celery`
		`- _______`
		`+ Once inside the container, run this command::`

		- 1. Start the celery worker in the background, or in a separate terminal window: ``celery -A happinesspackets worker -l info``
		`+ docker-compose exec web ./manage.py test -v 2 -p integration_test*.py --settings=happinesspackets.settings.tsting`

		+ Test ``fedora-messaging`` integration
		`+ =====================================`

		- The tests can be run directly from the project folder as described in the Using Docker section, without having to run ``docker-compose exec web sh`` beforehand.
		+ To test if messages are being sent to the RabbitMQ broker, open a new terminal while ``docker-compose`` is running.
		`+ Run the following commands::`

		`- As a reference:`
		`+ docker-compose exec web bash`
		`+ fedora-messaging consume --callback=fedora_messaging.example:printer`

		- - Run tests with testing settings, and produce coverage report: ``./t``
		- - Run integration tests: ``./manage.py test -v 2 -p integration_test*.py --settings=happinesspackets.settings.tsting``
		`+ The messages sent to the RabbitMQ broker, when a sender confirms sending a happiness packet, will be printed in this terminal.`

		- The web server can thus be run via the ``manage.py`` script: ``python manage.py runserver 0.0.0.0:8000``

		`- When the source code is changed, the web server should automatically reload, and apply the new changes.`
		`+ **********************`
		`+ Alternatives to Docker`
		`+ **********************`
		`+`
		`+ There are other ways to run Fedora Happiness Packets without containers or Docker.`
		`+ However, this is discouraged as current maintainers use containers to test changes and deploy Fedora Happiness Packets to production.`
		`+ If you choose to not use Docker and set up everything manually, you may run into unexpected issues.`
		`+ Project maintainers cannot easily help you if you choose this route (and may not be able to help you)!`
		`+ Therefore, please consider very carefully if you wish to run everything locally without containers.`
		`+`
		`+`
		`+ ***************`
		`+ Troubleshooting`
		`+ ***************`
		`+`
		+ Windows: ``alpinelinux.org error ERROR: unsatisfiable constraints``
		`+ ===================================================================`
		`+`
		+ On Windows, you might get the above error when running ``docker-compose``.
		`+ This can be resolved by following these steps:`
		`+`
		`+ #. Open Docker settings.`
		`+ #. Click on Network.`
		`+ #. Look for "DNS Server" section.`
		`+ #. It is set to Automatic by default.`
		`+ Change it to Fixed.`
		`+ #. The IP address should now be editable.`
		+ Try changing it to ``1.1.1.1``.
		`+ #. Save settings.`
		`+ #. Restart Docker.`

docs/setup/development_windows.rst

file removed

-89

		`@@ -1,89 +0,0 @@`
		`- ===================================================`
		`- Setting up a development environment on Windows`
		`- ===================================================`
		`-`
		`- This guide will take you through the process to setup your development environment on Windows OS.`
		`-`
		`- Prerequisites`
		`- ===============`
		`-`
		`- You will need the following programs installed on your system before proceeding with the setup.`
		`-`
		- #. `Git <https://git-scm.com/>`_
		- #. `Python <https://www.python.org/downloads/release/python-2716/>`_ (version = 2.7)
		`-`
		`- .. note::`
		- We will be soon migrating to `Python <https://www.python.org/downloads/>`_ (version >=3.5) as Python 2.7 will reach end of support in 2020. More details can be found `here <https://pagure.io/fedora-commops/fedora-happiness-packets/issue/150>`_.
		`-`
		`-`
		- #. `Docker Desktop for Windows <https://hub.docker.com/editions/community/docker-ce-desktop-windows>`_ (Download, installation, and run instructions available)
		- #. Docker Compose (Docker Compose is included as part of Docker Desktop, but if you still need to download it, you can do it from `here <https://docs.docker.com/compose/install/>`_)
		`-`
		`-`
		`- Setting up the development environment`
		`- ========================================`
		`-`
		- #. Fork the `current repository <https://pagure.io/fedora-commops/fedora-happiness-packets>`_ to your profile.
		- #. Clone this forked repository (ssh method is recommended, the steps can be found `here <https://docs.pagure.org/pagure/usage/first_steps.html>`_) to your system using Git with the following command::
		`-`
		`- git clone "https://pagure.io/forks/<user_name>/fedora-commops/fedora-happiness-packets.git"`
		`-`
		`- OR, if using ssh,`
		`-`
		`- git clone "ssh://git@pagure.io/forks/<user_name>/fedora-commops/fedora-happiness-packets.git"`
		`-`
		`- #. Once cloned, move inside the directory using the command::`
		`-`
		`- cd fedora-happiness-packets`
		`-`
		`- #. Start the Docker application(With elevated/admin access).`
		`- #. Run the client secret generation script::`
		`-`
		`- generate_client_secrets.sh`
		`-`
		`- Here, you might get an error saying::`
		`-`
		`- /bin/sh: ./generate_client_secrets.sh: not found`
		`-`
		`- This occurs mostly due to line endings being CRLF instead or the required LF.`
		- This can be changed by modifying the settings of your respective editor, or else from console using `this method <https://github.com/postlight/headless-wp-starter/issues/171#issuecomment-451682572>`_
		`-`
		`- #. To run the web server, run::`
		`-`
		`- docker-compose up`
		`-`
		- Once executed successfully, you should be able to view the main page on `http://localhost:8000/ <http://localhost:8000/>`_
		`-`
		`- Congratulations, you have successfully setup the development environment on your system.`
		`-`
		`- After making changes to any file, you'll have to run the command::`
		`-`
		`- docker-compose up --build`
		`-`
		- For ``docker-compose up`` or ``docker-compose up --build`` you might get an error of::
		`-`
		`- alpinelinux.org error ERROR: unsatisfiable constraints`
		`-`
		`- This can be resolved by simply following the below steps`
		`-`
		`- #. Open Docker settings.`
		`- #. Click on Network.`
		`- #. Look for 'DNS Server' section.`
		`- #. It is set to Automatic by default, change it to fixed.`
		- #. The IP address should now be editable. Change it from ``8.8.8.8`` to ``8.8.4.4``.
		`- #. Save the settings and restart Docker.`
		`-`
		- In order to run tests, make sure to execute ``docker-compose up`` command. Now in a new terminal(for the same container) run::
		`-`
		`- docker-compose exec web sh`
		`-`
		- Then the terminal will show a ``#`` symbol.
		- Simply type in ``./t`` (or ``t``) to initiate the test suite.
		`-`
		- (The test suite are run by running the ``./t`` script, which runs the tests with the appropriate testing settings and provides a coverage report.)
		`-`
		`- Integration tests are run via the following command::`
		`-`
		`- docker-compose exec web ./manage.py test -v 2 -p integration_test*.py --settings=happinesspackets.settings.tsting`
		`-`
		- If a ``file not found`` error occurs, try removing the ``./`` from the command and try again.

jflory7 commented 4 years ago

This commit does two things:

Combines the two development set-up guide docs into one
Refactors the combined page for better organization

I thought it was confusing to have two pages, since they mostly repeated
the other pages. This makes it easier for us to maintain one and only
one source of truth for setting up and developing on Fedora Happiness
Packets.

I also refactored the combined pages and reorganized parts to make it
easier to read. It is a fairly long page, so I used some more
ReStructuredText / Sphinx magic to improve readability where I could.

This page also now conforms to most of the Sphinx docs style guide too:

https://documentation-style-guide-sphinx.readthedocs.io/

Some of these, like filenames, I intentionally ignore because I think
they are quite silly.

Metadata Update from @jflory7:
- Pull-request tagged with: PASSED, improvement, type - docs, type - internal organization

4 years ago

Metadata Update from @jflory7:
- Pull-request tagged with: type - summer coding

4 years ago

jflory7 commented 4 years ago

@shraddhaag Could you please review this docs PR some time after Monday? I did not add anything new, but I did change some explanations and re-arrange a lot of things. Mostly I want to make sure it is up-to-date on how things are done today and/or if anything could be better explained.

It is also good practice to test it out and make sure it builds too. Since it's been a while since we have worked with the docs, here are two ways to test the docs:

With Pipenv, preferred:

cd docs/
pipenv shell
pipenv install
./test-docs.sh

With regular virtualenv:

# start virtualenv with your preferred method
cd docs/
pip install -r requirements.txt
./test-docs.sh

Once you build the docs, you can open the docs/_build/index.html page in your web browser to see them rendered.

Metadata Update from @jflory7:
- Pull-request tagged with: needs testing
- Request assigned

4 years ago

shraddhaag commented 4 years ago

@jflory7 Hi! Thank you for getting these changes addressed. The docs definitely needed a revamp. I like the idea of getting all development docs in one place to make it more succinct. A special thank you for the Alternatives to Docker section in the setup docs, we were especially behind on that part.

The only thing I see changing in near future are the integration tests. Hopefully after @alishapapun is done with them, we can get that addressed in near future too! :cake:

These changes look great! I'm +1 on merge. :smile:

jflory7 commented 4 years ago

@shraddhaag Thanks for looking this over! We can definitely revisit the integration tests later on, I agree.

Merging! :ocean:

Metadata Update from @jflory7:
- Pull-request untagged with: needs testing

4 years ago

Pull-Request has been merged by jflory7

4 years ago