#801 doc: how_to_upgrade_builders with aarch64 as well
Merged 4 years ago by msuchy. Opened 4 years ago by praiskup.

file modified
+7 -6
@@ -45,13 +45,14 @@

  * determine if it is properly checking the timeout from a dead instance

  * maybe dump out the PID of the worker that is running so we know which one to kill?

  

- (Done)

+ Done

+ ^^^^

  * have jobs as static records in redis, drop workers for each build and only have process checking build status and downloading results in the end

-   - advantage is that you don't lose the already done work on builds that were running when copr-be is restarted (which happens on upgrades or a component failure)

-   - disadvantage is that it includes quite some changes: keeping and maintaining jobs and their states + process with infinite loop that checkes status of the running jobs

-   - you also need to run the mockchain job on background with stdin and stoud disattached so that it does not halt on SIGHUP

-   - includes implementation of a nice way to check whether build is finished or not (e.g. check running processes for mockbuilder user could be ok)

-   - looks like all of this requires some `copr-builder` script to be run on builder?  Then `copr-builder` and `copr-backend` can have pre-defined API

+     - advantage is that you don't lose the already done work on builds that were running when copr-be is restarted (which happens on upgrades or a component failure)

+     - disadvantage is that it includes quite some changes: keeping and maintaining jobs and their states + process with infinite loop that checkes status of the running jobs

+     - you also need to run the mockchain job on background with stdin and stoud disattached so that it does not halt on SIGHUP

+     - includes implementation of a nice way to check whether build is finished or not (e.g. check running processes for mockbuilder user could be ok)

+     - looks like all of this requires some `copr-builder` script to be run on builder?  Then `copr-builder` and `copr-backend` can have pre-defined API

When I use two-space indent here, I keep hearing the warning:
/home/praiskup/rh/projects/copr/copr/doc/brainstorming.rst:56: WARNING: Unexpected indentation.

          - $ copr-builder --config /some/config.conf build <coprID>/<package> --chroot <chroot>

          - the config.conf identifies where 'dist-git' and 'frontend' can be found to do `git clone PKG` and `copr-cli mock-config`

          - outputs could look like /copr-builder/live-log (stdout + stderr), /copr-builder/results/, /copr-builder/build.running (status)

file removed
-628
@@ -1,628 +0,0 @@

- Configuration

- =============

- 

- Pagure offers a wide varieties of options that must or can be used to

- adjust its behavior.

- 

- 

- 

- Must options

- ------------

- 

- Here are the options you must set up in order to get pagure running.

- 

- 

- SECRET_KEY

- ~~~~~~~~~~

- 

- This configuration key is used by flask to create the session. It should be kept secret

- and set as a long and random string.

- 

- 

- SALT_EMAIL

- ~~~~~~~~~~

- 

- This configuration key is used to ensure that when sending

- notifications to different users, each one of them has a different, unique

- and un-fakable ``Reply-To`` header. This header is then used by the milter to find

- out if the response received is a real one or a fake/invalid one.

- 

- 

- DB_URL

- ~~~~~~

- 

- This configuration key indicates to the framework how and where to connect to the database

- server. Pagure uses `SQLAchemy <http://www.sqlalchemy.org/>`_ to connect

- to a wide range of database server including MySQL, PostgreSQL, and SQLite.

- 

- Examples values:

- 

- ::

- 

-     DB_URL = 'mysql://user:pass@host/db_name'

-     DB_URL = 'postgres://user:pass@host/db_name'

-     DB_URL = 'sqlite:////var/tmp/pagure_dev.sqlite'

- 

- Defaults to ``sqlite:////var/tmp/pagure_dev.sqlite``

- 

- 

- APP_URL

- ~~~~~~~

- 

- This configuration key indicates the URL at which this pagure instance will be made available.

- 

- Defaults to: ``https://pagure.org/``

- 

- 

- EMAIL_ERROR

- ~~~~~~~~~~~

- 

- Pagure sends email when it catches an un-expected error (which saves you from

- having to monitor the logs regularly; but if you like, the error is still

- present in the logs).

- This configuration key allows you to specify to which email address to send

- these error reports.

- 

- 

- GIT_URL_SSH

- ~~~~~~~~~~~

- 

- This configuration key provides the information to the user on how to clone

- the git repos hosted on pagure via `SSH <https://en.wikipedia.org/wiki/Secure_Shell>`_.

- 

- The URL should end with a slash ``/``.

- 

- Defaults to: ``'ssh://git@pagure.org/'``

- 

- 

- GIT_URL_GIT

- ~~~~~~~~~~~

- This configuration key provides the information to the user on how to clone

- the git repos hosted on pagure anonymously. This access can be granted via

- the ``git://`` or ``http(s)://`` protocols.

- 

- The URL should end with a slash ``/``.

- 

- Defaults to: ``'git://pagure.org/'``

- 

- 

- 

- Repo Directories

- ----------------

- 

- Each project in pagure has 4 git repositories:

- 

- - the main repo for the code

- - the doc repo showed in the doc server

- - the ticket repo storing the metadata of the tickets

- - the request repo storing the metadata of the pull-requests

- 

- There are then another 3 folders: one for specifying the locations of the forks, one

- for the remote git repo used for the remotes pull-requests (ie: those coming from

- a project not hosted on this instance of pagure), and one for user-uploaded tarballs.

- 

- 

- GIT_FOLDER

- ~~~~~~~~~~

- 

- This configuration key points to the folder where the git repos for the

- source code of the projects are stored.

- 

- 

- DOCS_FOLDER

- ~~~~~~~~~~~

- 

- This configuration key points to the folder where the git repos for the

- documentation of the projects are stored.

- 

- 

- TICKETS_FOLDER

- ~~~~~~~~~~~~~~

- 

- This configuration key points to the folder where the git repos for the

- metadata of the tickets opened against the project are stored .

- 

- 

- REQUESTS_FOLDER

- ~~~~~~~~~~~~~~~

- 

- This configuration key points to the folder where the git repos for the

- metadata of the pull-requests opened against the project are stored.

- 

- 

- REMOTE_GIT_FOLDER

- ~~~~~~~~~~~~~~~~~

- 

- This configuration key points to the folder where the remote git repos (ie:

- not hosted on pagure) that someone used to open a pull-request against a

- project hosted on pagure are stored.

- 

- 

- UPLOAD_FOLDER_PATH

- ~~~~~~~~~~~~~~~~~~

- 

- This configuration key points to the folder where user-uploaded tarballs

- are stored and served from.

- 

- 

- UPLOAD_FOLDER_URL

- ~~~~~~~~~~~~~~~~~~

- 

- Full URL to where the uploads are available. It is highly recommanded for

- security reasons that this URL lives on a different domain than the main

- application (an entirely different domain, not just a sub-domain).

- 

- Defaults to: ``/releases/``, unsafe for production!

- 

- 

- .. warning:: both `UPLOAD_FOLDER_PATH` and `UPLOAD_FOLDER_URL` must be

-             specified for the upload release feature to work

- 

- 

- SESSION_COOKIE_SECURE

- ~~~~~~~~~~~~~~~~~~~~~

- 

- When this is set to True, the session cookie will only be returned to the

- server via ssl (https). If you connect to the server via plain http, the

- cookie will not be sent. This prevents sniffing of the cookie contents.

- This may be set to False when testing your application but should always

- be set to True in production.

- 

- Defaults to: ``False`` for development, must be ``True`` in production with

- https.

- 

- 

- FROM_EMAIL

- ~~~~~~~~~~

- 

- This configuration key specifies the email address used by this pagure instance

- when sending emails (notifications).

- 

- Defaults to: ``pagure@pagure.org``

- 

- 

- DOMAIN_EMAIL_NOTIFICATIONS

- ~~~~~~~~~~~~~~~~~~~~~~~~~~

- 

- This configuration key specifies the domain used by this pagure instance

- when sending emails (notifications). More precisely, it is used

- when building the ``msg-id`` header of the emails sent.

- 

- Defaults to: ``pagure.org``

- 

- 

- VIRUS_SCAN_ATTACHMENTS

- ~~~~~~~~~~~~~~~~~~~~~~

- 

- This configuration key configures whether attachments are scanned for viruses on

- upload. For more information, see the install.rst guide.

- 

- Defaults to: ``False``

- 

- 

- 

- Configure Gitolite

- ------------------

- 

- Pagure uses `gitolite <http://gitolite.com/>`_ as an authorization layer.

- Gitolite relies on `SSH <https://en.wikipedia.org/wiki/Secure_Shell>`_ for

- the authentication. In other words, SSH lets you in and gitolite checks if you

- are allowed to do what you are trying to do once you are inside.

- 

- 

- GITOLITE_HOME

- ~~~~~~~~~~~~~

- 

- This configuration key points to the home directory of the user under which

- gitolite is ran.

- 

- 

- GITOLITE_VERSION

- ~~~~~~~~~~~~~~~~

- 

- This configuration key specifies which version of gitolite you are

- using, it can be either ``2`` or ``3``.

- 

- Defaults to: ``3``.

- 

- 

- GITOLITE_KEYDIR

- ~~~~~~~~~~~~~~~

- 

- This configuration key points to the folder where gitolite stores and accesses

- the public SSH keys of all the user have access to the server.

- 

- Since pagure is the user interface, it is pagure that writes down the files

- in this directory, effectively setting up the users to be able to use gitolite.

- 

- 

- GITOLITE_CONFIG

- ~~~~~~~~~~~~~~~

- 

- This configuration key points to the gitolite.conf file where pagure writes

- the gitolite repository access configuration.

- 

- 

- GL_RC

- ~~~~~

- 

- This configuration key points to the file ``gitolite.rc`` used by gitolite

- to record who has access to what (ie: who has access to which repo/branch).

- 

- 

- GL_BINDIR

- ~~~~~~~~~

- 

- This configuration key indicates the folder in which the gitolite tools can

- be found. It can be as simple as ``/usr/bin/`` if the tools have been installed

- using a package manager or something like ``/opt/bin/`` for a more custom

- install.

- 

- 

- 

- EventSource options

- -------------------

- 

- EVENTSOURCE_SOURCE

- ~~~~~~~~~~~~~~~~~~

- 

- This configuration key indicates the URL at which the EventSource server is

- available. If not defined, pagure will behave as if there are no EventSource

- server running.

- 

- 

- EVENTSOURCE_PORT

- ~~~~~~~~~~~~~~~~

- 

- This configuration key indicates the port at which the EventSource server is

- running.

- 

- .. note:: The EventSource server requires a redis server (see ``Redis options``

-          below)

- 

- 

- 

- Web-hooks notifications

- -----------------------

- 

- WEBHOOK

- ~~~~~~~

- 

- This configuration key allows turning on or off web-hooks notifications for

- this pagure instance.

- 

- Defaults to: ``False``.

- 

- .. note:: The Web-hooks server requires a redis server (see ``Redis options``

-          below)

- 

- 

- 

- Redis options

- -------------

- 

- REDIS_HOST

- ~~~~~~~~~~

- 

- This configuration key indicates the host at which the `redis <http://redis.io/>`_

- server is running.

- 

- Defaults to: ``0.0.0.0``.

- 

- REDIS_PORT

- ~~~~~~~~~~

- 

- This configuration key indicates the port at which the redis server can be

- contacted.

- 

- Defaults to: ``6379``.

- 

- REDIS_DB

- ~~~~~~~~

- 

- This configuration key indicates the name of the redis database to use for

- communicating with the EventSource server.

- 

- Defaults to: ``0``.

- 

- 

- 

- Authentication options

- ----------------------

- 

- ADMIN_GROUP

- ~~~~~~~~~~~

- 

- List of groups, either local or remote (if the openid server used supports the

- group extension), that are the site admins. These admins can regenerate the

- gitolite configuration, the ssh key files, and the hook-token for every project

- as well as manage users and groups.

- 

- 

- PAGURE_ADMIN_USERS

- ~~~~~~~~~~~~~~~~~~

- 

- List of local users that are the site admins. These admins have the same rights as

- the users in the admin groups listed above as well as admin rights to

- all projects hosted on this pagure instance.

- 

- 

- 

- Optional options

- ----------------

- 

- SSH_KEYS

- ~~~~~~~~

- 

- It is a good practice to publish the fingerprint and public SSH key of a

- server you provide access to.

- Pagure offers the possibility to expose this information based on the values

- set in the configuration file, in the ``SSH_KEYS`` configuration key.

- 

- See the `SSH hostkeys/Fingerprints page on pagure.io <https://pagure.io/ssh_info>`_.

- 

- .. warning: The format is important

- 

-     SSH_KEYS = {'RSA': {'fingerprint': '<foo>', 'pubkey': '<bar>'}}

- 

- Where `<foo>` and `<bar>` must be replaced by your values.

- 

- 

- ITEM_PER_PAGE

- ~~~~~~~~~~~~~

- This configuration key allows you to configure the length of a page by

- setting the number of items on the page. Items can be commits, users, groups,

- or projects for example.

- 

- Defaults to: ``50``.

- 

- 

- SMTP_SERVER

- ~~~~~~~~~~~

- 

- This configuration key specifies the SMTP server to use when

- sending emails.

- 

- Defaults to: ``localhost``.

- 

- 

- SMTP_PORT

- ~~~~~~~~~

- 

- This configuration key specifies the SMTP server port.

- 

- SMTP by default uses TCP port 25. The protocol for mail submission is

- the same, but uses port 587.

- SMTP connections secured by SSL, known as SMTPS, default to port 465

- (nonstandard, but sometimes used for legacy reasons).

- 

- Defaults to: ``25``

- 

- 

- SMTP_SSL

- ~~~~~~~~

- 

- This configuration key specifies whether the SMTP connections

- should be secured over SSL.

- 

- Defaults to: ``False``

- 

- 

- SMTP_USERNAME

- ~~~~~~~~~~~~~

- 

- This configuration key allows usage of SMTP with auth.

- 

- Note: Specify SMTP_USERNAME and SMTP_PASSWORD for using SMTP auth

- 

- Defaults to: ``None``

- 

- 

- SMTP_PASSWORD

- ~~~~~~~~~~~~~

- 

- This configuration key allows usage of SMTP with auth.

- 

- Note: Specify SMTP_USERNAME and SMTP_PASSWORD for using SMTP auth

- 

- Defaults to: ``None``

- 

- SHORT_LENGTH

- ~~~~~~~~~~~~

- 

- This configuration key specifies the length of the commit ids or

- file hex displayed in the user interface.

- 

- Defaults to: ``6``.

- 

- 

- BLACKLISTED_PROJECTS

- ~~~~~~~~~~~~~~~~~~~~

- 

- This configuration key specifies a list of project names that are forbidden.

- This list is used for example to avoid conflicts at the URL level between the

- static files located under ``/static/`` and a project that would be named

- ``static`` and thus be located at ``/static``.

- 

- Defaults to:

- 

- ::

- 

-     [

-         'static', 'pv', 'releases', 'new', 'api', 'settings',

-         'logout', 'login', 'users', 'groups'

-     ]

- 

- 

- CHECK_SESSION_IP

- ~~~~~~~~~~~~~~~~

- 

- This configuration key specifies whether to check the user's IP

- address when retrieving its session. This makes things more secure but

- under certain setups it might not work (for example if there

- are proxies in front of the application).

- 

- Defaults to: ``True``.

- 

- 

- PAGURE_AUTH

- ~~~~~~~~~~~~

- 

- This configuration key specifies which authentication method to use.

- Pagure currently supports two authentication methods: one relying on the

- Fedora Account System `FAS <https://admin.fedoraproject.org/accounts>`_,

- and the other using only the local database.

- It can therefore be either ``fas`` or ``local``.

- 

- Defaults to: ``fas``.

- 

- 

- IP_ALLOWED_INTERNAL

- ~~~~~~~~~~~~~~~~~~~

- 

- This configuration key specifies which IP addresses are allowed

- to access the internal API endpoint. These endpoints are accessed by the

- milters for example and allow performing actions in the name of someone else

- which is sensitive, thus the origin of the request using

- these endpoints is validated.

- 

- Defaults to: ``['127.0.0.1', 'localhost', '::1']``.

- 

- 

- MAX_CONTENT_LENGTH

- ~~~~~~~~~~~~~~~~~~

- 

- This configuration key specifies the maximum file size allowed when

- uploading content to pagure (for example, screenshots to a ticket).

- 

- Defaults to: ``4 * 1024 * 1024`` which corresponds to 4 megabytes.

- 

- 

- ENABLE_TICKETS

- ~~~~~~~~~~~~~~

- 

- This configuration key activates or de-activates the ticketing system

- for all the projects hosted on this pagure instance.

- 

- Defaults to: ``True``

- 

- 

- ENABLE_NEW_PROJECTS

- ~~~~~~~~~~~~~~~~~~~

- 

- This configuration key permits or forbids creation of new projects via

- the user interface of this pagure instance.

- 

- Defaults to: ``True``

- 

- 

- ENABLE_DEL_PROJECTS

- ~~~~~~~~~~~~~~~~~~~

- 

- This configuration key permits or forbids deletiion of projects via

- the user interface of this pagure instance.

- 

- Defaults to: ``True``

- 

- 

- EMAIL_SEND

- ~~~~~~~~~~

- 

- This configuration key enables or disables all email notifications for

- this pagure instance. This can be useful to turn off when developing on

- pagure, or for test or pre-production instances.

- 

- Defaults to: ``False``.

- 

- .. note::

-     This does not disable emails to the email address set in ``EMAIL_ERROR``.

- 

- 

- OLD_VIEW_COMMIT_ENABLED

- ~~~~~~~~~~~~~~~~~~~~~~~

- 

- In version 1.3, pagure changed its URL scheme to view the commit of a

- project in order to add support for pseudo-namespaced projects.

- 

- For pagure instances older than 1.3, who care about backward compatibility,

- we added an endpoint ``view_commit_old`` that brings URL backward

- compatibility for URLs using the complete git hash (the 40 characters).

- For URLs using a shorter hash, the URLs will remain broken.

- 

- This configuration key enables or disables this backward compatibility

- which is useful for pagure instances running since before 1.3 but is not

- for newer instances.

- 

- Defaults to: ``False``.

- 

- 

- PAGURE_CI_SERVICES

- ~~~~~~~~~~~~~~~~~~

- 

- Pagure can be configure to integrate results of a Continuous Integration (CI)

- service to pull-requests open against a project.

- 

- To enable this integration, follow the documentation on how to install

- pagure-ci and set this configuration key to ``['jenkins']`` (Jenkins being

- the only CI service supported at the moment).

- 

- Defaults to: ``None``.

- 

- .. warning:: Requires `Redis` to be configured and running.

- 

- 

- INSTANCE_NAME

- ~~~~~~~~~~~~~

- 

- This allows giving a name to this running instance of pagure. The name is

- then used in the welcome screen showns upon first login.

- 

- Defaults to: ``Pagure``

- 

- .. note: the welcome screen currently does not work with the `local`

-          authentication.

- 

- 

- USER_NAMESPACE

- ~~~~~~~~~~~~~~

- 

- This configuration key allows to enforce that project are namespaced under

- the user's username, behaving in this way in a similar fashion as github.com

- or gitlab.com.

- 

- Defaults to: ``False``

- 

- 

- DOC_APP_URL

- ~~~~~~~~~~~

- 

- This configuration key allows you to specify where the documentation server

- is running (preferably in a different domain name entirely).

- If not set, the documentation page will show an error message saying that

- this pagure instance does not have a documentation server.

- 

- Defaults to: ``None``

- 

- 

- 

- Deprecated configuration keys

- -----------------------------

- 

- FORK_FOLDER

- ~~~~~~~~~~~

- 

- This configuration key used to be use to specify the folder where the forks

- are placed. Since the release 2.0 of pagure, it has been deprecated, forks

- are now automatically placed in a sub-folder of the folder containing the

- mains git repositories (ie ``GIT_FOLDER``).

- 

- See the ``UPGRADING.rst`` file for more information about this change and

- how to handle it.

- 

- 

- UPLOAD_FOLDER

- ~~~~~~~~~~~~~

- 

- This configuration key used to be use to specify where the uploaded releases

- are available. It has been replaced by `UPLOAD_FOLDER_PATH` in the release

- 2.10 of pagure.

@@ -1,3 +1,5 @@

+ :orphan:

+ 

  .. _copr_outdated_chroots_removal_policy:

  

  Outdated chroots removal policy

@@ -1,3 +1,5 @@

+ :orphan:

+ 

  .. _custom_source_method:

  

  Custom source method

file modified
+10 -13
@@ -16,19 +16,16 @@

  Working with the code

  ---------------------

  

- * :ref:`how_to_install` How to Install - Copr is intended as service. You most likely do not want to set up your own instance. But if you really want to, Copr set up is documented here.

- 

- * :ref:`contribute` How to Contribute - Do you want to help us? Start here.

- 

- * :ref:`git_guide` Git Guide -- how to work with Git, the source control of choice for Copr

- 

- * :ref:`patch_process` -- how to go about creating a new patch and submitting it to the mailing list

- 

- * :ref:`building_package` -- how to build Copr itself

- 

- * `Database schema <http://miroslav.suchy.cz/copr/coprdb/>`_

- 

- * `Documentation of python code <http://miroslav.suchy.cz/copr/python-doc/>`_ - most up-to-date documentation are in copr-frontend-doc and copr-backend-doc packages.

+ .. toctree::

+    :maxdepth: 1

+ 

+    How to install Copr as build service <how_to_install>

+    Contribution, do you want to help us? <contribute>

+    How to work with Git (the version control system for Copr project) <git_guide>

+    Do you prefer sending patches? <patch_process>

+    Building Copr packages (backend, frontend, ...) <building_package>

+    Database schema <http://miroslav.suchy.cz/copr/coprdb/>

+    Documentation of python code (or check copr-frontend-doc.rpm and copr-backend-doc.rpm) <http://miroslav.suchy.cz/copr/python-doc/>

  

  History

  -------

file modified
+1 -1
@@ -1,7 +1,7 @@

  .. _git_guide:

  

  `Git <http://git.or.cz/>`_ Guide

- ===============================

+ ================================

  

  Copr git web interface

  ----------------------

@@ -1,3 +1,5 @@

+ :orphan:

+ 

  .. _how_to_enable_repo:

  

  How to enable repo

file modified
+206 -41
@@ -3,7 +3,8 @@

  How to upgrade builders

  =======================

  

- This article explains how to upgrade the Copr builders in OpenStack to a newer Fedora once it is released.

+ This article explains how to upgrade the Copr builders in OpenStack (ppc64le,

+ x86_64) and libvirt (aarch64) to a newer Fedora once it is released.

  

  Keep amending this page if you find something not matching reality or expectations.

  
@@ -11,28 +12,47 @@

  Requirements

  ------------

  

- * ssh access to copr-be-dev.cloud.fedoraproject.org

- * an account on https://fedorainfracloud.org, and

- * OpenStack RC File for that ^^ account for `coprdev` tenant

+ * ssh access to `staging backend box`_

+ * an account on `Fedora Infra OpenStack`_, ..

+ * downloaded OpenStack RC File (go to OpenStack dasboard -- Access & Security --

+   API Access -- Download OpenStack RC File) ..

+ * and ``python3-openstackclient`` package installed

+ * ssh access to the main aarch64 hypervisor

+   ``copr@virthost-aarch64-os01.fedorainfracloud.org``

  

  

- Provide the image

- -----------------

+ Find source images

+ ------------------

  

- The first thing you need to figure out is what image should you use and where to get it. Luckily there is an image registry on https://fedorainfracloud.org/dashboard/project/images/. By default you see only the project images, to see all of them, click on the ``Public`` button.

+ The first thing you need to figure out is what image should you use and where to

+ get it.

  

- Search for the ``Fedora-Cloud-Base-*`` images of the particular Fedora. You need to have both x86_64 and ppc64le images. Are both of them available? Then you can jump right to the next step. Otherwise, you need to submit them.

+ The Cloud Base image for x86_64 can be obtained on `Fedora Cloud page`_.  Pick

+ the one with ``.qcow2`` extension.  The ppc64le and aarch64 images can be found

+ on the `Alternate Architectures page`_.  Don't confuse PPC64LE with PPC64.

  

- The Cloud Base image for x86_64 can be obtained on

- https://alt.fedoraproject.org/cloud.  Pick the one with ``.qcow2`` extension.

- The ppc64le image can be found on a page with alternative architectures

- https://alt.fedoraproject.org/alt. Don't mistake PPC64LE for PPC64.

  If neither that url provides the expected cloud image version (yet), there

- should exist at least the "compose" version in

- https://kojipkgs.fedoraproject.org/compose/cloud/, look for

- `latest-Fedora-Cloud-<VERSION>/compose/Cloud/ppc64le/images` directory.

+ should exist at least a "compose" version in `Koji compose directory listing`_,

+ look for ``latest-Fedora-Cloud-<VERSION>/compose/Cloud/<ARCH>/images``

+ directory.

  

- Download the image, and upload it to the `fedorainfracloud.org` OpenStack:

+ 

+ Prepare OpenStack source images

+ -------------------------------

+ 

+ (x86_64 and ppc64le architectures)

+ 

+ For OpenStack, there is an image registry on `OpenStack images dashboard`_.  By

+ default you see only the project images; to see all of them, click on the

+ ``Public`` button.

+ 

+ Search for the ``Fedora-Cloud-Base-*`` images of the particular Fedora.  Are

+ both x86_64 and ppc64le images available?  Then you can jump right to the next

+ section.

+ 

+ Download the image, and upload it to the infra OpenStack.  Be careful to keep

+ sane ``Fedora-Cloud-Base*`` naming, and to make it public, so others can later

+ use it as well:

  

  ::

  
@@ -52,11 +72,14 @@

          --property hw_rng_model=virtio \

          Fedora-Cloud-Base-30-1.2.x86_64

  

+ Note also the ``--property hw_rng_model=virtio`` option which guarantees that

+ the VMs won't wait indefinitely for random seed.

+ 

  

- Prepare the base image

- ----------------------

+ Prepare VM for snapshot

+ ^^^^^^^^^^^^^^^^^^^^^^^

  

- Open a ssh connection to copr-be-dev.cloud.fedoraproject.org and edit the

+ Open a ssh connection to ``copr-be-dev.cloud.fedoraproject.org`` and edit the

  ``/home/copr/provision/builderpb_nova.yml`` playbook (resp.

  ``/home/copr/provision/builderpb_nova_ppc64le.yml`` for ppc64le variant).  There

  is the following part
@@ -87,23 +110,27 @@

      su copr

      ansible-playbook /home/copr/provision/builderpb_nova.yml

  

- It will most likely fail because of missing python package on the host (i.e. in the image). It needs to be there because the Ansible modules are written in python. Continue to next section to see how to solve this problem.

+ It can likely fail (for various reasons, missing packages, changes in Fedora,

+ etc.).  Continue to next paragraph to see how to solve this problem.

  

- 

- Creating a snapshot

- -------------------

- 

- After running the ``builderpb_nova.yml`` playbook, you will get an IP address of a spawned builder. You can ssh into that builder, make changes and then create a snapshot (i.e. image) from that builder. To fix the previous issue with missing python package, run

+ After running the ``builderpb_nova.yml`` playbook, you will get an IP address of

+ a spawned builder.  You can ssh into that builder, make changes and then create

+ a snapshot (i.e. image) from that builder.  To fix the previous issue with

+ missing python package, run

  

  ::

  

      [copr@copr-be-dev ~][STG]$ ssh fedora@172.XX.XXX.XXX

      [fedora@172.XX.XXX.XXX ~]$ sudo dnf install python

  

- Open the https://fedorainfracloud.org again and go to the ``Instances``. Make

- sure that your "Current Project" is the project that you expect (``coprdev``).

- There are many instances so how can you be a hundred percent sure which one you

- modified? Use the IP address as an identifier.

+ 

+ Creating a snapshot

+ ^^^^^^^^^^^^^^^^^^^

+ 

+ Open the `OpenStack instances dashboard`_.  Make sure that your "Current

+ Project" is the project that you expect (``coprdev``).  There are many instances

+ so how can you be a hundred percent sure which one you modified?  Use the IP

+ address as an identifier.

  

  Optionally, click on ``More -> Shut Off Instance`` for that instance (sometimes

  it happens that OpenStack doesn't allow us to create snapshot from running
@@ -115,25 +142,27 @@

  

  In addition, make sure to make the snapshot Public, so we can use it also for production servers and Protected, so other people can't delete it.

  

- Edit the ``builderpb_nova.yml`` playbook as you did in the :ref:`previous section <image_name>` and set the new image name. Now run the playbook again

+ Configure also the snapshot image to use emulated "hardware" random generator

+ (otherwise with our OpenStack and new guest kernels the boot would take insanely

+ long on gathering entropy):

  

  ::

  

-     [copr@copr-be-dev ~][STG]$ ansible-playbook /home/copr/provision/builderpb_nova.yml

+     $ openstack image set --property hw_rng_model=virtio <THE_SNAPSHOT_UUID>

  

- Iterate this process until it ends successfully.

  

- Configure also the snapshot image to use emulated "hardware" random generator

- (otherwise with our OpenStack and new guest kernels the boot would take insanely

- long on gathering entropy):

+ Edit the ``builderpb_nova.yml`` playbook as you did in the :ref:`previous section <image_name>` and set the new image name. Now run the playbook again

  

  ::

  

-     $ openstack image set --property hw_rng_model=virtio <THE_SNAPSHOT_UUID>

+     [copr@copr-be-dev ~][STG]$ ansible-playbook /home/copr/provision/builderpb_nova.yml

+ 

+ Iterate this process until it ends successfully.

  

  

- Finishing up

- ------------

+ 

+ Finishing up OpenStack images

+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  

  Once you successfully provisioned a builder, you are almost done. First, create a snapshot of that builder.

  We learned how to do that in the previous section. Then set the :ref:`prepare_base_image <prepare_base_image>`
@@ -151,10 +180,138 @@

  Try to build some packages and you are done.

  

  

+ Prepare libvirt source images

+ -----------------------------

+ 

+ (aarch64 architecture only)

+ 

+ We can not prepare the image locally (on x86 laptops), so we have to create it

+ on some remote aarch64 box.  Since we have two aarch64 hypervisors for Copr now,

+ so we'll work with one of them.  But since the aarch64 hypervisors are

+ configured so they are using all the availalbe resources (namely disks), we have

+ to have some VMs shut down first to have some space (note the ``_dev`` keyword,

+ we are not deleting production builders in this step):

+ 

+ ::

+ 

+     $ ssh root@copr-be-dev.cloud.fedoraproject.org

+     # set 'aarch64_01_dev.max' option to 0 to disable spawner on hypervisor 1

+ 

+     [root@copr-be-dev ~][STG]# vim /etc/resallocserver/pools.yaml

+ 

+     # and terminate all already running resources there;  if there are some

+     # STARTING instances, please wait till they are not UP

+     [resalloc@copr-be-dev ~][STG]$ resalloc-maint resource-list | grep aarch64_01_dev

+     138 - aarch64_01_dev_00000138_20190613_051611 pool=aarch64_01_dev tags=aarch64 status=UP

+     140 - aarch64_01_dev_00000140_20190613_051613 pool=aarch64_01_dev tags=aarch64 status=UP

+ 

+     [resalloc@copr-be-dev ~][STG]$ resalloc-maint resource-delete 138 140

+ 

+     # check that all are deleted (no output)

+     [resalloc@copr-be-dev ~][STG]$ resalloc-maint resource-list | grep aarch64_01_dev

+ 

+ Now start the work on the aarch64 box:

+ 

+ ::

+ 

+     $ ssh copr@virthost-aarch64-os01.fedorainfracloud.org

+ 

+     # just in case you wanted to call /bin/virsh directly in this session

+     [copr@virthost-aarch64-os01 ~][PROD]$ export VIRSH_DEFAULT_CONNECT_URI=qemu:///system

+ 

+ Download the image, and prepare it for upload

+ 

+ ::

+ 

+     [copr@virthost-aarch64-os01 ~][PROD]$ wget --directory-prefix=/tmp \

+         https://mirrors.nic.cz/fedora/linux/releases/30/Cloud/aarch64/images/Fedora-Cloud-Base-30-1.2.aarch64.qcow2

+ 

+     [copr@virthost-aarch64-os01 ~][PROD]$ cd ~/vm-manage

+ 

+     # prepare the image, it takes ~15 minutes

+     [copr@virthost-aarch64-os01 ~][PROD]$ ./prepare-disk /tmp/Fedora-Cloud-Base-30-1.2.aarch64.qcow2

+     ...

+     + cp /tmp/Fedora-Cloud-Base-30-1.2.aarch64.qcow2 /tmp/newdisk.qcow2

+     ...

+ 

+ Upload the image to libvirt instances:

+ 

+ ::

+ 

+     [copr@virthost-aarch64-os01 vm-manage][PROD]$ ./upload-disk /tmp/newdisk.qcow2

+     ...

+     + virsh ... vol-upload copr-builder-20190614_123554 ... /tmp/newdisk.qcow2

+     ...

+     uploaded images copr-builder-20190614_123554

+ 

+ Test that the image spawns correctly:

+ 

+ ::

+ 

+     $ ssh root@copr-be-dev.cloud.fedoraproject.org

+     Last login: Fri Jun 14 12:16:48 2019 from 77.92.220.242

+ 

+     # temporarily use different image, set the option:

+     # img_volume = 'copr-builder-20190614_123554'

+     [root@copr-be-dev ~][PROD]# vim /var/lib/resallocserver/resalloc_provision/vm-aarch64-new

+ 

+     # re-enable spawner, set 'aarch64_01_dev.max' option to 2

+     [root@copr-be-dev ~][STG]# vim /etc/resallocserver/pools.yaml

+ 

+     # wait a minute for newly spawned VMs

+     [root@copr-be-dev ~][STG]# su - resalloc

+     Last login: Fri Jun 14 12:43:16 UTC 2019 on pts/0

+ 

+     [resalloc@copr-be-dev ~][STG]$ resalloc-maint resource-list

+     141 - aarch64_02_dev_00000141_20190613_051613 pool=aarch64_02_dev tags=aarch64 status=UP

+     139 - aarch64_02_dev_00000139_20190613_051611 pool=aarch64_02_dev tags=aarch64 status=UP

+     144 - aarch64_01_dev_00000144_20190614_124441 pool=aarch64_01_dev tags= status=STARTING

+     145 - aarch64_01_dev_00000145_20190614_124441 pool=aarch64_01_dev tags= status=STARTING

+ 

+     [resalloc@copr-be-dev ~][STG]$ tail -f /var/log/resallocserver/hooks/000145_alloc

+     ...

+     DEBUG:root: -> exit_status=0, time=233.029s

+     DEBUG:root:cleaning up workdir

+     38.145.48.106

+ 

+ 

+ If the log doesn't look good, you'll have to start over again (perhaps fix

+ spawner playbooks, or the ``prepare-disk`` script).  But if you see the VM IP

+ address, you are mostly done:

+ 

+ ::

+ 

+     [resalloc@copr-be-dev ~][STG]$ resalloc-maint resource-list | grep 00145

+     145 - aarch64_01_dev_00000145_20190614_124441 pool=aarch64_01_dev tags=aarch64 status=UP

+ 

+ Reset the temporary image back to ``img_volume=copr-builder``:

+ 

+ ::

+ 

+     [resalloc@copr-be-dev ~][STG]$ exit

+     [root@copr-be-dev ~][PROD]# vim /var/lib/resallocserver/resalloc_provision/vm-aarch64-new

+ 

+ Swap the image-names in hypervisors:

+ 

+ ::

+ 

+     $ ssh copr@virthost-aarch64-os01.fedorainfracloud.org

+     [copr@virthost-aarch64-os01 ~][PROD]$ cd ~/vm-manage

+ 

+     [copr@virthost-aarch64-os01 ~][PROD]$ ./promote-disk copr-builder-20190614_123554

+     ...

+     copr-builder == copr-builder-20190614_123554 now

+ 

+ Done.  You can still kill all VMs by ``resalloc-maint resource-delete ...``, but

+ if you are not in hurry - old VMs will slowly get terminated - and all the new

+ VMs will be started from the freshly updated ``copr-builder`` image.

+ 

+ 

  Production

  ----------

  

- There is a substantially less work for production instance. You just need to edit this playbook

+ There is a substantially less work for production instance. You just need to

+ edit this playbook (and only for x86_64 and ppc64le images)

  

  https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/copr/backend/files/provision/builderpb_nova.yml

  
@@ -162,9 +319,17 @@

  Then you need to commit the change and push it to the repository. If you don't have a write permission for it, then

  ask someone who does.

  

- Once the change is pushed, you need to re-provision the backend instance or ask someone to do it.

- 

+ Once the change is pushed, you need to re-provision the backend instance or ask someone to do it:

  

  ::

  

      rbac-playbook groups/copr-backend.yml -t provision_config

+ 

+ 

+ .. _`staging backend box`: https://copr-be-dev.cloud.fedoraproject.org

+ .. _`Fedora Infra OpenStack`: https://fedorainfracloud.org

+ .. _`Fedora Cloud page`: https://alt.fedoraproject.org/cloud

+ .. _`Alternate Architectures page`:  https://alt.fedoraproject.org/alt

+ .. _`Koji compose directory listing`: https://kojipkgs.fedoraproject.org/compose/cloud/

+ .. _`OpenStack images dashboard`: https://fedorainfracloud.org/dashboard/project/images/

+ .. _`OpenStack instances dashboard`: https://fedorainfracloud.org/dashboard/project/instances/

@@ -1,14 +1,18 @@

  .. _maintenance_documentation:

  

  Maintenance Documentation

- =======================

+ =========================

  

  This section contains information about maintenance topics. You may also be interested in :ref:`user_documentation` and :ref:`developer_documentation`.

  

- * :ref:`how_to_release_copr` -- how to release Copr

+ .. toctree::

+    :caption: Table of Contents

+    :maxdepth: 1

  

- * :ref:`how_to_upgrade_builders` -- how to upgrade openstack builders

- 

- * :ref:`how_to_manage_chroots` -- how to manage active chroots

- 

- * :ref:`how_to_delete_outdated_chroots` -- sending notifications and removing data from outdated chroots

+    How to release copr RPM packages <how_to_release_copr>

+    how_to_upgrade_builders

+    How to manage active chroots <how_to_manage_chroots>

+    Sending notifications and removing data from outdated chroots <how_to_delete_outdated_chroots>

+    sanity_tests

+    beaker_tests

+    seeddb

file removed
-18
@@ -1,18 +0,0 @@

- .. _quick_start:

- 

- Quick start

- ===========

- 

- So you'd like to send in a patch to Copr?  Great!

- 

- 

- First, get the code: :ref:`downloads`

- 

- 

- Then read the git instructions on the :ref:`git_guide`. You'll want to send in a patch via `git-format-patch1` to the Developer mailing list (see :ref:`communication`).

- 

- 

- As mentioned on the other page, patches should be applied to HEAD in git, and we'll take care of applying them to other releases as appropriate.

- 

- 

- If you send some (aprox. 10) patches which will have no issues, you may get direct write access to copr.git.

@@ -1,3 +1,5 @@

+ :orphan:

+ 

  .. _screenshots_tutorial:

  

  Screenshots tutorial

file modified
+2
@@ -1,3 +1,5 @@

+ :orphan:

+ 

  .. _SrcRpmUpload:

  

  Upload of src.rpm

file modified
+11 -7
@@ -26,6 +26,10 @@

  

  See :ref:`screenshots_tutorial` for interacting with `copr.fedoraproject.org <http://copr.fedoraproject.org/>`_

  

+ 

+ How to enable copr repository?

+ ------------------------------

+ 

  :ref:`how_to_enable_repo`

  

  Build Source Types
@@ -75,13 +79,13 @@

  **rpkg**: The default choice and the most versatile one. Apart from building packages from any Git or SVN repository,

  it also supports building directly from any `DistGit <https://clime.github.io/2017/05/20/DistGit-1.0.html>`_ repository.

  Note that **rpkg** (as well as **tito**) is not only a tool to generate SRPMs but, in fact, it is also a full-fledged package manager

- that you can use from your command-line to maintain your packages. You can read more about this tool `here <https://pagure.io/rpkg-util>`_.

+ that you can use from your command-line to maintain your packages. You can read more about this tool `here <https://pagure.io/rpkg-util>`__.

I've never heard of "anonymous links" before, so for the record, there is some documentation http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#anonymous-hyperlinks

  

  **tito**: is a robust RPM package manager with lots of features and if your project is managed with Tito, this is the tool you want to pick for SRPM generation (which is

  one of the many package manager's features). When this option is selected, the latest package GIT tag will be used to build an SRPM. Note that this utility has currently

  no support for specifying an alternative .spec file, which means the **Spec Field** is simply ignored when this option is used and .spec file will be always auto-located.

  Note that the basic difference between this tool and **rpkg** is that the target repository needs to be initialized with ``tito init`` first before this tool can be used

- to build SRPMs from it. You can read more `here <https://github.com/dgoodwin/tito>`_.

+ to build SRPMs from it. You can read more `here <https://github.com/dgoodwin/tito>`__.

  

  **tito test**: With this option selected, again the `tito <https://github.com/dgoodwin/tito>`_ utility will be used to build an SRPM but this time, the **Committish**

  value specified above (or HEAD of the master branch if no **Committish** is specified) will be used to build an SRPM. This corresponds to using ``--test`` switch for
@@ -204,11 +208,11 @@

  

  * For building package from git:

  

- 1) `Tito <https://github.com/dgoodwin/tito>`_ (`blog post <http://miroslav.suchy.cz/blog/archives/2013/12/29/how_to_build_in_copr/index.html>`_ and `another about Tito+Git annex <http://m0dlx.com/blog/Reproducible_builds_on_Copr_with_tito_and_git_annex.html>`_) 

+ 1) `Tito <https://github.com/dgoodwin/tito>`_ (`blog post <http://miroslav.suchy.cz/blog/archives/2013/12/29/how_to_build_in_copr/index.html>`__ and `another about Tito+Git annex <http://m0dlx.com/blog/Reproducible_builds_on_Copr_with_tito_and_git_annex.html>`_)

  

- 2) `dgroc <https://github.com/pypingou/dgroc>`_ (`blog post <http://blog.pingoured.fr/index.php?post/2014/03/20/Introducing-dgroc>`_)

+ 2) `dgroc <https://github.com/pypingou/dgroc>`_ (`blog post <http://blog.pingoured.fr/index.php?post/2014/03/20/Introducing-dgroc>`__)

  

- * `Jenkins plugin <https://wiki.jenkins-ci.org/display/JENKINS/Copr+Plugin>`_ (`blog post <http://michal-srb.blogspot.cz/2014/04/jenkins-plugin-for-building-rpms-in-copr.html>`_)

+ * `Jenkins plugin <https://wiki.jenkins-ci.org/display/JENKINS/Copr+Plugin>`_ (`blog post <http://michal-srb.blogspot.cz/2014/04/jenkins-plugin-for-building-rpms-in-copr.html>`__)

  

  Multilib

  --------
@@ -428,9 +432,9 @@

  

  If you want to know more about tools to generate srpm from a Git repo, see:

  

- 1) `Tito <https://github.com/dgoodwin/tito>`_ (`blog post <http://miroslav.suchy.cz/blog/archives/2013/12/29/how_to_build_in_copr/index.html>`_)

+ 1) `Tito <https://github.com/dgoodwin/tito>`_ (`blog post <http://miroslav.suchy.cz/blog/archives/2013/12/29/how_to_build_in_copr/index.html>`__)

  

- 2) `dgroc <https://github.com/pypingou/dgroc>`_ (`blog post <http://blog.pingoured.fr/index.php?post/2014/03/20/Introducing-dgroc>`_)

+ 2) `dgroc <https://github.com/pypingou/dgroc>`_ (`blog post <http://blog.pingoured.fr/index.php?post/2014/03/20/Introducing-dgroc>`__)

  

  .. _`How do I get notifications about finished builds?`:

  

file modified
+2
@@ -1,3 +1,5 @@

+ :orphan:

+ 

  .. _webhook_hacking:

  

  COPR auto-rebuilds with custom Git repositories

no initial comment

1 new commit added

  • doc: nicer toctree for maintenance topics
4 years ago

3 new commits added

  • doc: use anonymous links
  • doc: broken indent in brainstorming.rst
  • doc: fix header underlining
4 years ago

2 new commits added

  • doc: drop leftover docs files
  • doc: fix another set of toctree warnings
4 years ago

I've never heard of "anonymous links" before, so for the record, there is some documentation http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#anonymous-hyperlinks

The indentation is a bit off to me. Looks like all of the lists in this file are indented with two spaces, this sublist is indented by three spaces and the sublist under this particular point is indented by five spaces. Can you please correct them?

I kinda liked the "By default you see only the project images, to see all of them, click on the Public button." hint. I am sure, that I will be confused by it the next time I will upgrade builders :-D

Taking the comment back, it is much lower in the document.

I would note here, to avoid killing "RUNNING" or what instances, so we don't interrupt any running builds (duh, but better safe than sorry)

I would prefer this whole block of code separated by one empty line between its parts, e.g. having a blank line after command output and blank line above a commented command.
Same for the big chunk of code a bit lower. Since there is no syntax highlighting when it is rendered, it is just a huge blob of text and it is really hard to read (at least for me).

Is that a plus sign copied from some diff? :-)

Also, I am missing a prompt here

I am not sure, what this line suppose to mean

Also copy-pasted plus sign from some diff? Also, the ... should be explained. Personally, I don't use virsh on a daily basis (or ever, for that matter), so I don't know how to substitute them.

I am also not sure how to interpret this line

Too bad, that this is a one big PR instead of small several ones, but thanks to commits descriptions, it is quite review-able.

Very good job on getting rid of those compilation errors and also the maintenance documentation TOC tree.

I've made you some notes for the actual doc/how_to_upgrade_builders.rst document.

:-) it's actually habit/style to put double-space after periods ... I'd prefer
to keep that if you don't mind much (but I won't start dogma-fight here).

we are deleting _dev instances, but I'll not that there...

That + signs just result of bash -x, and are part of the script's output. I'm trying your different idea to separate commands by space ... and maybe that will be clearer then; or if you have better idea?

Again, this is just part of the script output; ideas?

rebased onto c3b9831ea5e61a8197be3062bf3defe5f4502a42

4 years ago

updated in the meantime (those parts I did know how to fix :-))

When I use two-space indent here, I keep hearing the warning:
/home/praiskup/rh/projects/copr/copr/doc/brainstorming.rst:56: WARNING: Unexpected indentation.

rebased onto 1768dde

4 years ago

:-) it's actually habit/style to put double-space after periods ... I'd prefer
to keep that if you don't mind much (but I won't start dogma-fight here).

I didn't even know that was a thing ... It seems that it has no effect on the rendered HTML, so the output looks consistent. Personally, I find two spaces after periods pointless, but I am not really against them. At the same time, I would not enforce them.

we are deleting _dev instances, but I'll not that there...

Ah, the note is assuring though.

That + signs just result of bash -x, and are part of the script's output. I'm trying your different idea to separate commands by space ... and maybe that will be clearer then; or if you have better idea?

Aha, okay. Well, first of all, the spacing IMHO really helped. Are those output + lines that important, that we need to highlight them in the command output here? If admin doesn't care about them and is fine with zero command status or e.g. "uploaded images copr-builder-20190614_123554" I would remove them.

When I use two-space indent here, I keep hearing the warning:
/home/praiskup/rh/projects/copr/copr/doc/brainstorming.rst:56: WARNING: Unexpected indentation

That's true and very strange to me. Well, just leave it as is.

Aha, okay. Well, first of all, the spacing IMHO really helped. Are those output + lines that important, that we need to highlight them in the command output here? If admin doesn't care about them and is fine with zero command status or e.g. "uploaded images copr-builder-20190614_123554" I would remove them.

I tried to keep only the important lines, e.g. admin should be 100% sure what the image name is, so he can later test that it is working (e.g. promote-disk).

Maybe I should simplify the steps so the output was not that much important? Or reformat the output a bit?

Pull-Request has been merged by msuchy

4 years ago