#2179 frontend: new command 'copr-frontend chroots-template'
Merged 3 months ago by praiskup. Opened 4 months ago by praiskup.
copr/ praiskup/copr templated-dictionary  into  main

@@ -19,6 +19,7 @@ 

      copr-frontend alter-chroot --action deactivate <name>

      copr-frontend branch-fedora <new-branched-version>

      copr-frontend rawhide-to-release <rawhide-chroot> <newly-created-chroot>

+     copr-frontend chroots-template [--template PATH]

  

  However, `enablement process upon Fedora branching <#branching-process>`_ and also

  `chroot deactivation when Fedora reaches it's EOL phase <#eol-deactivation-process>`_, are not that simple.
@@ -108,6 +109,48 @@ 

  See the :ref:`the disable chroots template <disable_chroots_template>`.

  

  

+ .. _managing_chroot_comments:

+ 

+ Managing chroot comments

+ ------------------------

+ 

+ Some of the available Mock chroots deserve a special care in documentation, e.g.

+ that `epel-8-*` chroots are nowadays built against Red Hat Enterprise Linux 8,

+ not CentOS 8 (which is EOL).  There's an administrator command for this::

+ 

+     $ copr-frontend comment-chroot --chroot epel-8-x86_64 --comment '<strong>Built against RHEL 8!</strong>'

+ 

+ Note that HTML text is supported!

+ 

+ This was though a single-chroot command.  There's a better option for those Copr

+ instances that contain dozens of chroots::

+ 

+     $ copr-frontend chroots-template [--template /etc/copr/chroots.conf]

+ 

+ This file reads the template file in the following format (a Python file

+ defining the ``config`` dictionary)::

+ 

+     config = {}

+     config["emulated"] = "This is an emulated chroot"

+     config["rules"] = [{

+         "match": "fedora-rawhide-i386",

+         "comment": "This is soon to be removed",

+     }, {

+         "match": ["fedora-32", "fedora-33"],

+         "comment": "<strong>Currently EOL</strong>, on your own risk",

+     },

+     {

+         "match": ["aarch64", "ppc64le"],

+         "match_type", "arch",

+         "comment_append": "{{ emulated }}",

+     }]

+ 

+ When (manually) executed, the command recursively iterates across all the active

+ Mock chroots, and applies the specified rules (only ``comment`` or

+ ``comment_append`` currently) when the chroot matches the rules (``match`` and

+ ``match_type`` statements).

+ 

+ 

  Mailing lists

  -------------

  

@@ -85,6 +85,7 @@ 

  BuildRequires: python3-dateutil

  BuildRequires: python3-decorator

  BuildRequires: python3-flask

+ BuildRequires: python3-templated-dictionary

  %if 0%{?fedora} >= 31

  BuildRequires: python3-flask-caching

  %else
@@ -171,6 +172,7 @@ 

  Requires: python3-pytz

  Requires: python3-redis

  Requires: python3-requests

+ Requires: python3-templated-dictionary

  Requires: python3-wtforms >= 2.2.1

  Requires: python3-zmq

  Requires: xstatic-bootstrap-scss-common
@@ -352,6 +354,7 @@ 

  %config(noreplace)  %{_sysconfdir}/copr/copr.conf

  %config(noreplace)  %{_sysconfdir}/copr/copr_devel.conf

  %config(noreplace)  %{_sysconfdir}/copr/copr_unit_test.conf

+ %config(noreplace)  %{_sysconfdir}/copr/chroots.conf

  

  %defattr(-, root, root, -)

  %config %{_sysconfdir}/cron.hourly/copr-frontend

@@ -0,0 +1,80 @@ 

+ """

+ The 'copr-frontend chroots-template' command implementation.

+ """

+ 

+ import importlib

+ import click

+ from templated_dictionary import TemplatedDictionary

+ 

+ from coprs import db_session_scope

+ from coprs.logic import coprs_logic

+ 

+ 

+ def apply_rule(chroot, rule):

+     """

+     The rule is to be applied to this chroot.

+     """

+     already_modified = getattr(chroot, "_already_modified", False)

+     setattr(chroot, "_already_modified", True)

+     if "comment" in rule:

+         chroot.comment = rule["comment"]

The "comment" always re-sets the value.

+         return

+     if "comment_append" in rule:

+         if not already_modified:

+             chroot.comment = ""

Comment append also resets the value, if it is specified as the first rule. Both "comment" and "comment_append" set the already_modified value.

+         chroot.comment += rule["comment_append"]

Will this append, and append, and append, on each run?
I think we want something like

if not chroot.comment.endswith(rule["comment_append"]):
    chroot.comment += rule["comment_append"]

It is not re-appending. That's the purpose of the already_modified flag...

I still have doubts.
For sure, the _already_modified will make sure the comment will not be appended twice within one execution of copr-frontend chroots-template. But when you run the script three times, I'd bet it gets appended each time. Or not?

I think it is fine, take a look above (chroot.comment is always reset before it is appended, for each of the command runs).

+         return

+     raise click.ClickException("Unknown rule type.")

+ 

+ 

+ def matches(string, rule):

+     """

+     Helper to accept both string and list() values in match argument.

+     """

+     rule = rule["match"]

+     if isinstance(rule, list):

+         return string in rule

+     return string == rule

+ 

+ 

+ def apply_rules(chroot, config):

+     """

+     Iterate over the rules from the config file, and attempt to apply them.

+     """

+     for rule in config["rules"]:

+         match_type = rule.get("match_type", "name")

+         if match_type == "name":

+             if matches(chroot.name, rule) or matches(chroot.name_release, rule):

+                 apply_rule(chroot, rule)

+             continue

+         if match_type == "arch" and matches(chroot.arch, rule):

+             apply_rule(chroot, rule)

+             continue

+ 

+ 

+ @click.command()

+ @click.option(

+     "-t", "--template",

+     help="Batch-configure the enabled chroots from a template file.",

+     type=click.Path(exists=True, readable=True),

+     default="/etc/copr/chroots.conf",

+ )

+ def chroots_template(template):

+     """

+     Load the MockChroot configuration from /etc/copr/chroots.conf.  For more

+     info take a look at

+     https://docs.pagure.org/copr.copr/how_to_manage_chroots.html#managing-chroot-comments

+     """

+ 

+     # Meh, this used to be just 'imp.load_source()' (but 'imp' is deprecated).

+     loader = importlib.machinery.SourceFileLoader("config", template)

+     spec = importlib.util.spec_from_loader(loader.name, loader)

+     module = importlib.util.module_from_spec(spec)

+     spec.loader.exec_module(module)

+ 

+     obj = getattr(module, "config")

+     config = TemplatedDictionary(obj)

+     config["__jinja_expand"] = True

+     with db_session_scope():

+         for ch in coprs_logic.MockChrootsLogic.get_multiple(active_only=True):

+             apply_rules(ch, config)

@@ -0,0 +1,35 @@ 

+ # Configuration for the 'copr-frontend chroots-template'.  That command is to be

+ # executed manually by the Copr admin (e.g. via Ansible).

+ 

+ config = {}

+ config["rules"] = []

+ 

+ # Possible options:

+ # config = {

+ #         # Supports the recursive Jinja2 templating.

+ #         "emulated": 'An emulated chroot',

+ #         "stream": 'For now built against <a href="https://www.centos.org/centos-stream/">CentOS Stream</a>, will be moved against RHEL once available.',

+ #         "released": 'Built against a GA software',

+ # }

+ #

+ # config["rules"] = [{

+ #         # Reset comments for aarch64 chroots

+ #         "match": "aarch64",

+ #         "match_type": "arch",

+ #         "comment": "",

+ # }, {

+ #         # Set a comment for a single chroot

+ #         "match": "epel-9",

+ #         "comment": '{{ stream }}',

+ # }, {

+ #         # Multiple chroots matched

+ #         "match": ["epel-8", "epel-9"],

+ #         "comment": "Built against RHEL",

+ # } {

+ #         # Multiple architectures matched, and comment _appended_

+ #         "match_type": "arch",

+ #         "match": ["s390x", "ppc64le", "armhfp"],

+ #         "comment_append": "{{ emulated }}",

+ # }]

+ #

+ # vim: ft=python

@@ -37,6 +37,7 @@ 

  import commands.clean_old_builds

  import commands.delete_orphans

  import commands.fixup_unnoticed_chroots

+ import commands.chroots_template

  

  from coprs import app

  
@@ -63,6 +64,7 @@ 

      "branch_fedora",

      "comment_chroot",

      "fixup_unnoticed_chroots",

+     "chroots_template",

  

      # User commands

      "alter_user",

We have dozens of chroots, and it has been tedious task to configure the
comments for them manually (copy-pasting the long comments, maintaining
links, etc.). The new command automatizes this by applying rules from
a new /etc/copr/chroots.conf configuration file.

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

rebased onto 8f7f39ecbdab359c221cd8d78208b794e597f816

4 months ago

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

rebased onto 9cd855fd0fc356d560962fa7b9302895d814994c

4 months ago

Build succeeded.

rebased onto 11130d8

4 months ago

Good point, updated. PTAL

Build succeeded.

Will this append, and append, and append, on each run?
I think we want something like

if not chroot.comment.endswith(rule["comment_append"]):
    chroot.comment += rule["comment_append"]

It is not re-appending. That's the purpose of the already_modified flag...

I still have doubts.
For sure, the _already_modified will make sure the comment will not be appended twice within one execution of copr-frontend chroots-template. But when you run the script three times, I'd bet it gets appended each time. Or not?

Comment append also resets the value, if it is specified as the first rule. Both "comment" and "comment_append" set the already_modified value.

I think it is fine, take a look above (chroot.comment is always reset before it is appended, for each of the command runs).

Commit 2e35ab0 fixes this pull-request

Pull-Request has been merged by praiskup

3 months ago