PR#321: Execute pungi-config-validate and allow setting schema_overrides. - odcs

odcs

#321 Execute pungi-config-validate and allow setting schema_overrides.

Merged 4 years ago by lsedlar. Opened 4 years ago by jkaluza.

jkaluza/odcs config-validate into master

Fix typos in raw_config doc.

Jan Kaluza • 4 years ago

f074add

Execute pungi-config-validate and allow setting schema_overrides.

Jan Kaluza • 4 years ago

7b8b327

docs/index.rst

file modified

		`@@ -17,6 +17,7 @@`
		`about`
		`client`
		`api`
		`+ raw_config`

		`Indices and tables`
		`==================`

docs/raw_config.rst

file added

+153

		`@@ -0,0 +1,153 @@`
		`+ ===================`
		`+ Raw config composes`
		`+ ===================`
		`+`
		`+ ODCS can also be used to generate composes defined by the Pungi configuration`
		`+ file stored in an external git repository. There is special compose source type`
		+ called ``raw_config`` which is used for these composes.
		`+`
		`+ This document describes how to configure ODCS server to generate`
		+ ``raw_config`` composes.
		`+`
		`+`
		`+ High level overview`
		`+ ===================`
		`+`
		`+`
		`+ Raw config composes are submitted by the ODCS client in the`
		+ ``raw_config_name#commit_hash`` format. For example
		+ ``fedora-30-nightly#master``.
		`+`
		+ The ODCS server uses ``raw_config_name`` (in our case ``fedora-30-nightly``)
		+ to find matching record in the ``raw_config_urls`` option stored in its main
		`+ configuration file.`
		`+`
		+ The ``raw_config_urls`` defines the git repository, branch and config file name.
		`+ This is used to fetch the Pungi configuration and ensures that Raw config`
		`+ configuration can be stored only on trusted git repositories.`
		`+`
		+ The ``raw_config_urls`` option can also define extra options influencing
		+ the generated compose (for example ``schema_override``).
		`+`
		+ ODCS then fetches the git repository associated with the ``raw_config`` and
		+ wraps the main Pungi configuration file using the ``raw_config_wrapper.conf``
		`+ configuration file. This configuration file defines deployment related Pungi`
		`+ options like Koji profile or for example runroot methods. This ensures`
		`+ these options cannot be changed by the ODCS user.`
		`+`
		`+ Before executing Pungi, final Pungi configuration file is validated using the`
		+ ``pungi-config-validate`` tool. It is possible to make the validation more
		`+ strict by overriding default JSON schema used by Pungi. This can be done on`
		+ global level using the ``raw_config_schema_override`` ODCS configuration
		+ option or per ``raw_config_name`` using the ``schema_override`` option in the
		+ ``raw_config_urls``. It is also possible to combine these two options.
		`+`
		`+ After the validation is done, ODCS executes Pungi and generates the compose.`
		`+`
		`+ Symlinks pointing to the generated compose are stored in the directories`
		+ named according to compose type - for example ``nightly``, ``ci`, ``test``
		+ or ``production``.
		`+`
		`+`
		`+ ODCS server configuration`
		`+ =========================`
		`+`
		`+`
		+ Preparing the ``raw_config_wrapper.conf``
		`+ -----------------------------------------`
		`+`
		+ The ``/etc/odcs/raw_config_wrapper.conf`` is simple Pungi configuration
		+ script. It must always start with ``from raw_config import *`` to include
		+ the real ``raw_config`` configuration file.
		`+`
		+ Any further options are then used to override the real ``raw_config``
		`+ configuration. For example to set particular Koji profile:`
		`+`
		`+`
		`+ .. sourcecode:: none`
		`+`
		`+ from raw_config import *`
		`+ koji_profile = 'odcs_stg'`
		`+`
		`+`
		+ Adding the ``raw_config_urls`` record
		`+ -------------------------------------`
		`+`
		+ The ``raw_config_urls`` in the ODCS configuration is a dict with the name
		+ of ``raw_config`` as a key and another dict as value. The dict used as value
		+ defines the ``raw_config`` and can have following keys:
		`+`
		+ - ``url`` - URL to git repository. It is later passed to ``git clone``.
		+ - ``commit`` [optional] - If set, it is not possible to specify commit using
		`+ the ODCS client and only commit hash (or branch name) defined by this`
		`+ option will be used.`
		+ - ``path`` [optional] - If set, defines the path within the git repository
		`+ where the configuration files are stored. This is useful in case when`
		`+ there are multiple independent Pungi configuration trees stored in`
		`+ the single git repository.`
		+ - ``config_filename`` - Name of the main Pungi configuration file.
		+ - ``schema_override`` [optional] - If set, defines the path to JSON schema
		`+ override file to be used when validating the main Pungi configuration file.`
		`+`
		`+`
		+ Enabling ``pungi-config-validate``
		`+ -------------------------------------`
		`+`
		+ By default, the ``pungi-config-validate`` script is not executed for
		+ ``raw_config`` composes. It is however recommended to enable it, otherwise
		+ it is not possible to use the ``schema_override`` options.
		`+`
		+ To enable it, set the ``pungi_config_validate`` ODCS option to
		+ ``"pungi-config-validate"`` (Or to full path to the pungi-config-validate
		`+ script).`
		`+`
		`+`
		+ Preparing the ``schema_override`` JSON file
		`+ -------------------------------------------`
		`+`
		`+ Raw Pungi configuration files can be used to execute any command on the ODCS`
		`+ backend which might be a security issue in case the people editing the`
		`+ configuration files cannot be trusted. It is also for example possible to`
		`+ generate composes with external files coming from untrusted repositories.`
		`+`
		`+ It is therefore possible to handle cases like this using the extended JSON`
		`+ schema validation which will allow only certain values for certain options.`
		`+`
		+ This is possible by creating global ``schema_override.json`` file and setting
		+ it using the ``raw_config_schema_override`` ODCS option. It is also possible
		+ to specify this for each ``raw_config`` using the ``schema_override`` option
		+ in the ``raw_config_urls`` ODCS option.
		`+`
		+ The ``schema_override.json`` format is the same as the one used by Pungi
		`+ for the default JSON schema. The default schema can be obtained by running`
		+ ``pungi-config-validate --dump-schema``.
		`+`
		+ The ``schema_override.json`` is merged with this default JSON schema and
		+ overrides its values. For example, to allow only ``koji`` ``pkgset_source``,
		+ the ``schema_override.json`` would look like this:
		`+`
		`+ .. sourcecode:: none`
		`+`
		`+ {`
		`+ "properties": {`
		`+ "pkgset_source": {`
		`+ "enum": ["koji"]`
		`+ }`
		`+ }`
		`+ }`
		`+`
		`+`
		+ Allowing users/groups to generate ``raw_config`` composes
		`+ ---------------------------------------------------------`
		`+`
		+ This is done by setting the ``raw_config`` source_type in
		+ the ``allowed_clients`` ODCS option like this:
		`+`
		`+`
		`+ .. sourcecode:: none`
		`+`
		`+ allowed_clients = {`
		`+ "some_username": {"source_types": ["raw_config", ...]}`
		`+ }`
		`+`
		`+`

server/odcs/server/config.py

file modified

+12

		`@@ -122,6 +122,11 @@`
		`'type': str,`
		`'default': "pungi-koji",`
		`'desc': 'Name or full-path to pungi-koji binary.'},`
		`+ 'pungi_config_validate': {`
		`+ 'type': str,`
		`+ 'default': "",`
		`+ 'desc': 'Name or full-path to pungi-config-validate binary. '`
		`+ 'If set to empty string, no validation is done.'},`
		`'pungi_timeout': {`
		`'type': int,`
		`'default': 3600,`
		`@@ -313,6 +318,13 @@`
		`'file. This file holds Pungi configuration which should '`
		`'import real pungi configuration from raw_config.conf '`
		`'in order to override some variables.'},`
		`+ 'raw_config_schema_override': {`
		`+ 'type': str,`
		`+ 'default': "",`
		`+ 'desc': 'Full path to the JSON file defining Pungi config schema '`
		`+ 'override. This file is passed to "pungi-config-validate" '`
		`+ 'using the --schema-override option and can influence '`
		`+ 'the raw_config validation.'},`
		`'pungi_koji_args': {`
		`'type': list,`
		`'default': [],`

server/odcs/server/pungi.py

file modified

+31

		`@@ -58,6 +58,9 @@`
		`"""Write configuration into files"""`
		`raise NotImplementedError('Concrete config object must implement.')`

		`+ def validate(self, topdir, compose_dir):`
		`+ """Validate configuration. Raises an exception of error found."""`
		`+ pass`

		`class RawPungiConfig(BasePungiConfig):`

		`@@ -108,6 +111,33 @@`
		`if config_path != main_cfg_path:`
		`shutil.copy2(config_path, main_cfg_path)`

		`+ def validate(self, topdir, compose_dir):`
		`+ if not conf.pungi_config_validate:`
		`+ return`
		`+ pungi_config_validate_cmd = [conf.pungi_config_validate, "--old-composes"]`
		`+`
		`+ # Apply global schema override.`
		`+ if conf.raw_config_schema_override:`
		`+ pungi_config_validate_cmd += [`
		`+ "--schema-override", conf.raw_config_schema_override]`
		`+`
		`+ # Apply raw_config specific schema override.`
		`+ if "schema_override" in self.pungi_cfg:`
		`+ pungi_config_validate_cmd += [`
		`+ "--schema-override", self.pungi_cfg["schema_override"]]`
		`+`
		`+ # Add raw_config configuration file to validate.`
		`+ pungi_config_validate_cmd.append(os.path.join(topdir, "pungi.conf"))`
		`+`
		`+ # Run the pungi-config-validate. The execute_cmd raises an exception`
		`+ # if config is invalid.`
		`+ log_out_path = os.path.join(compose_dir, "pungi-config-validate-stdout.log")`
		`+ log_err_path = os.path.join(compose_dir, "pungi-config-validate-stderr.log")`
		`+ with open(log_out_path, "w") as log_out:`
		`+ with open(log_err_path, "w") as log_err:`
		`+ odcs.server.utils.execute_cmd(`
		`+ pungi_config_validate_cmd, stdout=log_out, stderr=log_err)`
		`+`

		`class PungiConfig(BasePungiConfig):`
		`def __init__(self, release_name, release_version, source_type, source,`
		`@@ -395,6 +425,7 @@`
		`td = tempfile.mkdtemp()`
		`self._write_cfgs(td)`
		`compose_dir = self._prepare_compose_dir(compose, td, conf.target_dir)`
		`+ self.pungi_cfg.validate(td, compose_dir)`
		`pungi_cmd = self.get_pungi_cmd(td, conf.target_dir, compose_dir)`

		# Commit the session to ensure that all the `compose` changes are

server/tests/test_pungi.py

file modified

+24 -1

		`@@ -26,7 +26,7 @@`
		`import unittest`
		`import time`

		`- from mock import patch, MagicMock, mock_open`
		`+ from mock import patch, MagicMock, mock_open, call`
		`from kobo.conf import PyConfigParser`

		`from odcs.server.pungi import (`
		`@@ -491,6 +491,29 @@`
		`'http://localhost/test.git', AnyStringWith("/raw_config_repo"),`
		`commit='hash')`

		`+ @patch("odcs.server.utils.execute_cmd")`
		`+ def test_raw_config_validate(self, execute_cmd):`
		`+ fake_raw_config_urls = {`
		`+ 'pungi.conf': {`
		`+ "url": "http://localhost/test.git",`
		`+ "config_filename": "pungi.conf",`
		`+ "schema_override": "/etc/odcs/extra_override.json"`
		`+ }`
		`+ }`
		`+ with patch.object(conf, 'raw_config_schema_override', new="/etc/odcs/default_override.json"):`
		`+ with patch.object(conf, 'raw_config_urls', new=fake_raw_config_urls):`
		`+ with patch.object(conf, 'pungi_config_validate', new="pungi-config-validate"):`
		`+ pungi = Pungi(1, RawPungiConfig('pungi.conf#hash'))`
		`+ pungi.run(self.compose)`
		`+`
		`+ self.assertEqual(execute_cmd.mock_calls[0], call(`
		`+ ['pungi-config-validate', '--old-composes',`
		`+ '--schema-override', '/etc/odcs/default_override.json',`
		`+ '--schema-override', '/etc/odcs/extra_override.json',`
		`+ AnyStringWith('pungi.conf')],`
		`+ stderr=AnyStringWith("pungi-config-validate-stderr.log"),`
		`+ stdout=AnyStringWith("pungi-config-validate-stdout.log")))`
		`+`

		`class TestPungiLogs(ModelsBaseTest):`

jkaluza commented 4 years ago

This commit also adds some documentation for raw_configs.

jkaluza commented 4 years ago

This needs https://pagure.io/pungi/pull-request/1341.

lsedlar commented on line 124 of docs/raw_config.rst 4 years ago

This link would break sooner or later. If the goal is to show what the schema looks like, maybe mention it can be shown by running pungi-config-validate --dump-schema?

lsedlar commented 4 years ago

The pungi dependency is not strict, right? It is only required when there are some schema overrides defined.

There are a few typos, please pull in
https://pagure.io/fork/lsedlar/odcs/c/83c8d53e99455006d1afb409177ed4f85e62b864?branch=typo-fixes

jkaluza commented on line 124 of docs/raw_config.rst 4 years ago

Ah, that's what --dump-schema does. That's really nice. I will update this doc.

jkaluza commented 4 years ago

@lsedlar, if you set conf.pungi_config_validate, then the pungi-config-validate will be executed even if there is no schema-override defined. I'm not sure if it's useful or not. Maybe we can execute it only if schema-override is defined in ODCS config for this particular raw_config. What do you think?

jkaluza commented 4 years ago

It is executed only for raw_config composes.

lsedlar commented 4 years ago

I think it's fine the way it is. Running the validation without any override is only a small slowdown, since the config is validated when starting the compose as well.

My main point was to clarify that ODCS can be updated independently of Pungi. Only when server configuration changes will there be some dependency.

1 new commit added