PR#337: Introduce new odcs.client.odcs.ODCS.request_compose method to make requests easier. - odcs

odcs

#337 Introduce new odcs.client.odcs.ODCS.request_compose method to make requests easier.

Merged 4 years ago by lsedlar. Opened 4 years ago by jkaluza.

jkaluza/odcs better-api into master

Introduce new odcs.client.odcs.ODCS.request_compose method to make requests easier.

Jan Kaluza • 4 years ago

1730395

client/contrib/odcs

file modified

+154 -12

		`@@ -36,7 +36,7 @@`

		`parser = argparse.ArgumentParser(`
		`description='''\`
		`- %(prog)s - Submit a test compose.`
		`+ %(prog)s - Command line client.`

		`If you have problems authenticating with OpenID Connect, try:`

		`@@ -44,10 +44,6 @@`

		`Example usage:`

		`- # Assume a Python virtual environment is activated`
		`- export PYTHONPATH=client`
		`- %(prog)s --staging --source f26 --source-type tag --flag no_deps \\`
		`- python-requests python-urllib3`
		`''',`
		`formatter_class=argparse.RawDescriptionHelpFormatter,`
		`)`
		`@@ -76,7 +72,7 @@`


		`create_parser = subparsers.add_parser(`
		`- 'create', help='create a new compose')`
		`+ 'create', help='Low-level command to create a new compose')`
		`create_parser.set_defaults(command='create')`
		`create_parser.add_argument(`
		`'source_type', default=None,`
		`@@ -131,6 +127,123 @@`
		`'--compose-type', default=None,`
		`help="Compose type for raw_config compose.")`

		`+ create_tag_parser = subparsers.add_parser(`
		`+ 'create-tag', help='Create new compose from Koji tag.')`
		`+ create_tag_parser.set_defaults(command='create-tag')`
		`+ create_tag_parser.add_argument(`
		`+ 'tag', default="",`
		`+ help="Koji tag name.")`
		`+ create_tag_parser.add_argument(`
		`+ '--flag', default=[], action='append',`
		`+ help="Flag to pass to influence the compose.")`
		`+ create_tag_parser.add_argument(`
		`+ '--sigkey', default=[], action='append',`
		`+ help="Signing key ID using which the packages must be signed - "`
		`+ "\none\" for no signing key.")`
		`+ create_tag_parser.add_argument(`
		`+ '--koji-event', default=None,`
		`+ help="Koji event for populating package set")`
		`+ create_tag_parser.add_argument(`
		`+ '--arch', default=[], action='append',`
		`+ help="Koji arch to build the compose for.")`
		`+ create_tag_parser.add_argument(`
		`+ 'packages', metavar='package', nargs='*',`
		`+ help='Koji packages to be included in the compose.')`
		`+ create_tag_parser.add_argument(`
		`+ '--build', default=[], action='append',`
		`+ help='Builds to be included in the compose.')`
		`+ create_tag_parser.add_argument(`
		`+ '--module-defaults-url', default="",`
		`+ metavar='module_defaults_url',`
		`+ help="URL to git repository with module defaults.")`
		`+ create_tag_parser.add_argument(`
		`+ '--module-defaults-commit', default="",`
		`+ metavar='module_defaults_commit',`
		`+ help="Git commit/branch from which to take the module defaults.")`
		`+ create_tag_parser.add_argument(`
		`+ '--modular-tag', default=[], action='append',`
		`+ metavar="modular_koji_tags",`
		`+ help="Koji tag with module builds.")`
		`+ create_tag_parser.add_argument(`
		`+ '--lookaside-repo', default=[], action='append',`
		`+ metavar="lookaside_repos",`
		`+ help="Koji tag with module builds.")`
		`+`
		`+ create_module_parser = subparsers.add_parser(`
		`+ 'create-module', help='Create new compose from modules.')`
		`+ create_module_parser.set_defaults(command='create-module')`
		`+ create_module_parser.add_argument(`
		`+ '--flag', default=[], action='append',`
		`+ help="Flag to pass to influence the compose.")`
		`+ create_module_parser.add_argument(`
		`+ '--sigkey', default=[], action='append',`
		`+ help="Signing key ID using which the packages must be signed - "`
		`+ "\none\" for no signing key.")`
		`+ create_module_parser.add_argument(`
		`+ '--arch', default=[], action='append',`
		`+ help="Koji arch to build the compose for.")`
		`+ create_module_parser.add_argument(`
		`+ 'modules', metavar='nsvc', nargs='+',`
		`+ help='Modules to be included in the compose.')`
		`+ create_module_parser.add_argument(`
		`+ '--module-defaults-url', default="",`
		`+ metavar='module_defaults_url',`
		`+ help="URL to git repository with module defaults.")`
		`+ create_module_parser.add_argument(`
		`+ '--module-defaults-commit', default="",`
		`+ metavar='module_defaults_commit',`
		`+ help="Git commit/branch from which to take the module defaults.")`
		`+ create_module_parser.add_argument(`
		`+ '--lookaside-repo', default=[], action='append',`
		`+ metavar="lookaside_repos",`
		`+ help="Koji tag with module builds.")`
		`+`
		`+ create_pulp_parser = subparsers.add_parser(`
		`+ 'create-pulp', help='Create new compose from Pulp content_sets.')`
		`+ create_pulp_parser.set_defaults(command='create-pulp')`
		`+ create_pulp_parser.add_argument(`
		`+ '--flag', default=[], action='append',`
		`+ help="Flag to pass to influence the compose.")`
		`+ create_pulp_parser.add_argument(`
		`+ 'content_sets', metavar='content_set', nargs='+',`
		`+ help='Content sets to be included in the compose.')`
		`+`
		`+ create_raw_config_parser = subparsers.add_parser(`
		`+ 'create-raw-config', help='Create new compose from Pungi raw configuration.')`
		`+ create_raw_config_parser.set_defaults(command='create-raw-config')`
		`+ create_raw_config_parser.add_argument(`
		`+ 'raw_config_name',`
		`+ help='Name of raw_config compose as defined in ODCS Server.')`
		`+ create_raw_config_parser.add_argument(`
		`+ 'raw_config_commit',`
		`+ help='Commit or branch name to get raw_config from.')`
		`+ create_raw_config_parser.add_argument(`
		`+ '--flag', default=[], action='append',`
		`+ help="Flag to pass to influence the compose.")`
		`+ create_raw_config_parser.add_argument(`
		`+ '--label', default=None,`
		`+ help="Label for raw_config compose.")`
		`+ create_raw_config_parser.add_argument(`
		`+ '--compose-type', default=None,`
		`+ help="Compose type for raw_config compose.")`
		`+ create_raw_config_parser.add_argument(`
		`+ '--koji-event', default=None,`
		`+ help="Koji event for populating package set")`
		`+`
		`+ create_build_parser = subparsers.add_parser(`
		`+ 'create-build', help='Create new compose from Koji builds.')`
		`+ create_build_parser.set_defaults(command='create-build')`
		`+ create_build_parser.add_argument(`
		`+ '--flag', default=[], action='append',`
		`+ help="Flag to pass to influence the compose.")`
		`+ create_build_parser.add_argument(`
		`+ 'builds', metavar='NVR', nargs='+',`
		`+ help='Koji builds NVRs.')`
		`+ create_build_parser.add_argument(`
		`+ '--sigkey', default=[], action='append',`
		`+ help="Signing key ID using which the packages must be signed - "`
		`+ "\none\" for no signing key.")`
		`+`
		`wait_parser = subparsers.add_parser(`
		`'wait', help='wait for a compose to finish')`
		`wait_parser.set_defaults(command='wait')`
		`@@ -207,9 +320,22 @@`
		`odcs_url,`
		`auth_mech=odcs.client.odcs.AuthMech.Kerberos,`
		`)`
		`+`
		`+ request_args = {}`
		`+ if getattr(args, "flag", False):`
		`+ request_args["flags"] = args.flag`
		`+ if getattr(args, "arch", False):`
		`+ request_args["arches"] = args.arch`
		`+ if getattr(args, "lookaside_repo", False):`
		`+ request_args["lookaside_repos"] = args.lookaside_repo`
		`+ if getattr(args, "label", False):`
		`+ request_args["label"] = args.label`
		`+ if getattr(args, "compose_type", False):`
		`+ request_args["compose_type"] = args.compose_type`
		`+`
		`try:`
		`+ args.sigkey = [key.replace('none', '') for key in getattr(args, "sigkey", [])]`
		`if args.command == "create":`
		`- args.sigkey = [key.replace('none', '') for key in args.sigkey]`
		`result = client.new_compose(`
		`source=args.source,`
		`source_type=args.source_type,`
		`@@ -217,16 +343,32 @@`
		`results=args.result,`
		`sigkeys=args.sigkey,`
		`koji_event=args.koji_event,`
		`- flags=args.flag,`
		`- arches=args.arch,`
		`builds=args.builds,`
		`modular_koji_tags=args.modular_tag,`
		`module_defaults_url=args.module_defaults_url,`
		`module_defaults_commit=args.module_defaults_commit,`
		`- lookaside_repos=args.lookaside_repo,`
		`- label=args.label,`
		`- compose_type=args.compose_type`
		`+ **request_args`
		`)`
		`+ elif args.command == "create-tag":`
		`+ source = odcs.client.odcs.ComposeSourceTag(`
		`+ args.tag, args.packages, args.build, args.sigkey, args.koji_event,`
		`+ args.modular_tag, args.module_defaults_url, args.module_defaults_commit)`
		`+ result = client.request_compose(source, **request_args)`
		`+ elif args.command == "create-module":`
		`+ source = odcs.client.odcs.ComposeSourceModule(`
		`+ args.modules, args.sigkey, args.module_defaults_url, args.module_defaults_commit)`
		`+ result = client.request_compose(source, **request_args)`
		`+ elif args.command == "create-pulp":`
		`+ source = odcs.client.odcs.ComposeSourcePulp(args.content_sets)`
		`+ result = client.request_compose(source, **request_args)`
		`+ elif args.command == "create-raw-config":`
		`+ source = odcs.client.odcs.ComposeSourceRawConfig(`
		`+ args.raw_config_name, args.raw_config_commit, args.koji_event)`
		`+ result = client.request_compose(source, **request_args)`
		`+ elif args.command == "create-build":`
		`+ source = odcs.client.odcs.ComposeSourceBuild(`
		`+ args.builds, args.sigkey)`
		`+ result = client.request_compose(source, **request_args)`
		`elif args.command == "wait":`
		`result = {"id": int(args.compose_id)}`
		`elif args.command == "delete":`

client/odcs/client/odcs.py

file modified

+151

		`@@ -69,6 +69,131 @@`
		`value_error='per_page must be greater than or equal to 1.')`


		`+ class ComposeSourceGeneric(object):`
		`+ def __init__(self, source, source_type, packages=None, builds=None, sigkeys=None,`
		`+ koji_event=None, modular_koji_tags=None, module_defaults_url=None,`
		`+ module_defaults_commit=None, **kwargs):`
		`+ self.source = {'source': source, 'type': source_type}`
		`+ if packages:`
		`+ self.source['packages'] = packages`
		`+ if builds:`
		`+ self.source['builds'] = builds`
		`+ if sigkeys:`
		`+ self.source['sigkeys'] = sigkeys`
		`+ if koji_event:`
		`+ self.source['koji_event'] = koji_event`
		`+ if modular_koji_tags:`
		`+ self.source['modular_koji_tags'] = modular_koji_tags`
		`+ if module_defaults_url:`
		`+ self.source['module_defaults_url'] = module_defaults_url`
		`+ if module_defaults_commit:`
		`+ self.source['module_defaults_commit'] = module_defaults_commit`
		`+ self.source.update(kwargs)`
		`+`
		`+`
		`+ class ComposeSourceTag(ComposeSourceGeneric):`
		`+ """`
		`+ Compose source taking Koji tag as input.`
		`+ """`
		`+ def __init__(self, tag, packages=None, builds=None, sigkeys=None,`
		`+ koji_event=None, modular_koji_tags=None, module_defaults_url=None,`
		`+ module_defaults_commit=None, **kwargs):`
		`+ """`
		`+ Creates new ComposeSourceTag instance.`
		`+`
		`+ :param str tag: Koji tag to use as a source for compose.`
		`+ :param list packages: List of Koji packages to include in the compose. Note`
		`+ that this is not a list of RPM names. If unset, all packages tagged in`
		+ the Koji ``tag`` will be included in a compose.
		`+ :param list builds: List of NVRs of Koji builds to be included in the compose`
		+ on top of the Koji builds directly tagged in the Koji ``tag``. If unset,
		+ only Koji builds tagged in the Koji ``tag`` will be used.
		`+ :param list sigkeys: List of signature keys by which the RPMs in the compose must be`
		`+ signed. Empty string in the list allows unsigned packages.`
		`+ :param int koji_event: Particular Koji event id to generate compose from. If unset,`
		`+ latest Koji event will be used.`
		`+ :param list modular_koji_tags: List of Koji tags containing modules which should also`
		+ be included in the resulting compose on top of Koji builds tagged in the Koji ``tag``.
		`+ :param str module_defaults_url: URL of module defaults repository.`
		`+ :param str module_defaults_commit: Commit or branch name defining particular point`
		`+ module defaults repository.`
		`+ """`
		`+ super(ComposeSourceTag, self).__init__(`
		`+ tag, "tag", packages, builds, sigkeys, koji_event, modular_koji_tags,`
		`+ module_defaults_url, module_defaults_commit, **kwargs)`
		`+`
		`+`
		`+ class ComposeSourceModule(ComposeSourceGeneric):`
		`+ """`
		`+ Compose source taking list of modules as input.`
		`+ """`
		`+ def __init__(self, modules, sigkeys=None, module_defaults_url=None,`
		`+ module_defaults_commit=None, **kwargs):`
		`+ """`
		`+ Creates new ComposeSourceModule instance.`
		`+`
		`+ :param list modules: List of modules in N:S, N:S:V or N:S:V:C format.`
		`+ :param list sigkeys: List of signature keys by which the RPMs in the compose must be`
		`+ signed. Empty string in the list allows unsigned packages.`
		`+ :param str module_defaults_url: URL of module defaults repository.`
		`+ :param str module_defaults_commit: Commit or branch name defining particular point`
		`+ module defaults repository.`
		`+ """`
		`+ super(ComposeSourceModule, self).__init__(`
		`+ " ".join(modules), "module", sigkeys=sigkeys,`
		`+ module_defaults_url=module_defaults_url,`
		`+ module_defaults_commit=module_defaults_commit, **kwargs)`
		`+`
		`+`
		`+ class ComposeSourcePulp(ComposeSourceGeneric):`
		`+ """`
		`+ Compose source taking list of Pulp content_sets as input.`
		`+ """`
		`+ def __init__(self, content_sets, **kwargs):`
		`+ """`
		`+ Creates new ComposeSourcePulp instance.`
		`+`
		`+ :param list content_sets: List of Pulp content-sets. Repositories defined by these`
		`+ content-sets will be included in the compose.`
		`+ """`
		`+ super(ComposeSourcePulp, self).__init__(`
		`+ " ".join(content_sets), "pulp", **kwargs)`
		`+`
		`+`
		`+ class ComposeSourceRawConfig(ComposeSourceGeneric):`
		`+ """`
		`+ Compose source taking raw Pungi configuration file as input.`
		`+ """`
		`+ def __init__(self, config_name, commit, koji_event=None, **kwargs):`
		`+ """`
		`+ Creates new ComposeSourceRawConfig instance.`
		`+`
		`+ :param str config_name: The name of Raw Pungi configuration as configured in ODCS server.`
		`+ :param str commit: The commit hash or branch to get the Pungi configuration from.`
		`+ :param int koji_event: Particular Koji event id to generate compose from. If unset,`
		`+ latest Koji event will be used.`
		`+ """`
		`+ super(ComposeSourceRawConfig, self).__init__(`
		`+ "%s#%s" % (config_name, commit), "raw_config", koji_event=koji_event,`
		`+ **kwargs)`
		`+`
		`+`
		`+ class ComposeSourceBuild(ComposeSourceGeneric):`
		`+ """`
		`+ Compose source taking list of Koji builds as input.`
		`+ """`
		`+ def __init__(self, builds, sigkeys=None, **kwargs):`
		`+ """`
		`+ Creates new ComposeSourceModule instance.`
		`+`
		`+ :param list builds: List of NVRs of Koji builds to be included in the compose.`
		`+ :param list sigkeys: List of signature keys by which the RPMs in the compose must be`
		`+ signed. Empty string in the list allows unsigned packages.`
		`+ """`
		`+ super(ComposeSourceBuild, self).__init__(`
		`+ "", "build", builds=builds, sigkeys=sigkeys, **kwargs)`
		`+`
		`+`
		`class ODCS(object):`
		`"""Client API to interact with ODCS APIs"""`

		`@@ -217,6 +342,9 @@`
		`lookaside_repos=None, label=None, compose_type=None):`
		`"""Request a new compose`

		`+ .. warning::`
		+ This method is deprecated. Please use ``request_compose`` instead.
		`+`
		`:param str source: from where to grab and make new compose, different`
		value for different ``source_type``. For ``tag`` source type, name
		of the tag. For ``module`` source type, white-space separated list
		`@@ -279,6 +407,29 @@`
		`r = self._post('composes/', request_data)`
		`return r.json()`

		`+ def request_compose(self, source, **kwargs):`
		`+ """`
		`+ Request new compose.`
		`+`
		`+ Example:`
		`+`
		`+ .. sourcecode:: python`
		`+`
		`+ source = ComposeSourceTag("f32-updates", packages="httpd")`
		`+ odcs = ODCS(...)`
		`+ compose = odcs.request_compose(source, arches="x86_64", flags=["nodeps"])`
		`+ odcs.wait_for_compose(compose["id"])`
		`+`
		`+ :param ComposeSourceGeneric source: The source of compose.`
		`+ :param kwargs: Extra arguments passed to "/api/1/composes" POST JSON data.`
		`+ :return: the requested Compose object.`
		`+ :rtype: dict`
		`+ """`
		`+ request_data = {"source": source.source}`
		`+ request_data.update(kwargs)`
		`+ r = self._post("composes/", request_data)`
		`+ return r.json()`
		`+`
		`def renew_compose(self, compose_id, seconds_to_live=None):`
		`"""To regenerate an expired compose`

docs/conf.py

file modified

		`@@ -192,6 +192,9 @@`
		`issuetracker_url = 'https://pagure.io/'`
		`issuetracker_project = 'odcs'`

		`+ # To show __init__ in automodule docs.`
		`+ autoclass_content = 'both'`
		`+`
		`# Until Pagure support is merged: https://github.com/ignatenkobrain/sphinxcontrib-issuetracker/pull/15`
		`import requests # noqa`
		`from sphinxcontrib.issuetracker import Issue # noqa`

docs/index.rst

file modified

		`@@ -16,6 +16,7 @@`

		`about`
		`client`
		`+ module`
		`api`
		`raw_config`

docs/module.rst

file added

		`@@ -0,0 +1,6 @@`
		`+ =========================`
		`+ ODCS client Python module`
		`+ =========================`
		`+`
		`+ .. automodule:: odcs.client.odcs`
		`+ :members:`

jkaluza commented 4 years ago

The current ODCS.new_compose method combines all the possible compose arguments
to single method which is very confusing for people who try to use this API.
It is not clearly visible which argument can be used with which source_type.

In this commit, I'm trying to improve this situation by new request_compose
method which accepts source argument. This source argument is instance
of ComposeSourceGeneric subclass. These subclasses directly map to some
source type and their arguments are clearly defined now.

This commit also adds new docs/modules.rst documentation for the Python
module API in general.

The odcs client is also improved in this commit. New subcommands are
added to match the source type (for example create-tag, create-pulp,
create-raw-config, ...). These new commands also use new API.

Old API remains available in both ODCS python module and ODCS client
for backward compatibility.

Signed-off-by: Jan Kaluza jkaluza@redhat.com