From 4e025802be210e1edfbae05c2f8458891dba92ff Mon Sep 17 00:00:00 2001 From: Mike McLean Date: Sep 25 2017 19:37:51 +0000 Subject: PR#595: Release 1.14.0 Merges #595 https://pagure.io/koji/pull-request/595 --- diff --git a/docs/source/conf.py b/docs/source/conf.py index 7853283..24ec112 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -54,9 +54,9 @@ copyright = u'2017, Mike McLean, Mike B, Dennis Gilmore, Mathieu Bridon, Ian McL # built documents. # # The short X.Y version. -version = '1.13' +version = '1.14' # The full version, including alpha/beta/rc tags. -release = '1.13.0' +release = '1.14.0' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/docs/source/content_generator_metadata.rst b/docs/source/content_generator_metadata.rst index 4562399..47c8165 100644 --- a/docs/source/content_generator_metadata.rst +++ b/docs/source/content_generator_metadata.rst @@ -1,8 +1,8 @@ -======================================== -Koji Content Generator Metadata Proposal -======================================== +=============================== +Koji Content Generator Metadata +=============================== -This document describes the proposed Koji Content Generator Metadata +This document describes the Koji Content Generator Metadata Format (version 0). This is the metadata that should be provided by a Content Generator in order for the content to be imported and managed by Koji. If you have further questions about :doc:`Content @@ -108,7 +108,10 @@ Each map in the buildroots list contains the following entries: - checksum: The checksum of the file. - checksum\_type: The checksum type used. +.. _metadata-kojifile: + - For maps where **type = kojifile**, the following fields will be present: + - filename: The name of the file. - filesize: The size of the file. - checksum: The checksum of the file. diff --git a/docs/source/content_generators.rst b/docs/source/content_generators.rst index c54575d..738557a 100644 --- a/docs/source/content_generators.rst +++ b/docs/source/content_generators.rst @@ -12,7 +12,7 @@ provide stable APIs and interfaces to other processes. Along with the content to be managed by Koji, a Content Generator will provide enough metadata to enable a reasonable level of auditing and -reproduceability. The exact data provided and the format used is being +reproducibility. The exact data provided and the format used is being discussed, but will include information like the upstream source URL, build tools used, build environment contents, and any container/virtualization technologies used. diff --git a/docs/source/defining_hub_policies.rst b/docs/source/defining_hub_policies.rst index 0b7ceee..03c20f1 100644 --- a/docs/source/defining_hub_policies.rst +++ b/docs/source/defining_hub_policies.rst @@ -247,7 +247,7 @@ Available tests * for use with the channel policy ``method`` - * matches the task method name agaist glob pattern(s) + * matches the task method name against glob pattern(s) * true if the method name matches any of the patterns * for use with the channel policy diff --git a/docs/source/external_repo_server_bootstrap.rst b/docs/source/external_repo_server_bootstrap.rst index 0b6d790..596d014 100644 --- a/docs/source/external_repo_server_bootstrap.rst +++ b/docs/source/external_repo_server_bootstrap.rst @@ -83,7 +83,7 @@ Examples of urls to use for external Repositories all these examples use mirrors.kernel.org please find the closest mirror to yourself. Note that the Fedora minimal buildroots download ~100Mb then build dependencies on top. these are downloaded each build you can -save alot of network bandwidth by using a local mirror or running +save a lot of network bandwidth by using a local mirror or running through a caching proxy. NOTE: this uses $arch **NOT** $basearch diff --git a/docs/source/migrating_to_1.10.rst b/docs/source/migrating_to_1.10.rst index 8002abb..7ae49e6 100644 --- a/docs/source/migrating_to_1.10.rst +++ b/docs/source/migrating_to_1.10.rst @@ -67,7 +67,7 @@ And there are three new commands Other Configuration changes --------------------------- -The Koji web interface can now treate ``extra-footer.html`` as a Cheetah +The Koji web interface can now treat ``extra-footer.html`` as a Cheetah template. This behavior can be enabled by setting the ``LiteralFooter`` option to ``False`` in the kojiweb config. diff --git a/docs/source/migrating_to_1.12.rst b/docs/source/migrating_to_1.12.rst index 8481fe4..a99106b 100644 --- a/docs/source/migrating_to_1.12.rst +++ b/docs/source/migrating_to_1.12.rst @@ -71,7 +71,7 @@ Changes to calls: * repo related calls (e.g. ``repoInfo`` now include a boolean ``dist`` field * the ``editTag2`` call can now remove tag_extra data if the - ``remove_extra`` keywarg argument is used + ``remove_extra`` keyword argument is used * the listTaskOutput call supports a new ``all_volumes`` keyword argument. If true, the results are extended to deal with files in same relative paths on different volumes. diff --git a/docs/source/migrating_to_1.13.rst b/docs/source/migrating_to_1.13.rst index 4648fbf..f6321ef 100644 --- a/docs/source/migrating_to_1.13.rst +++ b/docs/source/migrating_to_1.13.rst @@ -4,7 +4,7 @@ Migrating to Koji 1.13 .. reStructured Text formatted -The 1.13 release of Koji includes a several changes that you should consider when +The 1.13 release of Koji includes several changes that you should consider when migrating. DB Updates diff --git a/docs/source/migrating_to_1.14.rst b/docs/source/migrating_to_1.14.rst new file mode 100644 index 0000000..4549b41 --- /dev/null +++ b/docs/source/migrating_to_1.14.rst @@ -0,0 +1,39 @@ +Migrating to Koji 1.14 +====================== + +.. + reStructured Text formatted + +You should consider the following changes when migrating to 1.14: + +DB Updates +---------- + +The schema updates this time are minor + + * dropped unused ``log_messages`` table + * new standard entries in the ``archivetypes`` table + +As in previous releases, we provide a migration script that updates the +database. + +:: + + # psql koji koji [ ...] + Usage: koji hostinfo [options] [ ...] (Specify the --help global option for a list of other help options) Options: @@ -75,7 +75,7 @@ Enhancements to restart-hosts The `restart-hosts` command is used by admins to safely restart the build hosts after a configuration change. -Because multiple restarts can confict, the command will now exit with a error +Because multiple restarts can conflict, the command will now exit with a error if a restart is already underway (can be overridden with --force). There are now options to limit the restart to a given channel or arch. diff --git a/docs/source/release_notes_1.14.rst b/docs/source/release_notes_1.14.rst new file mode 100644 index 0000000..6e1be87 --- /dev/null +++ b/docs/source/release_notes_1.14.rst @@ -0,0 +1,310 @@ +Koji 1.14 Release Notes +======================= + +Migrating from Koji 1.13 +------------------------ + +For details on migrating see :doc:`migrating_to_1.14` + + +Client Changes +-------------- + + +Fail fast option for builds +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +| PR: https://pagure.io/koji/pull-request/432 + + +When builders are configured with ``build_arch_can_fail = True`` then the +failure of a single buildArch task does not immediately cause the build +to fail. Instead, the remaining buildArch tasks are allowed to complete, +at which point the build will still fail. + +Sometimes developers would rather a build fail immediately, so we have added +the ``--fail-fast`` option to the build command, which overrides this setting. +The option only has an effect if the builders are configured to fail slow. + + +Custom Lorax templates +^^^^^^^^^^^^^^^^^^^^^^ + +| PR: https://pagure.io/koji/pull-request/419 + +Koji now supports custom Lorax templates for the ``spin-livemedia`` command. +The command accepts two new options: + +.. code-block:: text + + --lorax_url=URL The URL to the SCM containing any custom lorax + templates that are to be used to override the default + templates. + --lorax_dir=DIR The relative path to the lorax templates directory + within the checkout of "lorax_url". + + +The Lorax templates must come from an SCM, and the ``allowed_scms`` rules +apply. + +When these options are used, the templates will be fetched and an appropriate +``--lorax-templates`` option will be passed to the underlying livemedia-creator +command. + + +Allow profiles to request a specific python version +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +| PR: https://pagure.io/koji/pull-request/566 + +On platforms with python3 available, the Koji client is built to execute +with the python3 binary. However, there are a few client features that do not +work under python3, notably old-style (non-gssapi) Kerberos authentication. + +If this issue is affecting you, you can set ``pyver=2`` in your Koji +configuration. This can be done per profile. When Koji sees this setting +at startup, it will re-execute itself under the requested python binary. + + +New list-builds command +^^^^^^^^^^^^^^^^^^^^^^^ + +| PR: https://pagure.io/koji/pull-request/526 + +The command line now has a ``list-builds`` command that has similar +functionality to the builds tab of the web interface. + +.. code-block:: text + + Usage: koji list-builds [options] + (Specify the --help global option for a list of other help options) + + Options: + -h, --help show this help message and exit + --package=PACKAGE List builds for this package + --buildid=BUILDID List specific build from ID or nvr + --before=BEFORE List builds built before this time + --after=AFTER List builds built after this time + --state=STATE List builds in this state + --type=TYPE List builds of this type. + --prefix=PREFIX Only list packages starting with this prefix + --owner=OWNER List builds built by this owner + --volume=VOLUME List builds by volume ID + -k FIELD, --sort-key=FIELD + Sort the list by the named field + -r, --reverse Print the list in reverse order + --quiet Do not print the header information + + +New block-group command +^^^^^^^^^^^^^^^^^^^^^^^ + +| PR: https://pagure.io/koji/pull-request/509 + +The ``block-group`` command allows admins to block package group entries +without having to resort to the ``call`` command. + +.. code-block:: text + + Usage: koji block-group + (Specify the --help global option for a list of other help options) + + Options: + -h, --help show this help message and exit + + +Exit codes for some commands +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +| PR: https://pagure.io/koji/pull-request/558 +| PR: https://pagure.io/koji/pull-request/559 + +Several more commands will now return a non-zero exit code +when an error occurs: + + * the various image building commands + * the ``save-failed-tree`` command (provided by a plugin) + + +Easier for scripts to use activate_session +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +| PR: https://pagure.io/koji/pull-request/493 + +In Koji 1.13.0, it became possible for scripts to ``import koji_cli.lib`` and +gain access to the ``activate_session`` function that the command line tool +uses to authenticate. + +In this release, this function has been made easier for scripts to use: + + * the options argument can now be a dictionary + * less options need to be specified + + +Builder changes +--------------- + + +Normalize paths for scms +^^^^^^^^^^^^^^^^^^^^^^^^ + +| PR: https://pagure.io/koji/pull-request/591 + + +For many years, kojid has supported the ``allowed_scms`` option +(see: :ref:`scm-config`) for controlling which scms can be used for building. +In 1.13.0, Koji added the ability to explicitly block a host:path pattern. + +Unfortunately, 1.13.0 did not normalize the path before checking the pattern, +making it possible for users to use equivalent paths to route around the +block patterns. + +Now, Koji will normalize these paths before the ``allowed_scms`` check. + + +Graceful reload +^^^^^^^^^^^^^^^ + +| PR: https://pagure.io/koji/pull-request/565 + + +For a long time kojid has handled the USR1 signal by initiating a graceful restart. +This change exposes that in the systemd service config (and the init script +on older platforms). + +Now, ``service kojid reload`` will trigger the same sort of restart that the +``restart-hosts`` command accomplishes, but only for the build host you run it +on. When this happens, kojid will: + + * stop taking new tasks + * wait for current tasks to finish + * restart itself once all its tasks are completed + + +Friendlier runroot configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +| PR: https://pagure.io/koji/pull-request/539 +| PR: https://pagure.io/koji/pull-request/528 + +Two changes make it easier to write a configuration for the runroot plugin. + +The ``path_subs`` option is now more forgiving about whitespace: + + * leading and trailing whitespace is ignored for each line + * blank lines are ignored + +The ``[pathNN]`` sections are no longer required to have sequential numbers. +Previously, the plugin expected a sequence like ``[path0]``, ``[path1]``, +``[path2]``, etc, and would stop looking for entries if the next number +was missing. Now, any set of distinct numbers is valid and all ``[pathNN]`` +sections will be processed. + + +System changes +-------------- + +Deprecations +^^^^^^^^^^^^ + +| PR: https://pagure.io/koji/pull-request/554 +| PR: https://pagure.io/koji/pull-request/597 + +The following features are deprecated and will be removed in a future release: + + * the ``importBuildInPlace`` rpc call + * the ``use_old_ssl`` client configuration option (and the underlying + ``koji.compatrequests`` library) + + +Removed calls +^^^^^^^^^^^^^ + +| PR: https://pagure.io/koji/pull-request/497 +| PR: https://pagure.io/koji/pull-request/507 + +The deprecated ``buildFromCVS`` hub call has been removed. It was replaced +by the ``buildSRPMFromCVS`` call many years ago and has been deprecated since +version 1.6.0. + +The ``add_db_logger`` function has been removed from the koji library, along +with the ``log_messages`` table in the db. This extraneous call has never been +used in Koji. + + +Dropped mod_python support +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +| PR: https://pagure.io/koji/pull-request/508 + + +Koji no longer supports mod_python. This option has been deprecated since +mod_wsgi support was added in version 1.7.0. + +See also: :doc:`migrating_to_1.7` + + +Large integer support +^^^^^^^^^^^^^^^^^^^^^ + +| PR: https://pagure.io/koji/pull-request/571 + + +Koji uses xmlrpc for communications with the hub, and unfortunately the +baseline xmlrpc standard only supports 32-bit signed integers. This +results in errors when larger integers are encountered, typically +when a file is larger than 2 GiB. + +Starting with version 1.14.0, Koji will emit ``i8`` tags when encoding +large integers for xmlrpc. Integers below the limit are still encoded +with the standard ``int`` tag. The only time this makes a difference +is when Koji would previously have raised an ``OverflowError``. + +The ``i8`` tag comes from the +`ws-xmlrpc `__ +spec. Python's xmlrpc decoder has +for many years accepted and understood this tag, even though its encoder +would not emit it. + +Previous versions of Koji worked around such size issues by converting +large integers to strings in a few targeted places. Those targeted +workarounds have been left in place on the hub for the sake of backward +compatibility. + + +Test mode for protonmsg plugin +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +| PR: https://pagure.io/koji/pull-request/538 + +The ``protonmsg`` plugin now accepts a boolean ``test_mode`` configuration +option. When this option is enabled, the plugin will not actually +send messages, but will instead log them (at the DEBUG level). + +This option allows testing environments to run with the plugin enabled, but +without requiring a message bus to be set up for that environment. + + +Handling of debugsource rpms +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +| PR: https://pagure.io/koji/pull-request/524 + +Koji will now treat rpms ending in ``-debugsource`` the same way that it does +other debuginfo rpms. Such rpms are: + + * omitted from Koji's normal yum repos + * listed separately when displaying builds + * not downloaded by default in the ``download-build`` command + + +Added kojifile component type for content generators +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +| PR: https://pagure.io/koji/pull-request/506 + +Content generator imports now accept entries with type equal to ``kojifile`` +in the component lists for buildroots and images/archives. This type provides +a more reliable way to reference archive that come from Koji. + +See: :ref:`Example metadata `. diff --git a/docs/source/server_howto.rst b/docs/source/server_howto.rst index 4466593..be92b4d 100644 --- a/docs/source/server_howto.rst +++ b/docs/source/server_howto.rst @@ -566,7 +566,7 @@ Set Database To Listen On All Addresses ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The koji-hub service will attempt to connect to the database server in the manner you configure. If you use the system hostname, then the database will -need to be avalible on that address. To configure this please perform the +need to be available on that address. To configure this please perform the following: #. Edit /var/lib/pgsql/data/postgresql.conf @@ -987,7 +987,7 @@ Add the host entry for the koji builder to the database ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ You will now need to add the koji builder to the database so that they can be -utilized by kojij hub. Make sure you do this before you start kojid for the +utilized by koji hub. Make sure you do this before you start kojid for the first time, or you'll need to manually remove entries from the sessions and users table before it can be run successfully. @@ -1033,7 +1033,7 @@ the same as KojiDir) ; The directory root for temporary storage workdir=/tmp/koji -The root of the koji build directory (ie, ``/mnt/koji``) must be mounted on the +The root of the koji build directory (i.e., ``/mnt/koji``) must be mounted on the builder. A Read-Only NFS mount is the easiest way to handle this. :: @@ -1134,6 +1134,9 @@ a host, but allow from it otherwise alt-server.example.com:/repo/dist/repos*/:no:fedpkg,sources +The explicit block syntax was added in version 1.13.0. + + Add the host to the createrepo channel -------------------------------------- diff --git a/docs/source/using_the_koji_build_system.rst b/docs/source/using_the_koji_build_system.rst index a6c2b0a..0cc0e00 100644 --- a/docs/source/using_the_koji_build_system.rst +++ b/docs/source/using_the_koji_build_system.rst @@ -66,7 +66,7 @@ by running ``fedora-cert -n`` again. You can check if it has expired with ``fedora-cert -v``. the following are downloaded automatically by fedora-packager-setup and -dont need to be manually setup +don't need to be manually setup ``~/.fedora-upload-ca.cert`` (The certificate for the Certificate Authority used to sign the user keys.) : It can be manually downloaded @@ -258,7 +258,7 @@ the next group will begin building. For example: ``fedpkg chain-build libwidget libaselib : libgizmo :`` will cause libwidget and libaselib to be built in parallel, followed by -libgizmo and then the currect directory package. If no groups are +libgizmo and then the correct directory package. If no groups are defined, packages will be built sequentially. If a build fail, following builds are cancelled but the builds that @@ -406,7 +406,7 @@ any mock-config command to get a working mock-config from fedoras koji. .. _tuning-mock-per-tag: -Tuning mock's behaviour per tag +Tuning mock's behavior per tag ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Few options for mock can be configured per-tag. These options are stored in @@ -423,7 +423,7 @@ environment follows: package manager. Typically used with `yum` or `dnf` values. * `mock.new_chroot` - 0/1 value. If it is set, `--new-chroot` or `--old-chroot` option is appended to any mock call. If it is not set, - mock's default behaviour is used. + mock's default behavior is used. Using Koji to control tasks ^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/source/writing_a_plugin.rst b/docs/source/writing_a_plugin.rst index 23bc248..7702867 100644 --- a/docs/source/writing_a_plugin.rst +++ b/docs/source/writing_a_plugin.rst @@ -218,7 +218,7 @@ the command. In the first case, the command will not automatically authenticate with the hub (though the user can still override -this behaviour with `--force-auth` option). In the second case, the command +this behavior with `--force-auth` option). In the second case, the command will perform authentication by default (this too can be overridden by the user with the `--noauth` option). diff --git a/docs/source/writing_koji_code.rst b/docs/source/writing_koji_code.rst index b9b67b1..7f25f31 100644 --- a/docs/source/writing_koji_code.rst +++ b/docs/source/writing_koji_code.rst @@ -632,6 +632,7 @@ Here are some guidelines on producing preferable pull requests. - Code which is imported into CLI or needed for stand-alone API calls must run in both 2.6+ and 3.x python versions. We use the python-six library for compatibility. The affected files are: + - ``cli/*`` - ``koji/__init__.py`` - ``koji/auth.py`` @@ -639,6 +640,7 @@ Here are some guidelines on producing preferable pull requests. - ``koji/util.py`` - ``tests/test_lib/*`` - ``tests/test_cli/*`` + - Check, that unit tests are not broken. Simply run ``make test`` in main directory of your branch. For python3 compatible-code we have also ``make test3`` target. diff --git a/koji.spec b/koji.spec index 943d12e..30f5266 100644 --- a/koji.spec +++ b/koji.spec @@ -31,7 +31,7 @@ %define release %{baserelease} %endif Name: koji -Version: 1.13.0 +Version: 1.14.0 Release: %{release}%{?dist} License: LGPLv2 and GPLv2+ # koji.ssl libs (from plague) are GPLv2+ @@ -480,6 +480,58 @@ fi %endif %changelog +* Mon Sep 25 2017 Mike McLean - 1.14.0-1 +- PR#597: use_old_ssl is deprecated +- PR#591: Normalize paths for scms +- PR#432: override build_arch_can_fail settings +- PR#566: allow profiles to request a specific python version +- PR#554: deprecate importBuildInPlace hub call +- PR#590: support repo_include_all tag extra option +- PR#582: Content generator metadata documentation update +- PR#579: ignore inodes when running rpmdiff. +- PR#493: modify activate_session to be easily used without CLI +- PR#589: fix scratch ref for scm callback +- PR#587: add `build_tag` argument in `postSCMCheckout` callback +- PR#583: support rpm LONG*SIZE header fields +- PR#526: Added list builds command to koji CLI +- PR#581: Add a note to get_build docstring +- PR#575: add xjb and yaml type in archivetypes table +- PR#571: Support large ints over xmlrpc using i8 tag +- PR#538: protonmsg plugin: test mode +- PR#547: update version in sphinx config +- PR#548: set task arch for indirection image builds +- PR#568: spec: use correct macro - rhel instead redhat for RHEL version +- PR#558: cli: Fix exit code for building images +- PR#559: return result status in save-failed-tree +- PR#561: rename rpm-python to python*-rpm for EOL of F24 +- PR#562: fix serverca default in kojivmd +- PR#565: expose graceful reload in kojid service config and init script +- PR#544: incorrect parameter for error message +- PR#510: cli: change download-task to regular curl download +- PR#536: fix docs links, plus minor docs cleanup +- PR#539: runroot: friendlier parsing of path_subs config +- PR#542: check RPMTAG_LONGSIZE is RPMTAG_SIZE is null +- PR#419: Koji support for custom Lorax templates in LiveMedia tasks +- PR#546: fix test_krbv_disabled unit test +- PR#518: Error out if krbV is unavailable and gssapi did not work +- PR#535: datetime compatibility for plugins +- PR#524: Add support for debugsource +- PR#528: allow some missing path sections in runroot config +- PR#530: Spelling fixes +- PR#506: Track artifacts coming from koji itself +- PR#499: runroot: use /builddir/runroot.log instead of /tmp/runroot.log +- PR#509: CLI block-group command +- PR#514: Fix resubmit +- PR#521: update links in README.md +- PR#502: download-build: suppress output on quiet and add --noprogress +- PR#511: unit tests for delete_tag() [Open] +- PR#484: fix NoneType TypeError in deleteTag +- PR#490: getUserPerms should throw GenericError when no user found +- PR#497: remove deprecated buildFromCVS call +- PR#503: Remove deprecated compat_mode from runroot plugin +- PR#507: drop unused add_db_logger call and db table +- PR#508: drop mod_python support + * Fri Jun 30 2017 Mike McLean - 1.13.0-1 - PR#496 Makefile/spec fixes for building on el6 - PR#491 epel-compatible macro in spec