#41 Updates policy update
Merged 3 years ago by sgallagh. Opened 3 years ago by zbyszek.
fesco/ zbyszek/fesco-docs updates-update  into  master

The added file is too large to be shown here, see it at: fesco/modules/ROOT/assets/images/update_flow.svg
@@ -1,3 +1,5 @@ 

+ :toc:

+ 

  = Updates policy

  

  Fedora has differing policies for each of its branches. This document describes for maintainers what
@@ -5,206 +7,227 @@ 

  In the event of questions or clarifications, please file a

  link:https://pagure.io/fesco/new_issue[FESCo ticket] or discuss on the

  link:https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/[devel list]. In

- general, releases should go from less conservative (rawhide) to more so (the oldest supported stable

+ general, releases should go from less conservative (Rawhide) to more so (the oldest supported stable

  release). This document attempts to describe when and what kinds of updates maintainers should push

  to Fedora users of its various branches. The

- link:https://fedoraproject.org/wiki/Stable_release_updates_vision[Stable release updates vision] from the

+ link:https://fedoraproject.org/wiki/Stable_release_updates_vision#Vision_Statement[Stable release updates vision] from the

  Fedora Board includes more high level discussion and philosophy, while this

  document is more a practical guide. Refer to

  link:https://fedoraproject.org/wiki/Package_update_HOWTO[Package update HOWTO] for the technical steps on

  pushing the updates. The link:https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle[Fedora Release Life Cycle]

  provides a more detailed overview of the development process.

  

- == Rawhide / devel / master

+ == Common updates requirements for all Fedora releases

+ 

+ Some critera apply to any update to any fedora branch/release:

+ 

+ [[updating-inter-dependent-packages]]

+ === Updating inter-dependent packages

+ 

+ When one updated package requires one or more other packages, the packages should be submitted

+ together as a single update. For instance, if package A depends on packages B and C, and you want to

+ update to a new version of package A which requires new versions of B and C, you must submit a

+ single update containing the updated versions of all three packages. It is a bad idea to submit

+ three separate updates, because if the update for package A is pushed stable before the updates for

+ packages B and C, it will cause dependency problems. There is information on how to submit

+ multi-package updates in the

+ link:https://fedoraproject.org/wiki/Package_update_HOWTO#Updating_inter-dependent_packages[package update HOWTO]

+ and information about using side-tags for multiple updates in

+ link:https://docs.fedoraproject.org/en-US/rawhide-gating/multi-builds/[Rawhide Gating/multi-builds].

+ 

+ [[consumable-updates]]

+ === Consumable updates

+ 

+ link:https://bodhi.fedoraproject.org[Bodhi]

+ updates should only be created for builds which are expected to qualify for being pushed to stable.

+ Maintainers should not use Bodhi's testing states to test updates they never intend to push stable.

+ This sort of testing should be done in link:https://copr.fedorainfracloud.org/[Copr] or other

+ seperate public repositories. Consult with the QA team for further testing assistance.

+ 

+ == Rawhide

  

  link:https://fedoraproject.org/wiki/Releases/Rawhide[Rawhide] is the always-rolling development tree.  Package

- updates built for rawhide are composed every day and pushed out to all rawhide consumers.  New builds against

- this tree also are added to the build root (ie, other packages build from them).

+ updates built for Rawhide are composed every day and pushed out to all consumers.

+ New builds against this tree also are added to the build root (i.e., other packages build from them).

+ This tree is intended to meet the link:https://fedoraproject.org/wiki/Basic_Release_Criteria[Basic Release Critera]

+ for any successfull compose so maintainers can integrate their changes with everyone else.

  

  Repos available: link:https://fedoraproject.org/wiki/Repositories#rawhide[_rawhide_]

  

- Since link:https://fedoraproject.org/wiki/Changes/GatingRawhidePackages[Rawhide Gating change] was introduced,

- package updates in Fedora Rawhide need to pass verification before they land in the rawhide repositories. This

- is implemented as a check for a Bodhi update, which verifies that the update satisfies the Gating policy. See

- link:https://docs.fedoraproject.org/en-US/rawhide-gating/single-builds/[Rawhide Gating/single-builds] and

- link:https://docs.fedoraproject.org/en-US/rawhide-gating/multi-builds/[Rawhide Gating/multi-builds] for

+ Since link:https://fedoraproject.org/wiki/Changes/GatingRawhidePackages[Gating Rawhide Packages] change was introduced,

+ package updates in Fedora Rawhide need to pass verification before they land in the Rawhide repositories. This

+ is implemented as a check for the Bodhi update, which verifies that the update satisfies the Gating policy. See

+ link:https://docs.fedoraproject.org/en-US/rawhide-gating/single-builds/[Rawhide Gating/Single Builds] and

+ link:https://docs.fedoraproject.org/en-US/rawhide-gating/multi-builds/[Rawhide Gating/Multi Builds] for

  details.

  

- Currently the default gating policy is empty, thus Fedora Rawhide update can pass the gate, no matter the test

+ Currently the default gating policy is empty, thus a Rawhide update can pass the gate, no matter the test

  results. Package maintainer can opt-in for the gating of a package, by setting up individual gating policies,

- see link:https://docs.fedoraproject.org/en-US/rawhide-gating/optin/[Rawhide Gating/optin].

+ see link:https://docs.fedoraproject.org/en-US/rawhide-gating/optin/[How to Opt in to Gating].

  

- For updates to rawhide packages, Maintainers SHOULD:

+ As soon as a build is completed, a Bodhi update is automatically created.

+ The update is used to gather test results.

+ If the gating tests pass, the update is marked stable after a few minutes, pushed stable,

+ and will be added to the next nightly compose.

  

- * Try not to push a broken build (breaks the default buildroot package set, etc.).

+ For updates to Rawhide packages, maintainers MUST:

+ 

+ * not push any known broken builds (breaks the default buildroot package set, etc.).

+   This causes additional work for other maintainers trying to integrate their changes.

  * When a proposed update contains an ABI or API change: notify *a week in advance* both the

    link:https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/[devel list]

    and maintainers directly (using the ``_packagename_-\owner@fedoraproject.org`` alias)

    whose packages depend on yours to rebuild or offer to do these rebuilds for them.

- * Use a separate buildsystem tag when dealing with mass builds of many packages, so they can land at

-   the same time. File a link:https://pagure.io/releng/new_issue[rel-eng ticket] for this.

+ * Use a side-tag when dealing with mass builds of many packages, so they can land at

+   the same time. See

+   link:https://docs.fedoraproject.org/en-US/rawhide-gating/multi-builds/[Rawhide Gating/Multi Builds].

  * Feel free to push out the newest version of packages as long as they do not cause breakage. Also

    keep in mind when the next Fedora release will be branched off, and be fairly confident that there

    will be a stable enough release in time for the next Fedora release. Otherwise you may have to

    back down to an older, stable version after the branching, which may involve epochs and other

    inconveniences.

+ * Once a package has been added to the compose and that compose has finished and synced to the master

+   mirrors, it will normally not be untagged. This is required to allow others to depend on the

+   build once it has become visible. In exceptional cases, releng may untag packages.

  * If approved by FESCo, push pre-release versions of low level packages.

-   FESCo approves certain packages, including (but not limited to) ``glibc`` and ``gcc``,

+   FESCo approves certain packages, including (but not limited to) `glibc` and `gcc`,

    to provide pre-release versions here.

    The benefits of the early real-world testing of and upstream collaboration on these key

    packages far exceeds the risks that they may introduce.

  

+ image::update_flow.svg[Update Flow]

+ 

  == Branched release

  

  A link:https://fedoraproject.org/wiki/Releases/Branched[Branched] release exists for part of the

- development cycle. It is branched off Rawhide and eventually becomes the next stable release.

- After the <<bodhi-enabling>> point, Branched releases do use the update feedback system (Bodhi).

- There are several "phases" that a branched release goes through that affect what updates can and

- should be done. In general maintainers should keep in mind that this tree is being used to stabilize

- for the next release, so changes should be careful and considered and heading toward stability.

- 

- [[branched-pre-bodhi-enabling]]

- === https://fedoraproject.org/wiki/Releases/Branched[Branched], pre-Bodhi enabling

+ development cycle. It starts as a fork of Rawhide and eventually becomes the next stable release.

+ All successfull branched composes should meet the

+ link:https://fedoraproject.org/wiki/Basic_Release_Criteria[Basic Release Critera].

  

- For a short time after link:https://fedoraproject.org/wiki/Fedora_Release_Life_Cycle[the branch event

- happens] but before the <<bodhi-enabling>> point, the Branched release works exactly like Rawhide:

- builds submitted by packagers are considered

- link:https://fedoraproject.org/wiki/Repositories#stable[stable] immediately, and sent to the

- link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_ repository] directly in the next nightly

- compose. There are no restrictions beyond those for Rawhide at this point, but maintainers *SHOULD* be

- thinking about stabilization from this point onward, and making sure their packages will be in good

- condition well in advance of the stable release. Some packages will be signed at this point, but not

- all of them. You may have to use `--nogpgcheck` to install or update.

+ Branched releases use the update feedback system (link:https://bodhi.fedoraproject.org[Bodhi]): at first just like Rawhide (updates are

+ automatically created when a build finishes, tests run and builds are automatically pushed into the next

+ compose), but then after <<updates-testing-activation>> they switch to using it as stable releases do

+ (maintainers must create updates and submit them for testing, etc).

  

- Repos available: link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_]

+ There are several phases that a branched release goes through that affect what updates can and

+ should be done. In general maintainers should keep in mind that this tree is being stabilized

+ for the next release, so changes should be careful and considered and heading toward stability.

  

- [[bodhi-enabling]]

- === Bodhi enabling

+ After branching, there are three freezes,

+ <<post-branch-freeze>>, <<beta-freeze>>, and <<final-freeze>>.

  

- At the _Bodhi enabling point_, the Bodhi update system is enabled for the Branched release. After

- this point *for the entire remaining lifetime of the release*, all updates must go through Bodhi

- before they can be marked _stable_ and moved to either

- link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_] or

- link:https://fedoraproject.org/wiki/Repositories#updates[_updates_].

- At present, the _Bodhi enabling point_ occurs on the day of the

- link:https://fedoraproject.org/wiki/Milestone_freezes[Beta freeze],

- but it is important to remember that they are distinct events:

- the Beta freeze lasts until Beta release, but Bodhi enabling is permanent.

+ [[post-branch-freeze]]

+ === Post-branch Freeze

  

- Repos available: link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_],

- link:https://fedoraproject.org/wiki/Repositories#updates-testing[_updates-testing_]

+ Once the new release is branched off Rawhide, the flow of updates through Bodhi is stopped until

+ there has been a successful branched compose. This period usually lasts just a few days. Release

+ engineering may pass some updates to stable to get the compose to complete, but otherwise all updates

+ are paused until this initial branched compose is completed. This is to make sure we have a

+ compose to build on and aren't dealing with problems landing in new updates before we are

+ ready for them.

  

- At this point all packages will be signed and gpgcheck should be enabled for all installs/updates.

+ [[before-updates-testing-activation]]

+ === Before Updates-testing Activation

  

- [[pre-beta]]

- === Pre Beta

+ For a short time after branching but before the <<beta-freeze>>,

+ the Branched release works like Rawhide:

+ builds submitted by packagers are considered

+ _stable_ after passing through any gating tests

+ via a Bodhi update, and are sent to the

+ link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_] repository directly in the next nightly

+ compose. There are no restrictions beyond those for Rawhide at this point, but maintainers SHOULD be

+ thinking about stabilization from this point onward, and making sure their packages will be in good

+ condition well in advance of the stable release.

  

- This is the time between the <<bodhi-enabling>> point and the Beta release. During this time we are

- attempting to stabilize the major versions of software that will be shipped with the final release

- of the OS. Major updates can be tolerated, but breaking things for early testers should be avoided

- if possible. Additionally, as we get close to Alpha or Beta releases any change that breaks composes

- of Live media, install media or testing should be avoided. Packages for Changes should be landing

- and getting major testing.

+ Repos available: link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_]

  

- Repos available: link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_],

+ [[updates-testing-activation]]

+ === Updates-testing Activation

+ 

+ At this point, the Bodhi update system is changed for the branched release to behave

+ as it does for stable releases (see below) instead of Rawhide.

+ From this point onward maintainers must create updates before packages become available to users

+ and updates pass through link:https://fedoraproject.org/wiki/QA:Updates_Testing[_updates-testing_]

+ to allow for feedback.

+ Updates are moved from link:https://fedoraproject.org/wiki/Repositories#updates-testing[_updates-testing_]

+ repository to link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_]

+ repository after appropriate karma requirements have been reached.

+ Bodhi sets reasonable defaults for karma and enforces minimum requirements for updates.

+ <<karma-requirements>> for updates are described below.

+ 

+ Repositories available: link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_],

  link:https://fedoraproject.org/wiki/Repositories#updates-testing[_updates-testing_]

  

- From this point onward Maintainers

- **MUST**footnote:enforced[_Must_ requirements are enforced by Bodhi.]:

+ Maintainers SHOULD:

  

- * Push all updates first to updates-testing.

- * All link:https://fedoraproject.org/wiki/Critical_path_package[critical path] updates

-   *MUST* either get one +1 karma,

-   *OR* spend at least 14 days in updates-testing and have no negative karma

-   before being moved to stable.

- * All non critical path updates

-   *MUST* either reach the prescribed karma level for that update,

-   *OR* spend at least 3 days in updates-testing

-   before being allowed to move to stable.

+ * Avoid ABI/API changes where possible. If unavoidable, use a side-tag to rebuild packages.

+ * Avoid any change that breaks composes of Live media, install media or testing.

+ * Land any packages required for Changes planned for that release.

  

- and Maintainers *SHOULD* (not enforced):

+ [[beta-freeze]]

+ === Beta Freeze

  

- * Avoid ABI/API changes where possible. If unavoidable, should coordinate a side-tag to rebuild

-   packages in or a mass build/update, by filing a https://pagure.io/releng/new_issue[releng ticket].

- 

- [[milestone-freezes]]

- ==== Milestone freezes

- 

- During this period there are two link:https://fedoraproject.org/wiki/Milestone_freezes[Milestone freezes],

- _Alpha freeze_ and _Beta freeze_. Each is scheduled to run for the two weeks leading up to

- the release date (and in fact lasts until the release is signed off, even if it is delayed). During

- these times, builds will not be marked _stable_ and moved from

+ This freeze is scheduled to run for the three weeks leading up to the release date,

+ but lasts until the release is signed off, even if it is delayed.

+ During the freeze builds will not be marked _stable_ and moved from

  link:https://fedoraproject.org/wiki/Repositories#updates-testing[_updates-testing_] to

  link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_]

  (and hence included in the milestone release composes) except for those approved under the Fedora QA team

  link:https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process[blocker bug process] or

  link:https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process[freeze exception bug process].

- Once the milestone release is made, these freezes are lifted.

- These freezes are technically not a part of the Updates Policy,

- but are briefly mentioned here for clarity.

+ Once the beta release is made, the freeze is lifted.

  The link:https://fedoraproject.org/wiki/Milestone_freezes[Milestone freezes]

  page provides more details and is the canonical reference in case of any conflict.

  

- [[beta-to-pre-release]]

- === Beta to Pre Release

+ Once the Beta Freeze starts, we are

+ attempting to stabilize the major versions of software that will be shipped with the final release

+ of the OS. Major updates can be tolerated, but breaking things for early testers should be avoided

+ if possible.

  

- This is the time between the Beta release and the

- link:https://fedoraproject.org/wiki/Milestone_freezes[Final freeze].

+ During this period:

+ 

+ * All updates pulled into the release MUST fix an accepted blocker or freeze exception bug.

+ * All updates still go to

+   link:https://fedoraproject.org/wiki/Repositories#updates-testing[_updates-testing_].

+ 

+ Repositories available: link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_],

+ link:https://fedoraproject.org/wiki/Repositories#updates-testing[_updates-testing_]

+ 

+ [[beta-to-final-freeze]]

+ === Beta to Final Freeze

+ 

+ This is the time between the Beta release and the <<final-freeze>>.

  The branched tree should now be stabilized and prepared for release.

  Major changes should be avoided during this period.

  Bear in mind that in most cases, the state your package has reached in the stable

  link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_]

- repository at the time of the Final freeze is the state it will be in for the final release.

+ repository at the time of the Final Freeze is the state it will be in for the final release.

  

- Repos available: link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_],

+ Repositories available: link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_],

  link:https://fedoraproject.org/wiki/Repositories#updates-testing[_updates-testing_]

  

- From this point onward maintainers **MUST**footnote:enforced[]:

- 

- * Avoid Major version updates, ABI breakage or API changes if at all possible.

- * Push updates first to updates-testing.

- * All link:https://fedoraproject.org/wiki/Critical_path_package[critical path]

-   updates

-   *MUST* either have a sum of +2 karma,

-   *OR* spend at least 14 days in updates-testing

-   and have no negative karma.

- * All non critical path updates

-   *MUST* either reach the prescribed karma level for that update,

-   *OR* spend at least 7 days in updates-testing

-   before being allowed to go to stable.

- 

- [[pre-release]]

- === Pre release

- 

- This is the time after the link:https://fedoraproject.org/wiki/Milestone_freezes[Final freeze].

- During this time the release is being composed and only packages granted exceptions through the

- link:https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process[blocker bug process] or

- link:https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process[freeze exception bug process]

- are marked as _stable_ and moved to the

- link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_ repository].

+ [[final-freeze]]

+ === Final Freeze

+ 

+ This freeze leads to the creation of the final release.

+ It is similar to the <<beta-freeze>> described above and follows the same update rules.

+ 

  The link:https://fedoraproject.org/wiki/Repositories#updates[_updates_]

  repository is enabled at some point during this time,

  and packages other than freeze exception / blocker fixes are queued for so called "zero day" updates,

  meaning they will be available in the _updates_ repository at the time of the release (day zero).

  

- Repos available: link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_],

+ Repositories available: link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_],

  link:https://fedoraproject.org/wiki/Repositories#updates[_updates_],

  link:https://fedoraproject.org/wiki/Repositories#updates-testing[_updates-testing_]

  

  During this period:

  

- * All updates pulled into the release

-   **MUST**footnote:enforced[] fix an accepted blocker or freeze exception bug.

- * All update pushes will go to updates-testing.

- * All link:https://fedoraproject.org/wiki/Critical_path_package[critical path] updates

-   *MUST* either have a sum of +2 karma,

-   *OR* spend at least 14 days in updates-testing and have no negative karma.

- * All non critical path updates

-   *MUST* either reach the prescribed karma level for that update,

-   *OR* spend at least 7 days in updates-testing

-   before being allowed in stable.

- * Once the _updates_ repository is available, builds marked as

-   link:https://fedoraproject.org/wiki/Repositories#stable[_stable_]

+ * All updates pulled into the release MUST fix an accepted blocker or freeze exception bug.

+ * All updates still go to

+   link:https://fedoraproject.org/wiki/Repositories#updates-testing[_updates-testing_].

+ * Once the link:https://fedoraproject.org/wiki/Repositories#updates[_updates_]

+   repository is available, builds marked as _stable_

    will go there instead of to

    link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_].

  
@@ -223,9 +246,9 @@ 

  needed over time.

  

  This necessarily means that stable releases will not closely track the very latest upstream code for

- all packages. We have rawhide for that.

+ all packages. We have Rawhide for that.

  

- Rebases should be carefully considered with respect to their dependencies. A rebase that required

+ Updates should be carefully considered with respect to their dependencies. An update that required

  (or provided) a new Python ABI, for example, would almost certainly not be allowed. ABI changes in

  general are very strongly discouraged, they force larger update sets on users and they make life

  difficult for third-party packagers. Additionally, updates that convert resources or configuration
@@ -233,76 +256,27 @@ 

  less chance of backing out an update that did these things.

  

  Whenever possible packagers should work with upstream to come up with stable branch releases or

- common patches for older releases, particularly when rebasing would require large dependency chain

+ common patches for older releases, particularly when updating would require large dependency chain

  updates.

  

- Repos available: link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_],

+ Repositories available: link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_],

  https://fedoraproject.org/wiki/Repositories#updates[_updates_],

  link:https://fedoraproject.org/wiki/Repositories#updates-testing[_updates-testing_]

  

- [[updates-to-critical-path-packages]]

- === Updates to 'critical path' packages

- 

- Updates that constitute a part of the 'critical path' package set

- (defined below) *including security updates*

- *MUST* follow the rules as defined for

- link:https://fedoraproject.org/wiki/Critical_path_package[critical path packages]

- for pending releases, meaning:

- 

- * At the time of the request to stable, the update needs to have either a

-   link:https://fedoraproject.org/wiki/Bodhi[Bodhi karma]

-   sum of 2 *OR*

- * It must spend at least 14 days in updates-testing

-   *AND* have no negative

-   link:https://fedoraproject.org/wiki/Bodhi[Bodhi karma] points.

- 

- For the purposes of this policy, the 'critical path' package set is defined as the following:

- 

- * The current

-   link:https://fedoraproject.org/wiki/Critical_Path_Packages[critical path package set]

- * All major desktop environments' core functionality (GNOME, KDE, Xfce, LXDE)

- * Package updating frameworks

-   (link:https://src.fedoraproject.org/rpms/gnome-packagekit[``gnome-packagekit``],

-   link:https://src.fedoraproject.org/rpms/apper[``apper``])

- * Major desktop productivity apps.

-   An initial list would be

-   link:https://src.fedoraproject.org/rpms/firefox[``firefox``],

-   link:https://src.fedoraproject.org/rpms/kde-baseapps[``kde-baseapps``](konqueror),

-   link:https://src.fedoraproject.org/rpms/thunderbird[``thunderbird``],

-   link:https://src.fedoraproject.org/rpms/evolution[``evolution``],

-   link:https://src.fedoraproject.org/rpms/kdepim[``kdepim``](kmail).

- 

- Changes to this definition may only be made by xref:index.adoc[FESCo] or their delegate.

- 

- [[all-other-updates]]

- === All other updates

- 

- All other updates *MUST* either:

- 

- * reach the criteria laid out in the previous section *OR*

- * reach the positive Bodhi karma threshold specified by the updates submitter *OR*

- * spend some minimum amount of time in

-   link:https://fedoraproject.org/wiki/QA:Updates_Testing[_updates-testing_], currently one week

- 

- Package maintainers *MUST*:

- 

- * Avoid Major version updates, ABI breakage, or API changes if at all possible.

- * Avoid changing the user experience if at all possible.

- * Avoid updates that are trivial or don't affect any Fedora users.

- 

- Package maintainers *SHOULD*:

+ During this period:

  

- * Push *only* major bug fixes and security fixes to the previous stable releases

-   (i.e. the current Fedora _N_, and the previous Fedora _N-1_).

+ * All updates go to

+   link:https://fedoraproject.org/wiki/Repositories#updates-testing[_updates-testing_].

+ * Once <<karma-requirements>> are met, updates are moved to

+   link:https://fedoraproject.org/wiki/Repositories#updates[_updates_] repository.

  

- [[exceptions]]

  === Exceptions

  

  Some classes of software will not fit in these guidelines. If your package does not fit in one of

  the classes below, but you think it should be allowed to update more rapidly, propose a new

  exception class to FESCo and/or request an exception for your specific update case.

  

- Note that you should open this dialog *BEFORE* you build or push updates. In the event that an issue

+ Note that you should open this dialog BEFORE you build or push updates. In the event that an issue

  is raised in the middle of an update already in progress, make sure you turn off autokarma pushes —

  this can be done while the update is pending in Bodhi.

  
@@ -437,14 +411,13 @@ 

    changes are (removing the File menu would be rude, but moving the plugin configuration menu item

    would be acceptable).

  

- 

  * Firefox releases an update that only contains changes for other platforms.

-   This update could be pushed to rawhide (to keep up with the latest version),

+   This update could be pushed to Rawhide (to keep up with the latest version),

    but should not be pushed to stable releases, as it does no good to our users

    and wastes resources to build, update, mirror, and download to our users.

  

  * Terminal fails to build from source when tested in a mass rebuild.

-   An updated package should be pushed to rawhide.

+   An updated package should be pushed to Rawhide.

    Fixes for stable releases should be tested and even committed,

    but unless there is a problem with the previous existing build in the stable release,

    no update should be issued.
@@ -460,6 +433,57 @@ 

    amount of testing and end user visible changes.

    An exception like this would be on a case by case basis, based on all the above.

  

+ [[karma-requirements]]

+ == Karma requirements

+ 

+ This section describes the requirements for an update before it may be pushed

+ from link:https://fedoraproject.org/wiki/Repositories#updates-testing[_updates-testing_]

+ to link:https://fedoraproject.org/wiki/Repositories#fedora[_fedora_]

+ or https://fedoraproject.org/wiki/Repositories#updates[_updates_].

+ The requirements are based on (the sum of)

+ link:https://fedoraproject.org/wiki/Bodhi[Bodhi] karma points

+ and the number of days the update has spent in _updates-testing_ repository.

+ 

+ The push may be either by the maintainer, or automatically by Bodhi:

+ 

+ * The update becomes **eligible for being pushed manually** after reaching the _minimum_ "Stable by Karma" threshold

+ for Critical path updates (yes, the same limit applies to both types of updates),

+ OR the _minimum_ "Stable by Time" threshold.

+ * The update will be **pushed automatically** by Bodhi after reaching the _configured_ "Stable by Karma" threshold,

+ OR the _configured_ "Stable by Time" threshold,

+ if enabled ("Auto-request stable based on karma?" and "Auto-request stable based on time?").

+ * If the update has _any_ negative karma, the automatic push is disabled.

+ * If the update reaches the "Unstable by Karma" threshold, it will be unpushed,

+ i.e. removed from the _updates-testing_ repository.

+ 

+ The maintainer is free to set the thresholds, but they cannot be lower than the minimum values described below,

+ enforced by Bodhi.

+ The defaults are adequate for most packages, so usually there is no need to modify the thresholds.

+ 

+ *Security updates* are subject to the same thresholds as other updates.

+ 

+ [[non-critpath-updates]]

+ === Non-critical path update thresholds

+ 

+ Stable by Karma: minimum +1, default +3, between <<updates-testing-activation>> and <<beta-freeze>> +

+                        minimum +2, default +3, after <<beta-freeze>> +

+ Unstable by Karma: maximum -1, default -3 +

+ Stable by Time (days): minimum 3, default 3, between <<updates-testing-activation>> and <<beta-freeze>> +

+                        minimum 7, default 7, after <<beta-freeze>>

+ 

+ [[critpath-updates]]

+ === Critical path and EPEL update thresholds

+ 

+ "Critical path" updates contain at least one

+ link:https://fedoraproject.org/wiki/Critical_path_package[critical path package]. +

+ Changes to this definition may only be made by xref:index.adoc[FESCo] or their delegate.

+ 

+ Stable by Karma: minimum +1, default +3, between <<updates-testing-activation>> and <<beta-freeze>> +

+                        minimum +2, default +3, after <<beta-freeze>> +

+ Unstable by Karma: maximum -1, default -3 +

+ Stable by Time (days): minimum 3, default 3, between <<updates-testing-activation>> and <<beta-freeze>> +

+                        minimum 14, default 14, after <<beta-freeze>>

+ 

  [[problems-or-issues-with-updates]]

  == Problems or issues with Updates

  
@@ -468,50 +492,11 @@ 

  work to prevent the issue from happening again. A past record of such issues can be found at

  link:https://fedoraproject.org/wiki/Updates_Lessons[Updates Lessons].

  

- FESCo will work with QA and others to prevent or mitigate the issue.

+ [[openqa]]

+ link:https://openqa.fedoraproject.org[OpenQA] tests critpath updates in stable releases and compose artifacts

+ for Rawhide/branched composes/release candidates.

  

- [[updating-inter-dependent-packages]]

- == Updating inter-dependent packages

- 

- When one updated package requires another (or more than one other), the packages should be submitted

- together as a single update. For instance, if package A depends on packages B and C, and you want to

- update to a new version of package A which requires new versions of B and C, you must submit a

- single update containing the updated versions of all three packages. It is a bad idea to submit

- three separate updates, because if the update for package A is pushed stable before the updates for

- packages B and C, it will cause dependency problems. There is information on how to submit

- multi-package updates in the

- link:https://fedoraproject.org/wiki/Package_update_HOWTO#Updating_inter-dependent_packages[package update HOWTO].

- 

- [[taskotron]]

- == Taskotron

- 

- The Taskotron automated testing system runs two tests against all updates submitted to Bodhi: a

- dependency check (to ensure all dependencies of the package(s) can be fulfilled from the

- repositories) and an upgrade path check (to ensure the update does not break the upgrade path from

- release to release, i.e. that the update does not contain a package versioned higher than the newest

- version of the same package available in a later Fedora release). Taskotron adds comments to Bodhi

- with the results of these test runs. Failing either test will disable

- link:https://fedoraproject.org/wiki/Bodhi#Karma_Automatism[Bodhi karma automatism] for

- the update (if it was enabled). If you submit an update that fails one of these

- tests, please check the result and fix the problem. If you are sure there is an error in the

- test rather than an error in your package, you may still manually push the update or re-enable karma

- automatism.

- 

- Previously, the old AutoQA system ran similar checks to Taskotron.

- The scope of these automated checks is expected to expand in future.

- The link:https://fedoraproject.org/wiki/QA:Package_Update_Acceptance_Test_Plan[Package Update Acceptance Test Plan]

- was written as the original plan for AutoQA update validation,

- and may still be seen as a framework for future work on Taskotron.

- 

- [[bodhi-use]]

- == Bodhi Use

- 

- Bodhi is meant as the method for getting updates into an existing release or pre-release. It does

- provide a testing mechanism to make sure that nothing unforeseen will arise, but the expectation is

- that packages submitted are likely ready to ship as an update. As such, any packages submitted to

- Bodhi must be done with the intent that the package submitted is ready for general consumption. The

- users kind enough to test these packages have come to expect this, and doing anything else moves

- against their good will, and is likely to drive testers away. Bodhi and updates-testing are not a

- place for experimentation or advanced notification of potentially disruptive updates. These should

- be handled with packages in link:https://copr.fedorainfracloud.org/[Copr] or another public repository

- with a message to call for testing on the appropriate mailing lists.

+ [[fedora-ci]]

+ link:https://docs.fedoraproject.org/en-US/ci/[Fedora CI] runs tests on all Bodhi updates. If package

+ maintainers have marked the tests as blocking, the Bodhi update will be blocked

+ from going stable.

This is a continuation of #40. I incorporated @mattia's graph, but removed some text from it, to avoid too much detail.
I also tried to incorporate @decathorpe's description of update rules. I reworded things a bit. I hope I got the details right, but please triple-check.
I tried to address all the other comments on #40 (including my own ;]]]]).
See the first commit for a detailed description of changes.

Q: where is bodhi config? https://github.com/fedora-infra/bodhi/blob/develop/bodhi/server/config.py#L321 has critpath.min_karma, but no non-critpath.min_karma, afaics.

Q: where is bodhi config? https://github.com/fedora-infra/bodhi/blob/develop/bodhi/server/config.py#L321 has critpath.min_karma, but no non-critpath.min_karma, afaics.

non-critpath updates have no min karma at all, only a min amounts of days spent in testing.
The "min karma" for those updates is just set for auto-pushing the update to stable by the update submitter:
- an update is submitted with auto-push not enabled: after the min amount of days spent in testing the submitter can manually push the update to stable (or the update will stay in testing forever)
- an update is submitted with auto-push enabled and min karma set to 1: as soon as the update gets 1 positive karma it will be pushed to stable (even before the min days in testing)

non-critpath updates have no min karma at all, only a min amounts of days spent in testing.

Hmm, I don't understand. The rules were always described as "karma OR time", and I think that it correct, since we have had cases where updates got a lot of karma and were immediately pushed to stable without any testing period at all, e.g. firefox. So if it's OR, it means that if the update has had "min amount of days spent in testing", the karma minimum is of no relevance.

So... maybe propose a patch to the proposed text, if that text is wrong somewhere. This might be easier than trying to explain in comments.

For regular packages, if the update gets 2 karma, bodhi says (something like): "The maintainer may push the update to stable now." (even when the autokarma option is set to e.g. 5).

(even when the autokarma option is set to e.g. 5).

IIUC, the autokarma option is only relevant for autopushes. The maintainer is not bound by the setting. This makes sense, since they are free to set the value, so it would be strange to force them to reconfigure the setting down if they want to push the update manually.

For regular packages, if the update gets 2 karma, bodhi says (something like): "The maintainer may push the update to stable now."

That is strange. From all that I understood, it should say that after +1 karma, because that's the minimum autokarma for non-critpath packages.

But looking at some actual updates, I'm confused by what bodhi is doing:
https://bodhi.fedoraproject.org/updates/FEDORA-2020-2255b438a2
2020-09-02 11:07:06 +0000 (GMT): update created, CRITPATH
2020-09-02 11:23:47.645513 (karma: +1)
2020-09-02 16:21:02.927352 (karma: 0)
This update has been pushed to testing.
2020-09-02 16:21:30.283736 (karma: 0)
This update can be pushed to stable now if the maintainer wishes

This is a critpath update, so I'd expect +2 karma to be required, not just +1. But it seems it had F33BetaFreezeException, so maybe it was allowed because of that. If yes, I should mention that in the description.

https://bodhi.fedoraproject.org/updates/FEDORA-2021-3cd10a2ed3
Bodhi says "This update can be pushed to stable now if the maintainer wishes" after +2 karma and ~20h in updates-testing.

https://bodhi.fedoraproject.org/updates/FEDORA-2020-8481bac195
Bodhi says "This update can be pushed to stable now if the maintainer wishes" after +2 karma and ~4 days in updates-testing.

So I think minimum autokarma is +2 for stable releases. I based my text on https://pagure.io/fesco/fesco-docs/pull-request/40#comment-136587, but I think the old description got the value right as +2.

But I'm still quite confused about the systemd case.

But I'm still quite confused about the systemd case.

OK, I think I understand: the old docs had

Pre Beta
All critical path updates MUST either get one +1 karma, OR spend at least 14 days in updates-testing and have no negative karma before being moved to stable.

1 new commit added

  • Rework the minimum karma thresholds
3 years ago

Also note that the critpath packages have not been updated for a very long time. Hence e.g. python3 is critpath, but python3.9 is not.

non-critpath updates have no min karma at all, only a min amounts of days spent in testing.

Hmm, I don't understand. The rules were always described as "karma OR time", and I think that it correct, since we have had cases where updates got a lot of karma and were immediately pushed to stable without any testing period at all, e.g. firefox. So if it's OR, it means that if the update has had "min amount of days spent in testing", the karma minimum is of no relevance.

So... maybe propose a patch to the proposed text, if that text is wrong somewhere. This might be easier than trying to explain in comments.

That's exactly what I meant, but perhaps I was not clear.
Looking at Bodhi code, this is what happens:
non-critpath updates

  • autopush not enabled:
    • the update can be manually pushed to stable after mandatory_days_in_testing OR when the overall karma reaches critpath.min_karma
  • autopush enabled:
    • manual push as above
    • the update is autopushed to stable after stable days set by user (which is always >= mandatory_days_in_testing) OR when the overall karma reaches stable_karma threshold set by user (which can be as low as 1) - whatever condition is reached first.

critpath updates
Should work as normal updates, except that:

  • if the update gets any negative karma, autopush is disabled
  • the stable karma threshold can't be lower than critpath.min_karma

So, if the user sets stable_karma > critpath.min_karma or stable days > mandatory_days_in_testing, the update gets the comment This update can be pushed to stable now if the maintainer wishes. Usually that happens on +2 karma because Bodhi default for stable_karma is +3 and critpath.min_karma for stable Fedora releases is +2.

But that doesn't mean that a +2 karma is mandatory because as I wrote above the user can set stable_karma to +1, while for crtipath updates that's not possible.

  • autopush not enabled:
  • the update can be manually pushed to stable after mandatory_days_in_testing OR when the overall karma reaches critpath.min_karma
  • autopush enabled:
  • manual push as above
  • the update is autopushed to stable after stable days set by user (which is always >= mandatory_days_in_testing) OR when the overall karma reaches stable_karma threshold set by user (which can be as low as 1) - whatever condition is reached first.
    ...
    critpath.min_karma for stable Fedora releases is +2.

Looking just at the karma conditions, do I have this right that there might be a condition where the update will be pushed automatically by bodhi (if enabled), but the maintainer is now allowed to push it manually? Based on the description, that would be the case when stable_karma == 1 and critpath.min_karma == 2 and karma == +1.

if the update gets any negative karma, autopush is disabled

Does this only apply to critpath updates?

  • autopush not enabled:
  • the update can be manually pushed to stable after mandatory_days_in_testing OR when the overall karma reaches critpath.min_karma
  • autopush enabled:
  • manual push as above
  • the update is autopushed to stable after stable days set by user (which is always >= mandatory_days_in_testing) OR when the overall karma reaches stable_karma threshold set by user (which can be as low as 1) - whatever condition is reached first.
    ...
    critpath.min_karma for stable Fedora releases is +2.

Looking just at the karma conditions, do I have this right that there might be a condition where the update will be pushed automatically by bodhi (if enabled), but the maintainer is now allowed to push it manually? Based on the description, that would be the case when stable_karma == 1 and critpath.min_karma == 2 and karma == +1.

(assuming you meant "...but the maintainer is NOT allowed...")
Yes, that's right. But it's not a problem, as Bodhi will autopush the update as soon as the update reaches +1.
I think it's more likely a problem (and I don't like at all) that a user can cheat so easily the "mandatory days in testing"... With autopush enabled and karma threshold set to +1 the update could be pushed just a few minutes after it had been submitted, if someone gives the +1. We saw a lot of problems with that in the past with Firefox package getting pushed too fast in stable, for example.
In my opinion, the autopush should happen only after the "mandatory days in testing", while karma threshold could be the threshold to unlock the update for manual push. That, at least, would require manual intervention and a conscious decision by the packager.

if the update gets any negative karma, autopush is disabled

Does this only apply to critpath updates?

Yes, if I didn't overlook Bodhi code.

(assuming you meant "...but the maintainer is NOT allowed...")

Yeah. I was writing this late in the evening, and missed that despite rereading the text a few times ;(

Yes, that's right. But it's not a problem, as Bodhi will autopush the update as soon as the update reaches +1.

To clarify: if autopush is disabled, the maintainer has to wait longer than they would have to wait if autopush was enabled?

With autopush enabled and karma threshold set to +1 the update could be pushed just a few minutes after it had been submitted, if someone gives the +1. We saw a lot of problems with that in the past with Firefox package getting pushed too fast in stable, for example.

I think it's not such a big problem in practice because it matter mostly for very popular packages, and maintainers know to increase the karma thresholds for those packages... But I agree it would be nice to add some guard so that this cannot happen. But let's keep that separate from this PR: let's first nail down the description of current rules, so everybody is clear where we stand.

if the update gets any negative karma, autopush is disabled

Does this only apply to critpath updates?

Yes, if I didn't overlook Bodhi code.

I'll push another patch to the PR to update the description.

1 new commit added

  • Any negative karma only applies to critpath updates
3 years ago

if the update gets any negative karma, autopush is disabled

This applies to all updates. See https://bodhi.fedoraproject.org/updates/FEDORA-2020-ea37731516

Bodhi is disabling automatic push to stable due to negative karma. The maintainer may push manually if they determine that the issue is not severe.

6 new commits added

  • Rework the minimum karma thresholds
  • Graphical version of the updates policy
  • Update the update policy some more
  • Adjust some more for comments/feedback
  • Changes from pr review
  • Update the updates policy document
3 years ago

OK ;) I nuked the last commit.

(assuming you meant "...but the maintainer is NOT allowed...")

Yeah. I was writing this late in the evening, and missed that despite rereading the text a few times ;(

Yes, that's right. But it's not a problem, as Bodhi will autopush the update as soon as the update reaches +1.

To clarify: if autopush is disabled, the maintainer has to wait longer than they would have to wait if autopush was enabled?

Yes, with autopush disabled, the update can be pushed to stable when the karma reaches "critpath.min_karma" (usually +2), while with autopush enabled the user can set "stable_karma" to +1 (for non-critpath updates).

if the update gets any negative karma, autopush is disabled

This applies to all updates. See https://bodhi.fedoraproject.org/updates/FEDORA-2020-ea37731516

Bodhi is disabling automatic push to stable due to negative karma. The maintainer may push manually if they determine that the issue is not severe.

Yeah, I overlooked the code.
I think the original intention was:

  • if the update (any type) gets any negative karma, disable autopush
  • if a critpath update gets any negative karma, do not let it be pushed to stable (not even manually)

but I suppose the latter doesn't currently work because of a faulty test on an if statement. I need to check.

Yes, with autopush disabled, the update can be pushed to stable when the karma reaches "critpath.min_karma" (usually +2), while with autopush enabled the user can set "stable_karma" to +1 (for non-critpath updates).

I think this is a bug. I'll update the description though, since the point is to describe current state.

7 new commits added

  • Critpath karma req applies to both types of updates
  • Rework the minimum karma thresholds
  • Graphical version of the updates policy
  • Update the update policy some more
  • Adjust some more for comments/feedback
  • Changes from pr review
  • Update the updates policy document
3 years ago

I'm a bit torn between "Rawhide" and "rawhide". On the one hand it's a proper thing so "Rawhide" makes sense. On the other hand we are changing the branch to be 'rawhide' so perhaps it makes sense to align with that?

Do we want to add some note or infor about what critical path is? https://fedoraproject.org/wiki/Critical_path_package or perhaps here is better if we don't want to point to the wiki...

Otherwise looks great to me. Thanks for moving this forward.

I'm a bit torn between "Rawhide" and "rawhide".

I'm fine with either, but I think it should be either Rawhide (proper noun with capital letter) or rawhide (the symbol, with a different font), but not just "rawhide". I'll convert and see how it looks.

I would prefer the proper noun form.

I meant to update this quicker... Anyhow:

I'm a bit torn between "Rawhide" and "rawhide"

I think Rawhide looks OK, and rawhide risks a bit too much highlighting. So I'll lean towards @ngompa here, and leave it as is, unless more people lean the other way.

Do we want to add some note or infor about what critical path is?

There is a definition with a link to the wiki a few lines down.

MUST fix?

So actually the removal of "MUST" was on purpose here. From the POV of the packager, it's not an option to violate this, because they simply cannot push updates that are not linked to a bug. (And the links to QA SOPs are provided).

In general, I removed all "MUST" for stuff that a normal packager cannot "do", and is just stuff that is enforced by the software.

So... no updates in the end.

That's getting policy and enforcement of that policy mixed up. The rules should still say "MUST", but the fact that our implementation doesn't even allow this to be skipped is orthogonal. (Plus, if we change the enforcement implementation, we will want to point to the policy describing what may and may not be done.)

1 new commit added

  • Restore "MUST fix" in description of updates during freezes
3 years ago

OK, I changed "updates fix" to "updates MUST fix" in two places (for the beta freeze and the final freeze).

If there are no other comments, I'd love to get this merged ;)

As written right now: :+1:

(also: documents like this are not carved upon stone and can be tweaked later if needed)

We at +4 (counting me). Anyone else?

Pull-Request has been merged by sgallagh

3 years ago