#8 updatetrackers: support stg, use bugzilla api key, use argparse
Merged 2 years ago by adamwill. Opened 3 years ago by kparal.

file added
+2
@@ -0,0 +1,2 @@ 

+ [flake8]

+ max-line-length = 100

file modified
+74 -38
@@ -1,4 +1,4 @@ 

- #!/bin/python3

+ #!/usr/bin/env python3

  

  # Copyright Red Hat

  #
@@ -19,7 +19,7 @@ 

  

  """Fedora release blocker / freeze exception bug tracker update script."""

  

- import getpass

+ import argparse

  import sys

  

  import bugzilla
@@ -29,17 +29,38 @@ 

  MILESTONES = ("Beta", "Final")

  TYPES = (("Blocker", "blocker"), ("FreezeException", "freeze exception"))

  

- BLOCKTMP = """Fedora {release} {milestone} blocker tracker bug: http://fedoraproject.org/wiki/BugZappers/HouseKeeping/Trackers

+ BLOCKTMP = """Fedora {release} {milestone} blocker tracker bug: \

+ https://{fpo}/wiki/BugZappers/HouseKeeping/Trackers

  

- To propose a bug as blocking the Fedora {release} {milestone} release, mark it as blocking this bug, or use the webapp: https://qa.fedoraproject.org/blockerbugs/propose_bug . It will be reviewed according to https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process . If you can, please specify the release criterion that the bug violates. See https://fedoraproject.org/wiki/Fedora_Release_Criteria for details on the release criteria.

+ To propose a bug as blocking the Fedora {release} {milestone} release, mark it as blocking this \

+ bug, or use the webapp: https://qa.{fpo}/blockerbugs/propose_bug . It will be reviewed according \

+ to https://{fpo}/wiki/QA:SOP_blocker_bug_process . If you can, please specify the release \

+ criterion that the bug violates. See https://{fpo}/wiki/Fedora_Release_Criteria for details on the \

+ release criteria.

  

- If the bug is not serious enough to block release but you believe it merits being fixed during a milestone freeze (see https://fedoraproject.org/wiki/Milestone_freezes ), you can propose it as a 'freeze exception' instead. To do this, mark it as blocking the bug F{release}{milestone}FreezeException . See https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process for the freeze exception process."""

+ If the bug is not serious enough to block release but you believe it merits being fixed during a \

+ milestone freeze (see https://{fpo}/wiki/Milestone_freezes ), you can propose it as a 'freeze \

+ exception' instead. To do this, mark it as blocking the bug F{release}{milestone}FreezeException . \

+ See https://{fpo}/wiki/QA:SOP_freeze_exception_bug_process for the freeze exception process."""

  

- FETMP = """Fedora {release} {milestone} freeze exception bug tracker: http://fedoraproject.org/wiki/BugZappers/HouseKeeping/Trackers

+ FETMP = """Fedora {release} {milestone} freeze exception bug tracker: \

+ https://{fpo}/wiki/BugZappers/HouseKeeping/Trackers

  

- To propose a bug as a freeze exception bug for the Fedora {release} {milestone} release, mark it as blocking this bug, or use the webapp: https://qa.fedoraproject.org/blockerbugs/propose_bug . It will be reviewed according to https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process .

+ To propose a bug as a freeze exception bug for the Fedora {release} {milestone} release, mark it \

+ as blocking this bug, or use the webapp: https://qa.{fpo}/blockerbugs/propose_bug . It will be \

+ reviewed according to https://{fpo}/wiki/QA:SOP_freeze_exception_bug_process .

  

- If you think the bug is sufficiently serious to block the {milestone} release, instead mark it as blocking the bug F{release}{milestone}Blocker. See https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process for the blocker process."""

+ If you think the bug is sufficiently serious to block the {milestone} release, instead mark it as \

+ blocking the bug F{release}{milestone}Blocker. See \

+ https://{fpo}/wiki/QA:SOP_blocker_bug_process for the blocker process."""

+ 

+ def get_argparser():

+     parser = argparse.ArgumentParser()

+     parser.add_argument('release', metavar='RELEASE', type=int, help='The Fedora release number '

+                         'that was finalized, e.g. 36')

+     parser.add_argument('--stg', action='store_true', help='Use the staging instance of Bugzilla '

+                         'and Wiki')

+     return parser

  

  def get_mile_type(alias):

      """Find the milestone and blocker type from an alias, we do this
@@ -79,14 +100,13 @@ 

      }

      return bzapi.query(query)

  

- def update_current(rel, bzapi):

+ def update_current(rel, bzapi, fpohost):

      """Close the trackers for the new release and remove the generic

      aliases from them.

      """

      update = bzapi.build_update(status="CLOSED", resolution="CURRENTRELEASE",

-                                 comment="Fedora {0} was released. Bug closed per "

-                                 "https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Trackers"

-                                 .format(rel))

+                                 comment=f"Fedora {rel} was released. Bug closed per "

+                                 f"https://{fpohost}/wiki/BugZappers/HouseKeeping/Trackers")

      # build_update can't handle multiple aliases, so we do this manually.

      update['alias'] = {'remove': get_aliases()}

      # we have to go one by one, as you can't do alias changes on multiple
@@ -113,7 +133,7 @@ 

                  bzapi.update_bugs([bug.id], update)

      return bugs

  

- def update_nextnext(rel, bzapi):

+ def update_nextnext(rel, bzapi, fpohost):

      """Create the trackers for the release after next."""

      # sanity check: don't re-create

      existing = get_trackers(rel, bzapi)
@@ -125,11 +145,11 @@ 

      for alias in get_aliases(rel):

          (mile, typ) = get_mile_type(alias)

          if typ == "blocker":

-             desc = BLOCKTMP.format(release=rel, milestone=mile)

-             url = "https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process"

+             desc = BLOCKTMP.format(release=rel, milestone=mile, fpo=fpohost)

+             url = f"https://{fpohost}/wiki/QA:SOP_blocker_bug_process"

          else:

-             desc = FETMP.format(release=rel, milestone=mile)

-             url = "https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process"

+             desc = FETMP.format(release=rel, milestone=mile, fpo=fpohost)

+             url = f"https://{fpohost}/wiki/QA:SOP_freeze_exception_bug_process"

  

          print("Creating {0} tracker for {1} with alias {2}".format(typ, mile, alias))

          create = bzapi.build_createbug(
@@ -149,13 +169,13 @@ 

      # as it makes sure the alias info is included

      return get_trackers(rel, bzapi)

  

- def update_wiki(wiki, rel, currbugs, nxt, nxtbugs, nxnx, nxnxbugs):

+ def update_wiki(wiki, rel, currbugs, nxt, nxtbugs, nxnx, nxnxbugs, bzhost):

      """Edit the Trackers wiki page with all the changes."""

      print("Updating wiki page...")

      page = wiki.pages["BugZappers/HouseKeeping/Trackers"]

      pgtxt = page.text()

  

-     def build_table(rel, bugs, gencol=True):

+     def build_table(rel, bugs, bzhost, gencol=True):

          """Inner function to build the table for a particular release.

          If gencol is True, the generic aliases will be included in the

          table. If it's None, there will be no column for a generic
@@ -176,15 +196,18 @@ 

              name = "Fedora {0} {1}".format(rel, mile)

              if typ == "freeze exception":

                  name += " Freeze Exception"

+             if "stage" in bzhost:

+                 # the [[rhbug]] interwiki link always points to prod bz

+                 # there is no stg equivalent

+                 bugtext = f"[https://{bzhost}/show_bug.cgi?id={bugid} {bugid}]"

+             else:

+                 bugtext = f"[[rhbug:{bugid}|{bugid}]]"

              if gencol is None:

-                 text += "| {0} || {1} || [[rhbug:{2}|{2}]]\n|-\n".format(

-                     name, alias, bugid)

+                 text += f"| {name} || {alias} || {bugtext}\n|-\n"

              elif gencol:

-                 text += "| {0} || {1} || {2} || [[rhbug:{3}|{3}]]\n|-\n".format(

-                     name, generic, alias, bugid)

+                 text += f"| {name} || {generic} || {alias} || {bugtext}\n|-\n"

              else:

-                 text += "| {0} || - || {1} || [[rhbug:{2}|{2}]]\n|-\n".format(

-                     name, alias, bugid)

+                 text += f"| {name} || - || {alias} || {bugtext}\n|-\n"

          text += "|}\n\n"

          return text

  
@@ -192,13 +215,13 @@ 

      sep = "Past Tracker Bugs ==\n\n"

      pos = pgtxt.index(sep)

      newtxt = pgtxt[:pos+len(sep)]

-     newtxt += build_table(rel, currbugs, gencol=None)

+     newtxt += build_table(rel, currbugs, bzhost, gencol=None)

      newtxt += pgtxt[pos+len(sep):]

  

      # Re-create all the 'current' tracker text

      currtxt = "== Current Tracker Bugs ==\n\n"

-     currtxt += build_table(nxt, nxtbugs, gencol=True)

-     currtxt += build_table(nxnx, nxnxbugs, gencol=False)

+     currtxt += build_table(nxt, nxtbugs, bzhost, gencol=True)

+     currtxt += build_table(nxnx, nxnxbugs, bzhost, gencol=False)

  

      # Throw away the top of the page and replace it with currtext

      pos = newtxt.index("== Policy ==")
@@ -207,26 +230,39 @@ 

  

  def run():

      """Main logic."""

-     if len(sys.argv) != 2 or not sys.argv[1].isdigit():

-         sys.exit("Usage: {0} NN, where NN is the release that was finalized".format(sys.argv[0]))

+     parser = get_argparser()

+     args = parser.parse_args()

+     bzhost = "bugzilla.redhat.com"

+     fpohost = "fedoraproject.org"

+     if args.stg:

+         bzhost = "bugzilla.stage.redhat.com"

+         fpohost = "stg.fedoraproject.org"

+         print("Will use the staging instances.")

  

-     bzapi = bugzilla.Bugzilla("bugzilla.redhat.com")

+     bzapi = bugzilla.Bugzilla(url=bzhost, tokenfile=None)

+     if not bzapi.api_key:

+         print("ERROR: You must configure a Bugzilla API key. Log in to Bugzilla in a web browser, "

+               f"create an API key, and then run `bugzilla --bugzilla {bzhost} login --api-key` to "

+               "create a python-bugzilla config file with the API key stored.", file=sys.stderr)

+         sys.exit(1)

+     if not bzapi.logged_in:

+         print("ERROR: Bugzilla login failed for some reason. An invalid API key?", file=sys.stderr)

+         sys.exit(1)

      # this solves a problem with Bug instances returned by build_createbug

      # not knowing what their aliases are...

      bzapi.bug_autorefresh = True

-     if not bzapi.logged_in:

-         bzapi.interactive_login()

-     wiki = wikitcms.wiki.Wiki(host='fedoraproject.org')

+ 

+     wiki = wikitcms.wiki.Wiki(host=fpohost)

      wiki.login()

  

-     rel = int(sys.argv[1])

+     rel = args.release

      nxt = str(rel + 1)

      nxnx = str(rel + 2)

      rel = str(rel)

-     currbugs = update_current(rel, bzapi)

+     currbugs = update_current(rel, bzapi, fpohost)

      nxtbugs = update_next(nxt, bzapi)

-     nxnxbugs = update_nextnext(nxnx, bzapi)

-     update_wiki(wiki, rel, currbugs, nxt, nxtbugs, nxnx, nxnxbugs)

+     nxnxbugs = update_nextnext(nxnx, bzapi, fpohost)

+     update_wiki(wiki, rel, currbugs, nxt, nxtbugs, nxnx, nxnxbugs, bzhost)

  

  try:

      run()

Allows to update staging bugzilla and staging wiki (and also test it there,
if needed). Bugzilla API key is now required, due to changes in upstream
Bugzilla. Argparse is used instead of manual args handling.

Fixes: https://pagure.io/fedora-qa/qa-misc/issue/7


This was already tested on stg and it worked fine.

I found an issue. The stg wiki page doesn't correctly include links to the stage bugzilla, instead they link to the production bugzilla (the numbers are for staging, just the links are invalid, so all you need need to do is to add "stage" in the middle and you get the right link). The problem is that when creating the wiki, it uses the rhbug keyword:

| Fedora 37 Beta || BetaBlocker || F37BetaBlocker || [[rhbug:1996156|1996156]]

It is not the usual template, instead it's some kind of a builtin plugin or something? I don't know how to see the source code/documentation for rhbug, whether it supports links to bugzilla.stage.redhat.com, or whether there's a similar keyword for it, etc. We can either ignore it (the staging wiki is just for us anyway), or I can make the script include a full URL (at least for staging) instead of the rhbug keyword).

why did you want to update staging anyhow? isn't staging bugzilla regularly re-synced from prod bugzilla, so it'll get the trackers that way?

also note that I found bzapi.logged_in() actually crashes if you're not logged in.

Other than those issues, this looks fine...

Why? I don't mind a change to /usr/bin/python3 but there's no real clear benefit to using env AFAIK. We do not use it in standard shebangs for Fedora packages.

why did you want to update staging anyhow? isn't staging bugzilla regularly re-synced from prod bugzilla, so it'll get the trackers that way?

Unfortunately it isn't. If "regularly" doesn't mean every few years. If I want to test BBA against stg, I need to create the trackers manually, which I did up until now and hated it. Now I realized I could use your script.

also note that I found bzapi.logged_in() actually crashes if you're not logged in.

Interesting. It works just fine for me:

$ python
Python 3.10.4 (main, Mar 25 2022, 00:00:00) [GCC 12.0.1 20220308 (Red Hat 12.0.1-0)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import bugzilla
>>> bz = bugzilla.Bugzilla(url='bugzilla.stage.redhat.com', use_creds=False)
>>> bz.api_key
>>> bz.logged_in
False
>>> 
$ pip freeze | grep bugzilla
python-bugzilla==3.2.0

I don't mind a change to /usr/bin/python3 but there's no real clear benefit to using env AFAIK. We do not use it in standard shebangs for Fedora packages.

Yes, because in standard Fedora packages you don't want to pick up a different python interpreter from the environment. Miro et al. are trying hard to avoid any external influence (a most recent example). But for scripts that you run from a development directory, being able to run them from a virtualenv makes the experience much better (./updatetrackers instead of python updatetrackers). For actual installation (if it happens, not in this case), the shebang can be replaced with a hardcoded path, or a script created through setup.py, or a Fedora macro can take care of that, etc. I think we do it somewhere.

But I don't have any strong feelings here, it's just that I'm used to use env python3 for development-related scripts and simple tools without installation, and so I changed it. I can easily continue to run this is as python updatetrackers, I don't mind.

Merge whenever you feel ready with whichever changes you want ;-)

Metadata Update from @kparal:
- Request assigned

2 years ago

I was kinda expecting a resolution to the rhbug thing. So I looked into it now. It's an interwiki link: https://meta.wikimedia.org/wiki/Help:Interwiki_linking . The list of them is at https://fedoraproject.org/wiki/Special:Interwiki . However, it's not edited via the wiki, it needs to be edited directly in a database table, apparently. We'd have to file an infra ticket. I guess the sensible thing to do would be to ask for a rhstgbug entry or something which we could use to link to staging.

Ah, rhbug, I completely forgot about it :) Thanks for the investigation. I feel bad bugging infra about such a trifle. No-one cared about the stg page until now (why should we). It's staging and for our private use anyway. If that still bothers you, I'll implement using direct links instead of rhbug.

rebased onto 3f54720

2 years ago

I updated the PR with a fix for the staging issue, plus some other tweaks. I felt like the use of global vars was kinda superfluous, we can just pass the values from run() to the necessary functions. And I'm trying to standardize on double quotes (I know it's not consistent in the rest of the file yet!) Will merge like this. Untested, let's hope it works. I guess I'll try and remember to run it on stg before prod next time. :D

Pull-Request has been merged by adamwill

2 years ago

I felt like the use of global vars was kinda superfluous, we can just pass the values from run() to the necessary functions.

It's hard to strike the balance between abstraction, readability, maintainability, encapsulation, testability and all those other complex words :-) I personally don't see a problem in using global variables as long as they are mostly read-only (except some startup initialization). They make the code simpler.

And I'm trying to standardize on double quotes

Good luck :-D When I have time, I intend to try to use some code formatter (e.g. black) in my projects and see if I can a) live with the result b) be happy to let go and completely stop thinking about formatting when writing code.

It's a bit subjective yeah, in this case I kinda preferred how this way all the discovery and passing of the variables happens in one place rather than them being declared at the top and then having to use global later on...

I use black on most of the projects I maintain now, but this repo is shared and isn't a straightforward single python project so it's a bit different. It'd be nice to write tests for each script and a unified tox config to run them all plus linting/formatting, though...