#946 frontend: dump whooshee version when updating indexes
Merged 4 years ago by praiskup. Opened 4 years ago by frostyx.
copr/ frostyx/copr dump-whooshee-version  into  master

@@ -1,7 +1,7 @@ 

  from flask_script import Command

  from flask_whooshee import Whooshee

  from coprs import app

- from coprs.whoosheers import CoprWhoosheer

+ from coprs.whoosheers import CoprWhoosheer, WhoosheeStamp

  from coprs.logic import coprs_logic

  

  
@@ -26,3 +26,5 @@ 

          for copr in coprs_logic.CoprsLogic.get_all():

              CoprWhoosheer.insert_copr(writer, copr)

          writer.commit(optimize=True)

+ 

+         WhoosheeStamp.store()

@@ -0,0 +1,14 @@ 

+ import sys

+ from flask_script import Command

+ from coprs.whoosheers import WhoosheeStamp

+ 

+ 

+ class UpdateIndexesRequiredCommand(Command):

+     """

+     Is whooshee indexes rebuild required?

+     """

+ 

+     def run(self):

+         valid = WhoosheeStamp.is_valid()

+         print("no" if valid else "yes")

+         sys.exit(int(not valid))

@@ -1,8 +1,11 @@ 

+ import os

  import whoosh

  import time

  

+ from subprocess import Popen, PIPE

  from flask_whooshee import AbstractWhoosheer

  

+ from coprs import app

  from coprs import models

  from coprs import whooshee

  from coprs import db
@@ -110,3 +113,38 @@ 

                      WHERE copr.id = {1}

                      """.format(int(time.time()), copr_id)

                  )

+ 

+ 

+ class WhoosheeStamp(object):

+     """

+     When a whooshee package is updated, it is often needed to rebuild

+     indexes. This class manages a stamp file containing whooshee packages

+     versions and decides whether they are still up-to-date or not.

+     """

+ 

+     PATH = os.path.join(app.config["WHOOSHEE_DIR"], "whooshee-version")

+ 

+     @classmethod

+     def current(cls):

+         packages = ["python3-flask-whooshee", "python3-whoosh"]

+         cmd = ["rpm", "-q", "--qf", "%{NAME}-%{VERSION}\n"] + packages

+         process = Popen(cmd, stdout=PIPE, stderr=PIPE)

+         out, err = process.communicate()

+         return out.decode("utf-8").rstrip()

+ 

+     @classmethod

+     def store(cls):

+         with open(cls.PATH, "w") as f:

+             f.write(cls.current())

+ 

+     @classmethod

+     def read(cls):

+         try:

+             with open(cls.PATH, "r") as f:

+                 return f.read().rstrip()

+         except OSError:

+             return None

+ 

+     @classmethod

+     def is_valid(cls):

+         return cls.read() == cls.current()

@@ -29,11 +29,14 @@ 

      "add_user": "AddUserCommand",

      "dump_user": "DumpUserCommand",

  

+     # Whooshee indexes

+     "update_indexes": "UpdateIndexesCommand",

+     "update_indexes_quick": "UpdateIndexesQuickCommand",

+     "update_indexes_required": "UpdateIndexesRequiredCommand",

+ 

      # Other

      "get_admins": "GetAdminsCommand",

      "fail_build": "FailBuildCommand",

-     "update_indexes": "UpdateIndexesCommand",

-     "update_indexes_quick": "UpdateIndexesQuickCommand",

      "rawhide_to_release": "RawhideToReleaseCommand",

      "backend_rawhide_to_release": "BackendRawhideToReleaseCommand",

      "update_graphs": "UpdateGraphsDataCommand",

Once we have information about what whooshee version we used for
generating indexes, then we can update our provision playbooks
to rebuild them only if the version changed.

This format should be easily readable in the playbook with

- hosts: localhost
  tasks:
    - include_vars:
        file: variable-file.json
        name: variable
    - debug: var=variable

See https://stackoverflow.com/a/50642466 . Or do you prefer something else?

Does anyone know whether it is possible to match multiple names at once? Then we wouldn't need this ugly for loop.

Do we possibly care about a version of any other package?

I'm not actually sure we care about python3-flask-whooshee.

Instead of going with python-rpm, I'd personally only compare plain output of rpm -q --qf %{NAME}-%{VERSION}'\n' python3-whoosh python3-flask-whooshee

Yeah, that's what I wanted to sigh about. The python-rpm API is so user-friendly that I would also probably prefer to just call rpm -qf via subprocess.

Metadata Update from @praiskup:
- Pull-request tagged with: needs-work

4 years ago

1 new commit added

  • frontend: dump whooshee version using subprocess
4 years ago

Instead of going with python-rpm, I'd personally only compare plain output of rpm -q --qf %{NAME}-%{VERSION}'\n' python3-whoosh python3-flask-whooshee

I've updated the PR to do it this way. PTAL.

I will squash those commits into one before merging. Until then I kept it separated, so we can easily pick which solution we like best.

1 new commit added

  • frontend: add a command saying if whooshee indexes rebuild is required
4 years ago

3 new commits added

  • frontend: add a command saying if whooshee indexes rebuild is required
  • frontend: dump whooshee version using subprocess
  • frontend: dump whooshee version when updating indexes
4 years ago

We had an offlist discussion with @praiskup, that it would make things easier if we had a command deciding whether an indexes rebuild is required (instead of putting that logic into playbooks) and then do the playbook condition this way
https://docs.ansible.com/ansible/latest/user_guide/playbooks_error_handling.html#overriding-the-changed-result

I've updated the PR to reflect this. PTAL.
(Those commits needs to be squashed before a merge)

Hmm, thinking about it, we run

/usr/share/copr/coprs_frontend/manage.py update_indexes_quick 120

in cron @hourly. But I guess we probably don't want to store a that whooshee version stamp with this command, right? Because in that case, it will appear, that indexes rebuild is not required even though only indexes from the last 120 minutes have been rebuild, not the older ones.

in cron @hourly. But I guess we probably don't want to store a that whooshee version stamp with this command, right?

Good catch.

1 new commit added

  • frontend: don't store timestamp for quick update
4 years ago

Don't you want to adjust exit status accordingly? E.g. 0 if "is required" answer is True, 1 if not.

I like this approach, this is easy to employ by playbooks. I originally thought that we'll use only the update_indexes command and adjust it's exit status to:

0 => success
1 => fail
2 => full index not needed

But this approach works as well.

Edit: But if we had only one command -- we'd need something like --force
option probably (we might want to reindex sometimes manually, e.g. if the
cron gets stucked)
Edit 2: So I mean, your aproach is completely equivalent. Don't change it :-)

rebased onto 03a9f052de0589590eabe2d55cd8fdc81731a901

4 years ago

Don't you want to adjust exit status accordingly? E.g. 0 if "is required" answer is True, 1 if not.

Oh god, I did it exactly the other way. It makes more sense to me though.

Is the indexes update required? Yes! (RC 1)
Is the indexes update required? No! (RC 0)

Or in the meaning of non-zero status code signalizing errors.

Is the indexes update required? Nah, everything is alright (RC 0)
Is the indexes update required? Yep, it otherwise might not work (RC 1)

I've also squashed those commits together and removing the needs-work tag.

Metadata Update from @frostyx:
- Pull-request untagged with: needs-work

4 years ago

I don't really mind so keep this :-) just from commandline user POV, I often do:
selinuxenabled && do_something
(aka 0 when the answer is yes, or "success")

So I'd expect something like:
index_required && do_something

If you reverse the semantics, I'd expect the command to be named like:
index_ok || do_something

But anyways, the exit status is useful thing no matter the semantics you define.

+1 (but the CI badge should be green)

Ah, the CI problem is fixed by 0937dc6. Can you rebase on top of the commit? (I can't do it https://pagure.io/pagure/issue/4596)

rebased onto 8e83f66

4 years ago

Commit fc6855f fixes this pull-request

Pull-Request has been merged by praiskup

4 years ago