From 28de7c7b3ab00ccf417323b6045576bc0ca01f33 Mon Sep 17 00:00:00 2001 From: Kamil Páral Date: Sep 03 2021 08:28:37 +0000 Subject: rework the Update model (destructive changes) Background: Initially I found out that Bodhi updates are not displayed in BBA for Rawhide. While working on a fix, I realized that the whole Update model is ... slightly weird (esp. the `status` and `pending` fields didn't reflect the Bodhi structures at all and were very confusing), and the fix would result in an increasingly hard-to-manage pile of hacks. So I instead reworked the whole model, trimmed unneeded stuff, made it consistent with Bodhi structures, added safeguards and docstrings. It had rippling effects in controller and utils functions, most had to be touched or rewritten. I decided to drop all updates from the DB during this migration, otherwise the upgrade code would get even more complex, and we need none of this stuff for historical purposes anyway. High-level changes: * `Update` now better reflects the upstream Bodhi structure, unnecessary values are removed * `sync-updates` no longer performs a useless sync for an active release with all its milestones inactive (a common configuration mistake) * Bodhi updates for Rawhide are now visible * The Updates view now shows all relevant updates in a single table * Test coverage is improved * All synced updates are dropped during migration and need to be synced again Models and tables: * add `Update.updateid` to uniquely identify an Update (instead of `title`). `updateid` is required to be present and has a uniqueness guarantee. * change `Update.status` to have the same definition as in Bodhi structs (incl. `pending` value). Change it to Enum and support just selected values which we care about (anything else shouldn't be stored in DB). * drop `Update.pending`, its logic was confusing and not matching Bodhi structs * change `Update.request` also to Enum (support all possible values) * drop `Update.date_*` except for `date_submitted`. The other values are not important, we don't need to store them. * drop `Update.milestones`. It was a relationship which was re-calculated on each Update update, it was slightly confusing (Bodhi updates are not related to milestones) and served only as a way to simplify certain queries. Because there are just a couple of those queries, I opted for model simplification and readjusted those queries instead. This also means "update_milestones" table got dropped. * make many Update fields `nullable=False` to ensure data consistency * make some Update constructor values optional. Esp. nullable columns are not necessary to provide. Reorder the call params order and also the attribute order in the class, roughly by importance, optional ones last. * add a primary key over both columns in "update_fixes" table, that ensures we don't get a duplicate entry * add `Bug.is_proposed_accepted`, which allows to massively simplify certain queries Controllers: * rewrite affected `controllers/` functions with the new Update model in mind * rename some controller functions to better indicate their purpose * instead of `get_milestone_pending_stable_updates()` and `get_milestone_updates_testing()`, which seemed to be providing little value in this separation, create `get_milestone_updates()` which lists all relevant updates and display that in the Update view * remove a never used `display_updates_need_testing()` controller function Views/templates: * changed template filter `updatelabel()` into `updatestatus()`, which displays Update's current status and request in a pretty way (same as in Bodhi). Used it then across all related views (bug list, updates). * in a similar fashion, display `updateid` as a fallback when `title` is not defined for an Update in all related views. * in bug list view, indicate if there are multiple related updates (and not just one) * in update list view, show a list of all "relevant" updates, instead of splitting it into two categories (which don't even cover everything important). For each update, show components for each bug tracked, not just the first one. * in blocker list view, swap Updates and Review columns to make the columns look better aligned * fixed a bug in `updatetype()` which was not aware of accepted0day/prevrel blockers Utils: * rewrite `update_sync.py`. `extract_information()` is extremely simplified due to model changes. `get_update()` is removed because nobody used it. `search_updates()` now searches in containers as well, only searches for updates with a supported `status`, and has better error checking. `get_release_bugs()` is fixed to not return rejected bugs. `sync_updates()` and `sync_bug_updates()` and merged into `sync_updates()` and has better error checking. Merges: https://pagure.io/fedora-qa/blockerbugs/pull-request/203 --- diff --git a/alembic/versions/8ec130a51c2b_rework_update_model.py b/alembic/versions/8ec130a51c2b_rework_update_model.py new file mode 100644 index 0000000..b736a73 --- /dev/null +++ b/alembic/versions/8ec130a51c2b_rework_update_model.py @@ -0,0 +1,87 @@ +"""Rework Update model + +Revision ID: 8ec130a51c2b +Revises: bc58ec7c7d31 +Create Date: 2021-07-23 09:20:33.885039 + +""" +from alembic import op +import sqlalchemy as sa +from sqlalchemy.dialects import postgresql + +# revision identifiers, used by Alembic. +revision = '8ec130a51c2b' +down_revision = 'bc58ec7c7d31' + + +def upgrade(): + # Delete all rows in the 'update' table - some of the changes are hard to perform on existing + # data, and there is nothing important, everything will get synced again from Bodhi. + # But start with the 'update_fixes' table, because it has a foreign key to 'updates' + op.execute('DELETE FROM "update_fixes"') + # Now completely drop 'update_milestones' table, there's also a foreign key and we want to + # remove this table anyway. + op.execute('DROP TABLE "update_milestones"') + # There can still be a historical table 'used_updates', which was removed from code at + # 9d0bbd0650c, but not from the actual DB. + op.execute('DROP TABLE IF EXISTS "used_updates"') + # Now we can finally delete everything from the 'update' table + op.execute('DELETE FROM "update"') + + # Alembic can't create an enum type for Postgres automatically when adding a new column + status_enum = postgresql.ENUM('stable', 'pending', 'testing', name='update_status_enum') + status_enum.create(op.get_bind()) + request_enum = postgresql.ENUM('revoke', 'unpush', 'obsolete', 'stable', 'testing', + name='update_request_enum') + request_enum.create(op.get_bind()) + + # ### commands auto generated by Alembic - please adjust! ### + op.add_column('update', sa.Column('updateid', sa.Text(), nullable=False)) + op.alter_column('update', 'url', existing_type=sa.TEXT(), nullable=False) + op.alter_column('update', 'karma', existing_type=sa.INTEGER(), nullable=False) + op.alter_column('update', 'date_submitted', existing_type=postgresql.TIMESTAMP(), + nullable=False) + op.alter_column('update', 'release_id', existing_type=sa.INTEGER(), nullable=False) + op.create_unique_constraint(op.f('uq_update_updateid'), 'update', ['updateid']) + op.drop_column('update', 'pending') + op.drop_column('update', 'date_pushed_testing') + op.drop_column('update', 'date_pushed_stable') + op.drop_column('update', 'date_obsoleted') + # ### end Alembic commands ### + + # These changes are not autodetected by Alembic + op.alter_column('update', 'status', existing_type=sa.VARCHAR(length=80), nullable=False, + type_=status_enum, postgresql_using='status::update_status_enum') + op.alter_column('update', 'request', existing_type=sa.VARCHAR(length=80), + type_=request_enum, postgresql_using='request::update_request_enum') + op.create_primary_key('pk_update_fixes', 'update_fixes', ['bug_id', 'update_id']) + + +def downgrade(): + op.drop_constraint('pk_update_fixes', 'update_fixes', type_='primary') + op.alter_column('update', 'request', type_=sa.VARCHAR(length=80)) + op.alter_column('update', 'status', type_=sa.VARCHAR(length=80), nullable=True) + + # ### commands auto generated by Alembic - please adjust! ### + op.add_column('update', sa.Column('date_obsoleted', postgresql.TIMESTAMP(), autoincrement=False, nullable=True)) + op.add_column('update', sa.Column('date_pushed_stable', postgresql.TIMESTAMP(), autoincrement=False, nullable=True)) + op.add_column('update', sa.Column('date_pushed_testing', postgresql.TIMESTAMP(), autoincrement=False, nullable=True)) + op.add_column('update', sa.Column('pending', sa.BOOLEAN(), autoincrement=False, nullable=True)) + op.drop_constraint(op.f('uq_update_updateid'), 'update', type_='unique') + op.alter_column('update', 'release_id', existing_type=sa.INTEGER(), nullable=True) + op.alter_column('update', 'date_submitted', existing_type=postgresql.TIMESTAMP(), nullable=True) + op.alter_column('update', 'karma', existing_type=sa.INTEGER(), nullable=True) + op.alter_column('update', 'url', existing_type=sa.TEXT(), nullable=True) + op.drop_column('update', 'updateid') + op.create_table('update_milestones', + sa.Column('milestone_id', sa.INTEGER(), autoincrement=False, nullable=True), + sa.Column('update_id', sa.INTEGER(), autoincrement=False, nullable=True), + sa.ForeignKeyConstraint(['milestone_id'], ['milestone.id'], + name='fk_update_milestones_milestone_id_milestone'), + sa.ForeignKeyConstraint(['update_id'], ['update.id'], + name='fk_update_milestones_update_id_update') + ) + # ### end Alembic commands ### + + op.execute("DROP TYPE update_status_enum") + op.execute("DROP TYPE update_request_enum") diff --git a/blockerbugs/__init__.py b/blockerbugs/__init__.py index cf4df46..2f30e3a 100644 --- a/blockerbugs/__init__.py +++ b/blockerbugs/__init__.py @@ -2,6 +2,7 @@ import logging.handlers import os +from typing import Optional from flask import Flask, render_template from flask_sqlalchemy import SQLAlchemy @@ -126,6 +127,11 @@ if app.config["BEHIND_PROXY"]: app.wsgi_app = ProxyFix(app.wsgi_app, x_host=1) # type: ignore[assignment] +# === Extra imports === +# These need to be imported after all the basic setup above is done +import blockerbugs.models.update as model_update # noqa: E402 + + # === Flask views and stuff === @app.template_filter('tagify') def tagify(value): @@ -152,47 +158,58 @@ def getsubname(value): @app.template_filter('updatetype') -def updatetype(update): +def updatetype(update: Optional[model_update.Update]) -> str: + """Inspect bugs linked by this update, and return a string 'Blocker', 'FreezeException' or + 'Prioritized' (in this priority order) if there's a at least one bug proposed as such. + """ + if not update: + return '' + is_fe = False + is_prio = False for bug in update.bugs: - if bug.proposed_blocker or bug.accepted_blocker: - return 'blocker' + if (bug.proposed_blocker or + bug.accepted_blocker or + bug.accepted_0day or + bug.accepted_prevrel): + return 'Blocker' elif bug.proposed_fe or bug.accepted_fe: is_fe = True + elif bug.prioritized: + is_prio = True if is_fe: - return 'FE' - - -@app.template_filter('updatelabel') -def updatelabel(bug): - from .models.update import Update - label = [] - lowest_status_update = bug.updates.filter( - Update.status != 'obsolete', - Update.status != 'deleted', - Update.status != 'unpushed' - ).first() - if lowest_status_update: - if lowest_status_update.status == 'stable': - if bug.status in ['MODIFIED', 'ON_QA', 'VERIFIED', 'CLOSED']: - label.append('') - else: - label.append('') - if lowest_status_update.pending: - label.append('pending ') - label.append('stable') - - elif lowest_status_update.status == 'testing': - label.append('') - if lowest_status_update.pending: - label.append('pending ') - label.append('testing') - - return ''.join(label) + return 'FreezeException' + if is_prio: + return 'Prioritized' + + assert False, f"{update} doesn't seem to fix none of blocker/FE/prioritized" + + +@app.template_filter('updatestatus') +def updatestatus(update: Optional[model_update.Update]) -> str: + """Create a status description for an Update, regarding its current status and request. For + example 'testing' or 'testing -> stable' or 'pending -> testing'. Enclose in HTML with CSS + classes. + """ + if not update: + return '' + + text = f'{update.status}' + if update.request: + text += f' {update.request}' + + if update.status == 'stable': + css_class = 'badge badge-success' + elif update.status == 'testing': + css_class = 'badge badge-warning' + else: + css_class = 'badge badge-info' + + return f'{text}' @app.template_filter('datetime') -def datetime_format(value, format='%Y-%m-%d %H:%M:%S UTC'): +def datetime_format(value, format='%Y-%m-%d %H:%M UTC'): if value is not None: return value.strftime(format) return '' diff --git a/blockerbugs/controllers/api/api.py b/blockerbugs/controllers/api/api.py index 6ee1970..a5a2626 100644 --- a/blockerbugs/controllers/api/api.py +++ b/blockerbugs/controllers/api/api.py @@ -20,9 +20,12 @@ """RESTful API handlers""" +from typing import Any + from flask import Blueprint, request -from blockerbugs import app, db +from blockerbugs import app +from blockerbugs.controllers.main import update_to_milestones from blockerbugs.models.milestone import Milestone from blockerbugs.models.update import Update from blockerbugs.models.release import Release @@ -47,15 +50,17 @@ def api_error_handler(e): return JsonResponse(e.to_dict(), e.http_status_code) -def get_update_info(update): - update_simple_fields = ['title', 'url', 'karma', 'stable_karma', 'status', - 'pending'] +def get_update_info(update: Update) -> dict[str, Any]: + """Create a per-update response dictionary to be used in ``list_updates()``. + """ + update_simple_fields = ['updateid', 'title', 'url', 'karma', 'stable_karma', 'status', + 'request'] update_data = dict( (attr, getattr(update, attr)) for attr in update_simple_fields) update_data['release'] = update.release.number update_data['milestones'] = [{'version': m.version, - 'release': m.release.number, } - for m in update.milestones] + 'release': m.release.number if m.release else -1, } + for m in update_to_milestones(update)] update_data['bugs'] = [ {'bugid': bug.bugid, 'type': [tp for tp in ACCEPTED_BUGTYPES if getattr(bug, tp)]} @@ -70,22 +75,31 @@ def get_bug_info(bug): bug_info['type'] = [tp for tp in ACCEPTED_BUGTYPES if getattr(bug, tp)] return bug_info + @api_v0.route('/milestones///updates') -def list_updates(rel_num, milestone_version): +def list_updates(rel_num: int, milestone_version: str) -> JsonResponse: + """List all Updates which claim to fix a tracked bug created under the specified ``milestone``. + """ release = get_or_404(Release, number=rel_num) milestone = get_or_404(Milestone, release=release, version=milestone_version) - m_alias = db.aliased(Milestone) - updates = Update.query.join(m_alias, Update.milestones).filter( - m_alias.id == milestone.id) + + updates = Update.query.filter_by( + release=milestone.release, + ).join(Update.bugs).filter( + Bug.milestone == milestone, + Bug.active == True, # noqa: E712 + ) + if 'bugtype' in request.args: bugtype = request.args['bugtype'] if bugtype in ACCEPTED_BUGTYPES: bug_attr = getattr(Bug, bugtype) - updates = updates.filter(Update.bugs.any(bug_attr == True)) + updates = updates.filter(bug_attr == True) # noqa: E712 else: raise errors.InvalidArgumentError(arg_name='bugtype') - updates = updates.order_by(Update.date_submitted.desc()).all() + + updates = updates.order_by(Update.date_submitted.desc()).all() # type: ignore[attr-defined] updates_info = [get_update_info(up) for up in updates] return JsonResponse(updates_info) diff --git a/blockerbugs/controllers/main.py b/blockerbugs/controllers/main.py index 98ad6a7..106fbff 100644 --- a/blockerbugs/controllers/main.py +++ b/blockerbugs/controllers/main.py @@ -21,7 +21,7 @@ from flask import Blueprint, render_template, redirect, url_for, abort, g, flash, make_response, request import datetime -from sqlalchemy import func, desc, or_, and_ +from sqlalchemy import func, desc, or_ import bugzilla from flask_fas_openid import fas_login_required import json @@ -88,76 +88,67 @@ def get_milestone_bugs(milestone): return bugz -def get_milestone_pending_stable_updates(milestone): +def get_milestone_updates(milestone: Milestone) -> list[Update]: + """Get all Updates which claim to fix some blocker/FE/prioritized bug which is proposed or + accepted against the specified ``milestone``. """ - return list of updates associated with milestone, - which are 'pending stable' in bodhi - and are marked as fixing any non-rejected blocker/fe bug - """ - m_alias = db.aliased(Milestone) - updates = Update.query.join(m_alias, Update.milestones).filter( - m_alias.id == milestone.id, - Update.status == u'stable', Update.pending == True, - Update.bugs.any( - or_(Bug.accepted_fe == True, Bug.accepted_blocker == True, - Bug.accepted_0day == True, Bug.accepted_prevrel == True, - Bug.proposed_fe == True, Bug.proposed_blocker == True))).all() + updates = Update.query.filter_by( + release=milestone.release, + ).join(Update.bugs).filter( + Bug.milestone == milestone, + Bug.active == True, # noqa: E712 + Bug.is_proposed_accepted == True, + ).all() + return updates -def get_milestone_updates_testing(milestone): +def update_to_milestones(update: Update) -> list[Milestone]: + """Determine which milestones this update is relevant to. That is computed from update's + release, and milestones of bugs which this update fixes. """ - return list of updates associated with milestone, - which are 'updates testing' in bodhi (not updates testing pending) - and are marked as fixing any non-rejected blocker/fe bug - """ - m_alias = db.aliased(Milestone) - updates = Update.query.join(m_alias, Update.milestones).filter( - m_alias.id == milestone.id, Update.status == 'testing', - Update.karma < Update.stable_karma, - or_(Update.pending != True, Update.pending == None), - Update.bugs.any( - or_(Bug.accepted_fe == True, Bug.accepted_blocker == True, - Bug.accepted_0day == True, Bug.accepted_prevrel == True, - Bug.proposed_fe == True, Bug.proposed_blocker == True))).all() - return updates + milestones = Milestone.query.filter_by( + release=update.release, + ).join(Milestone.bugs).join(Bug.updates).filter( + Update.id == update.id, + ).all() -def get_milestone_all_nonstable_blocker_fixes(milestone): - """ - return list of all non-stable, non-obsolete updates which are - marked as fixing any accepted blocker bug for milestone + return milestones + + +def get_updates_nonstable_blockers(milestone: Milestone) -> list[Update]: + """Get a list of Updates which are not yet 'stable' in Bodhi, and they claim to fix some blocker + bug which is accepted as an 'AcceptedBlocker' or 'Accepted0Day' (but not + 'AcceptedPreviousRelease') against the specified ``milestone``. + + This is useful when creating requests for freeze pushes or new candidate composes. """ - m_alias = db.aliased(Milestone) - updates = Update.query.join(m_alias, Update.milestones).filter( - m_alias.id == milestone.id, ~Update.status.in_(['obsolete', 'deleted', 'unpushed']), - or_(Update.status != 'stable', Update.pending == True), - Update.bugs.any( - and_( - Bug.milestone_id == milestone.id, - or_( - Bug.accepted_blocker == True, Bug.accepted_0day == True - ) - ) - ) - ).all() + updates = Update.query.filter( + Update.release == milestone.release, + Update.status != 'stable', + ).join(Update.bugs).filter( + Bug.milestone == milestone, + or_(Bug.accepted_blocker == True, # noqa: E712 + Bug.accepted_0day == True), + ).all() + return updates -def get_milestone_all_nonstable_fe_fixes(milestone): - """ - return list of all non-stable, non-obsolete updates which are - marked as fixing any accepted FE bug for milestone + +def get_updates_nonstable_FEs(milestone: Milestone) -> list[Update]: + """Get a list of Updates which are not yet 'stable' in Bodhi, and they claim to fix some FE bug + which is accepted against the specified ``milestone``. + + This is useful when creating requests for freeze pushes or new candidate composes. """ - m_alias = db.aliased(Milestone) - updates = Update.query.join(m_alias, Update.milestones).filter( - m_alias.id == milestone.id, ~Update.status.in_(['obsolete', 'deleted', 'unpushed']), - or_(Update.status != 'stable', Update.pending == True), - Update.bugs.any( - and_( - Bug.milestone_id == milestone.id, - Bug.accepted_fe == True - ) - ) - ).all() + updates = Update.query.filter( + Update.release == milestone.release, + Update.status != 'stable', + ).join(Update.bugs).filter( + Bug.milestone == milestone, + Bug.accepted_fe == True, # noqa: E712 + ).all() + return updates @@ -312,16 +303,14 @@ def display_buglist(num, release_name): @main.route('/bug//updates') -def display_bug_updates(bugid): +def display_bug_updates(bugid: int) -> str: + """Return HTML with all Bodhi updates related to a certain Bugzilla ticket. + """ bug = Bug.query.filter_by(bugid=bugid).first() if not bug: abort(404) packagename = bug.component - updates = bug.updates.filter( - Update.status != 'obsolete', - Update.status != 'deleted', - Update.status != 'unpushed' - ).all() + updates = bug.updates.all() return render_template('bug_tooltip.html', packagename=packagename, updates=updates, bz_url=app.config['BUGZILLA_URL']) @@ -349,21 +338,21 @@ def display_irc_blockers(num, release_name): return response -@main.route('/milestone///updates') -def display_release_updates(num, release_name): - release = Release.query.filter_by(number=num).first() - milestone = Milestone.query.filter_by(release=release, version=release_name).first() +@main.route('/milestone///updates') +def display_release_updates(release_num: int, milestone_version: str) -> str: + """Render a template showing important updates for the selected milestone. + """ + release = Release.query.filter_by(number=release_num).first() + milestone = Milestone.query.filter_by(release=release, version=milestone_version).first() if not milestone: abort(404) milestone_info = get_milestone_info(milestone) - updates = { - 'Non Stable Updates': get_milestone_pending_stable_updates(milestone), - 'Updates Needing Testing': get_milestone_updates_testing(milestone), - } + updates = get_milestone_updates(milestone) return render_template('update_list.html', - info=milestone_info, updates=updates, - title="Fedora %s %s Blocker Bug Updates" % (milestone_info['number'], milestone_info['phase'])) + milestone=milestone, + title="Fedora %s %s Blocker Bug Updates" % (milestone_info['number'], + milestone_info['phase'])) @main.route('/milestone///requests') @@ -372,8 +361,8 @@ def display_release_requests(num, release_name): milestone = Milestone.query.filter_by(release=release, version=release_name).first() if not milestone: abort(404) - blocker_updates = get_milestone_all_nonstable_blocker_fixes(milestone) - fe_updates = get_milestone_all_nonstable_fe_fixes(milestone) + blocker_updates = get_updates_nonstable_blockers(milestone) + fe_updates = get_updates_nonstable_FEs(milestone) # if an update fixes both blockers and FEs, drop it from FE list fe_updates = [fe for fe in fe_updates if fe not in blocker_updates] # highlight accepted bugs which have some dependencies @@ -388,19 +377,6 @@ def display_release_requests(num, release_name): return response -@main.route('/milestone///need_testing') -def display_updates_need_testing(num, milestone_name): - release = Release.query.filter_by(number=num).first() - milestone = Milestone.query.filter_by(release=release, version=milestone_name).first() - if not milestone: - abort(404) - milestone_info = get_milestone_info(milestone) - updates = get_milestone_updates_testing(milestone) - return render_template('update_need_testing.html', updates=updates, - info=milestone_info, - title="Fedora %s %s Blocker Bug Updates" % (milestone_info['number'], milestone_info['phase'])) - - @main.route('/milestone///info') def display_milestone_info(num, milestone_name): release = Release.query.filter_by(number=num).first() diff --git a/blockerbugs/models/__init__.py b/blockerbugs/models/__init__.py index 91aa93a..a62d512 100644 --- a/blockerbugs/models/__init__.py +++ b/blockerbugs/models/__init__.py @@ -27,14 +27,8 @@ from blockerbugs import db update_fixes: Any = db.Table( 'update_fixes', - db.Column('bug_id', db.Integer, db.ForeignKey('bug.id')), - db.Column('update_id', db.Integer, db.ForeignKey('update.id')) -) - -update_milestones: Any = db.Table( - 'update_milestones', - db.Column('milestone_id', db.Integer, db.ForeignKey('milestone.id')), - db.Column('update_id', db.Integer, db.ForeignKey('update.id')) + db.Column('update_id', db.Integer, db.ForeignKey('update.id'), primary_key=True), + db.Column('bug_id', db.Integer, db.ForeignKey('bug.id'), primary_key=True), ) diff --git a/blockerbugs/models/bug.py b/blockerbugs/models/bug.py index 9a9d95e..a3fc36f 100644 --- a/blockerbugs/models/bug.py +++ b/blockerbugs/models/bug.py @@ -23,6 +23,9 @@ import datetime import json from typing import Any, Optional +from sqlalchemy import or_ +from sqlalchemy.ext.hybrid import hybrid_property + from blockerbugs import db, BaseModel, models import blockerbugs.models.milestone as model_milestone import blockerbugs.models.update as model_update @@ -79,7 +82,8 @@ class Bug(BaseModel): """A JSON list of bug numbers which this bug depends on""" updates: list['model_update.Update'] = db.relationship( 'Update', secondary=models.update_fixes, back_populates='bugs', lazy='dynamic', - order_by=(model_update.Update.status.desc(), model_update.Update.pending.desc())) + order_by=('[Update.status.desc(), Update.request.desc()]') + ) def __init__(self, bugid: Optional[int], @@ -126,6 +130,33 @@ class Bug(BaseModel): def depends_on(self, value: Optional[list[int]]) -> None: self._depends_on = json.dumps(value or []) + @hybrid_property + def is_proposed_accepted(self) -> bool: + """Return ``True`` if this bug is either proposed or accepted as a Blocker, FreezeException + or Prioritized. + """ + return (self.proposed_blocker or + self.proposed_fe or + self.accepted_blocker or + self.accepted_0day or + self.accepted_prevrel or + self.accepted_fe or + self.prioritized) + + @is_proposed_accepted.expression # type: ignore[no-redef] + def is_proposed_accepted(cls) -> bool: + """Return ``True`` if this bug is either proposed or accepted as a Blocker, FreezeException + or Prioritized. + """ + # The SQLAlchemy expression when using ``is_proposed_accepted`` in a query. + return or_(cls.proposed_blocker, + cls.proposed_fe, + cls.accepted_blocker, + cls.accepted_0day, + cls.accepted_prevrel, + cls.accepted_fe, + cls.prioritized) + def __repr__(self): return '' % (self.bugid, self.summary) diff --git a/blockerbugs/models/milestone.py b/blockerbugs/models/milestone.py index 4ad9c7e..c754653 100644 --- a/blockerbugs/models/milestone.py +++ b/blockerbugs/models/milestone.py @@ -21,8 +21,8 @@ from typing import Any, Optional -from blockerbugs import db, BaseModel, models -from blockerbugs.models import bug, criterion, update +from blockerbugs import db, BaseModel +from blockerbugs.models import bug, criterion from blockerbugs.models import release as model_release @@ -51,8 +51,6 @@ class Milestone(BaseModel): """Current milestone is the most relevant one currently. Usually it is the nearest milestone in the future. There should be at most one milestone marked as current.""" bugs: list['bug.Bug'] = db.relationship('Bug', back_populates='milestone', lazy='dynamic') - updates: list['update.Update'] = db.relationship('Update', secondary=models.update_milestones, - back_populates='milestones') criteria: list['criterion.Criterion'] = db.relationship('Criterion', back_populates='milestone', lazy='dynamic') succeeds_id = db.Column(db.Integer, db.ForeignKey('milestone.id'), nullable=True) diff --git a/blockerbugs/models/update.py b/blockerbugs/models/update.py index 02b2501..b298e76 100644 --- a/blockerbugs/models/update.py +++ b/blockerbugs/models/update.py @@ -19,75 +19,114 @@ """Database model for Bodhi updates""" +from typing import Optional, Any +from datetime import datetime + from blockerbugs import db, BaseModel, models -from blockerbugs.models import bug, milestone +from blockerbugs.models import bug from blockerbugs.models import release as model_release class Update(BaseModel): - id = db.Column(db.Integer, primary_key=True) - title = db.Column(db.Text, unique=False) - url = db.Column(db.Text, unique=False) - karma = db.Column(db.Integer, unique=False) - stable_karma = db.Column(db.Integer, unique=False) - status = db.Column(db.String(80), unique=False) - request = db.Column(db.String(80), unique=False, nullable=True) - pending = db.Column(db.Boolean(create_constraint=True, name='pending_bool'), unique=False) - date_submitted = db.Column(db.DateTime) - date_pushed_testing = db.Column(db.DateTime, nullable=True) - date_pushed_stable = db.Column(db.DateTime, nullable=True) - date_obsoleted = db.Column(db.DateTime, nullable=True) + """A representation of a Bodhi update. + + Bodhi update states are documented here: + https://bodhi.fedoraproject.org/docs/user/update_states.html + And here's a JSON schema for the different fields: + https://bodhi.fedoraproject.org/docs/server_api/messages/update.html#json-schemas + """ + id: int = db.Column(db.Integer, primary_key=True) + updateid: str = db.Column(db.Text, nullable=False, unique=True) + """E.g. FEDORA-2021-5282b5cafd""" + release_id: int = db.Column(db.Integer, db.ForeignKey('release.id'), nullable=False) + release: 'model_release.Release' = db.relationship('Release', back_populates='updates') + """The release this update was created for in Bodhi""" + status: str = db.Column(db.Enum('stable', 'pending', 'testing', create_constraint=True, + name='update_status_enum'), + nullable=False) + """One of: 'stable', 'pending', 'testing'. A Bodhi update can have additional values, but we + only allow these three here. Because this is an enum, Updates can be sorted by this in the + specified order (updates in a need of testing have higher value).""" + karma: int = db.Column(db.Integer, nullable=False) + """Current karma value""" + url: str = db.Column(db.Text, nullable=False) + """E.g. https://bodhi.fedoraproject.org/updates/FEDORA-2021-5282b5cafd""" + date_submitted: datetime = db.Column(db.DateTime, nullable=False) + """When the update was created""" + request: Optional[str] = db.Column(db.Enum('revoke', 'unpush', 'obsolete', 'stable', 'testing', + create_constraint=True, name='update_request_enum')) + """One of: 'revoke', 'unpush', 'obsolete', 'stable', 'testing', None. Because this is an enum, + Updates can be sorted by this in the specified order (updates in a need of testing have higher + value).""" + title: Optional[str] = db.Column(db.Text) + """E.g. firewalld-0.9.4-1.fc34""" + stable_karma: Optional[int] = db.Column(db.Integer) + """Target karma value for allowing the update to go stable (via auto or manual push).""" bugs: list['bug.Bug'] = db.relationship( 'Bug', secondary=models.update_fixes, back_populates='updates') - release_id = db.Column(db.Integer, db.ForeignKey('release.id')) - release: 'model_release.Release' = db.relationship('Release', back_populates='updates') - milestones: list['milestone.Milestone'] = db.relationship( - 'Milestone', secondary=models.update_milestones, back_populates='updates') + """A list of bugs this update claims to fix *and* we track them""" - def __init__(self, title, url, karma, status, bugs, release, - milestones, stable_karma=3, date_submitted=None): - self.title = title - self.url = url - self.karma = karma + _tmpstr = 'A placeholder value when creating incomplete Update objects' + + def __init__(self, + updateid: str, + release: 'model_release.Release', + status: str, + karma: int, + url: str, + date_submitted: datetime, + request: Optional[str] = None, + title: Optional[str] = None, + stable_karma: Optional[int] = None, + bugs: list['bug.Bug'] = []) -> None: + self.updateid = updateid + self.release = release self.status = status - self.request = None - self.bugs = bugs + self.karma = karma + self.url = url self.date_submitted = date_submitted - self.release = release - if milestones: - self.milestones = milestones - self.date_pushed_testing = None - self.date_pushed_stable = None - self.date_obsoleted = None + self.request = request + self.title = title + self.stable_karma = stable_karma + self.bugs = bugs - def __str__(self): - return 'update: %s' % (self.title) + def __repr__(self) -> str: + return f'' - def sync(self, updateinfo): - self.title = updateinfo['title'] - self.url = updateinfo['url'] - self.karma = updateinfo['karma'] - self.stable_karma = updateinfo['stable_karma'] + def sync(self, updateinfo: dict[str, Any]) -> None: + self.updateid = updateinfo['updateid'] self.status = updateinfo['status'] - self.request = updateinfo['request'] + self.karma = updateinfo['karma'] + self.url = updateinfo['url'] self.date_submitted = updateinfo['date_submitted'] - if updateinfo['date_pushed_testing']: - self.date_pushed_testing = updateinfo['date_pushed_testing'] - if updateinfo['date_pushed_stable']: - self.date_pushed_stable = updateinfo['date_pushed_stable'] - self.pending = updateinfo['pending'] - current_bugkeys = [(currentbug.bugid, currentbug.milestone) for currentbug in self.bugs] - for bugid in updateinfo['bugs']: - newbugs = bug.Bug.query.filter_by(bugid=bugid).all() - for newbug in newbugs: - if not (newbug.bugid, newbug.milestone) in current_bugkeys: - self.bugs.append(newbug) - current_bugkeys.append((newbug.bugid, newbug.milestone)) - if newbug.milestone and newbug.milestone not in self.milestones: - self.milestones.append(newbug.milestone) + self.request = updateinfo['request'] + self.title = updateinfo['title'] + self.stable_karma = updateinfo['stable_karma'] + + self.bugs.clear() + for bugid in set(updateinfo['bugs']): # deduplicate just to be sure + assert isinstance(bugid, int) + tracked_bugs = bug.Bug.query.filter_by(bugid=bugid).all() + for tracked_bug in tracked_bugs: + self.bugs.append(tracked_bug) + + # a quick check that mandatory values seem initialized + checkfields = [self.updateid, self.release, self.status, self.karma, self.url, + self.date_submitted, self.bugs] + assert None not in checkfields + assert self._tmpstr not in checkfields @classmethod - def from_data(cls, updateinfo, release): - newupdate = Update(updateinfo['title'], '', 0, '', [], release, None) + def from_data(cls, updateinfo: dict[str, Any], release: 'model_release.Release') -> 'Update': + newupdate = Update(updateid=updateinfo['updateid'], + release=release, + status=cls._tmpstr, + karma=-99, + url=cls._tmpstr, + date_submitted=datetime.utcfromtimestamp(0), + request=None, + title=cls._tmpstr, + stable_karma=None, + bugs=[]) newupdate.sync(updateinfo) return newupdate diff --git a/blockerbugs/static/css/blockerbugs.css b/blockerbugs/static/css/blockerbugs.css index 9d96816..70ccd08 100644 --- a/blockerbugs/static/css/blockerbugs.css +++ b/blockerbugs/static/css/blockerbugs.css @@ -188,4 +188,9 @@ thead th.wide { width: 60% !important; } +.badge { + font-size: 0.9em; + font-weight: inherit; +} + /*# sourceMappingURL=app-bootstrap.css.map */ diff --git a/blockerbugs/templates/blocker_list.html b/blockerbugs/templates/blocker_list.html index 420ca8e..0095164 100644 --- a/blockerbugs/templates/blocker_list.html +++ b/blockerbugs/templates/blocker_list.html @@ -66,8 +66,8 @@ Component Status Title + Updates Review - Updates @@ -87,6 +87,18 @@ {{ bug.component }} {{ bug.status }} {{ bug.summary }} + {% set update_html = bug.updates.first() | updatestatus | safe %} + {% if update_html %} + {% set num_updates = bug.updates | list | length %} + + {{ update_html }} + {% if num_updates > 1 %} + ({{ num_updates }}) + {% endif %} + + {% else %} + + {% endif %} {% if buglist.startswith('Proposed') and vote_info[bug.bugid][buglist] %} @@ -112,14 +124,6 @@ TBD {% endif %} - {% set update_html = bug | updatelabel | safe %} - {% if update_html %} - - {{ update_html }} - - {% else %} - - {% endif %} {% else %} diff --git a/blockerbugs/templates/bug_tooltip.html b/blockerbugs/templates/bug_tooltip.html index 73b46dc..c681285 100644 --- a/blockerbugs/templates/bug_tooltip.html +++ b/blockerbugs/templates/bug_tooltip.html @@ -1,27 +1,27 @@
-

Potential fixes

- - - - - - - - - - - {% for update in updates %} - - - - - - - {% endfor %} - -
UpdateKarmaStatusTime
{{ update.title }}{{ update.karma }}{{ update.status }}{{ update.date_submitted }}
+

Potential fixes

+ + + + + + + + + + + {% for update in updates %} + + + + + + + {% endfor %} + +
UpdateKarmaStatusCreated
{{ update.title or update.updateid }}{{ update.karma }}{% if update.stable_karma %}/{{ update.stable_karma }}{% endif %}{{ update | updatestatus | safe }}{{ update.date_submitted | datetime }}
diff --git a/blockerbugs/templates/requests.txt b/blockerbugs/templates/requests.txt index a690e7d..3ededfe 100644 --- a/blockerbugs/templates/requests.txt +++ b/blockerbugs/templates/requests.txt @@ -3,12 +3,12 @@ == Blockers == {% for update in blocker_updates %} -* [{{ update.title }}]({{ update.url }}) for {%- for bug in update.bugs if bug.milestone_id == milestone and (bug.accepted_blocker or bug.accepted_0day or bug.accepted_fe) %} [#{{ bug.bugid }}]({{ bug.url }}){% if bug.accepted_fe and not bug.accepted_blocker and not bug.accepted_prevrel and not bug.accepted_0day %} (FE){% endif %}{% endfor %} +* [{{ update.title or update.updateid }}]({{ update.url }}) for {%- for bug in update.bugs if bug.milestone_id == milestone and (bug.accepted_blocker or bug.accepted_0day or bug.accepted_fe) %} [#{{ bug.bugid }}]({{ bug.url }}){% if bug.accepted_fe and not bug.accepted_blocker and not bug.accepted_prevrel and not bug.accepted_0day %} (FE){% endif %}{% endfor %} {%- endfor %} == Freeze exceptions == {% for update in fe_updates %} -* [{{ update.title }}]({{ update.url }}) for {%- for bug in update.bugs if bug.milestone_id == milestone and (bug.accepted_blocker or bug.accepted_0day or bug.accepted_fe) %} [#{{ bug.bugid }}]({{ bug.url }}){% endfor %} +* [{{ update.title or update.updateid }}]({{ update.url }}) for {%- for bug in update.bugs if bug.milestone_id == milestone and (bug.accepted_blocker or bug.accepted_0day or bug.accepted_fe) %} [#{{ bug.bugid }}]({{ bug.url }}){% endfor %} {%- endfor %} @@ -16,12 +16,12 @@ == Blockers == {% for update in blocker_updates if update.request == 'stable' %} -* [{{ update.title }}]({{ update.url }}) for {%- for bug in update.bugs if bug.milestone_id == milestone and (bug.accepted_blocker or bug.accepted_0day or bug.accepted_fe) %} [#{{ bug.bugid }}]({{ bug.url }}){% if bug.accepted_fe and not bug.accepted_blocker and not bug.accepted_prevrel and not bug.accepted_0day %} (FE){% endif %}{% endfor %} +* [{{ update.title or update.updateid }}]({{ update.url }}) for {%- for bug in update.bugs if bug.milestone_id == milestone and (bug.accepted_blocker or bug.accepted_0day or bug.accepted_fe) %} [#{{ bug.bugid }}]({{ bug.url }}){% if bug.accepted_fe and not bug.accepted_blocker and not bug.accepted_prevrel and not bug.accepted_0day %} (FE){% endif %}{% endfor %} {%- endfor %} == Freeze exceptions == {% for update in fe_updates if update.request == 'stable' %} -* [{{ update.title }}]({{ update.url }}) for {%- for bug in update.bugs if bug.milestone_id == milestone and (bug.accepted_blocker or bug.accepted_0day or bug.accepted_fe) %} [#{{ bug.bugid }}]({{ bug.url }}){% endfor %} +* [{{ update.title or update.updateid }}]({{ update.url }}) for {%- for bug in update.bugs if bug.milestone_id == milestone and (bug.accepted_blocker or bug.accepted_0day or bug.accepted_fe) %} [#{{ bug.bugid }}]({{ bug.url }}){% endfor %} {%- endfor %} {% endautoescape %} diff --git a/blockerbugs/templates/update_list.html b/blockerbugs/templates/update_list.html index e98d3f3..7767d64 100644 --- a/blockerbugs/templates/update_list.html +++ b/blockerbugs/templates/update_list.html @@ -2,34 +2,20 @@ {% block jsheader %} - {% endblock %} +{% endblock %} -{# this should be imported from somewhere else, not pasted #} -{% macro statustext(update) %} -{% if update %} -{% if update.pending %} -pending -{% endif %} -{{ update.status }} -{% endif %} -{% endmacro %} {% block body %} -
- {% for update_list in ['Non Stable Updates', 'Updates Needing Testing'] %} -

{{ update_list }}

- +

Updates fixing tracked bugs

+
@@ -40,19 +26,21 @@ pending - {% for update in updates[update_list] %} + {% for update in updates %} + {% set relevant_bugs = update.bugs | selectattr('active') | + selectattr('milestone', '==', milestone) | selectattr('is_proposed_accepted') | + list %} - - - + + + @@ -67,7 +55,6 @@ pending {% endfor %}
Type
{{ update | updatetype }}{{ update.bugs[0].component }}'{{ update.title }}'{{ statustext(update) }}{{ relevant_bugs | map(attribute='component') | sort | join(', ') }}{{ update.title or update.updateid }}{{ update | updatestatus | safe }} - {% for bug in update.bugs %} - {% if (bug.proposed_blocker or bug.proposed_fe or bug.accepted_blocker or bug.accepted_fe or bug.accepted_0day or bug.accepted_prevrel) %} - - {{ bug.bugid }} - - {% endif %} + {% for bug in relevant_bugs %} + + {{- bug.bugid -}} + + {%- if not loop.last %}, {% endif %} {% endfor %}
- {% endfor %}
diff --git a/blockerbugs/util/update_sync.py b/blockerbugs/util/update_sync.py index 4f43335..1c6ed69 100644 --- a/blockerbugs/util/update_sync.py +++ b/blockerbugs/util/update_sync.py @@ -21,200 +21,183 @@ from datetime import datetime import logging +from typing import Optional, Any, Iterable, Sequence from fedora.client import ServerError from bodhi.client.bindings import BodhiClient +import flask_sqlalchemy from blockerbugs import app from blockerbugs.models.update import Update from blockerbugs.models.bug import Bug - -bodhi_baseurl = app.config['BODHI_URL'] +from blockerbugs.models.release import Release class UpdateSync(object): - def __init__(self, db, bodhi_interface=None): + """The main class for perfoming Update synchronization with Bodhi.""" + + def __init__(self, db: flask_sqlalchemy.SQLAlchemy, bodhiclient: Optional[BodhiClient] = None + ) -> None: self.db = db - if bodhi_interface: - self.bodhi_interface = bodhi_interface() - else: - # disable saving session on disk by cache_session=False - self.bodhi_interface = BodhiClient(base_url=bodhi_baseurl, cache_session=False) + # disable saving session on disk by cache_session=False + self.bodhi = bodhiclient or BodhiClient(base_url=app.config['BODHI_URL'], + cache_session=False) self.log = logging.getLogger('update_sync') - self._releases = [] # all known releases + self._releases: list[dict[str, Any]] = [] + """All releases known to Bodhi""" @property - def releases(self): - '''All releases known to Bodhi, as retrieved from /releases/ endpoint''' + def releases(self) -> list[dict[str, Any]]: + """All releases known to Bodhi, as retrieved from /releases/ endpoint""" if self._releases: # already retrieved return self._releases - self._releases = self.bodhi_interface.get_releases( - rows_per_page=100)['releases'] - - self.log.debug('Retrieved %d known releases from Bodhi' % - (len(self._releases))) + self._releases = self.bodhi.get_releases(rows_per_page=100)['releases'] + self.log.debug('Retrieved %d known releases from Bodhi', len(self._releases)) return self._releases - def extract_information(self, update): - updateinfo = {} - date_pushed = None - if update.date_pushed: - date_pushed = datetime.strptime(update.date_pushed, - '%Y-%m-%d %H:%M:%S') - updateinfo['date_pushed_testing'] = None - updateinfo['date_pushed_stable'] = None - updateinfo['pending'] = False - # this will be None if there is no request - updateinfo['request'] = update.request - if update.status == 'pending': - updateinfo['pending'] = True - - if update.request == 'stable': - updateinfo['status'] = 'stable' - updateinfo['date_pushed_testing'] = date_pushed - elif update.request == 'testing': - updateinfo['status'] = 'testing' - else: - updateinfo['status'] = 'undefined' - else: - updateinfo['status'] = str(update.status) - if update.status == 'testing': - updateinfo['date_pushed_testing'] = date_pushed - elif update.status == 'stable': - updateinfo['date_pushed_stable'] = date_pushed - - updateinfo['title'] = str(update.title) - updateinfo['karma'] = update.karma - updateinfo['stable_karma'] = update.stable_karma - updateinfo['url'] = update.url - - updateinfo['date_submitted'] = datetime.strptime(update.date_submitted, + def extract_information(self, update: dict) -> dict[str, Any]: + """Create a dict with extracted Update information. See the code to learn the dict keyvals. + + :param update: the update object as retrieved from Bodhi API + """ + updateinfo: dict[str, Any] = {} + updateinfo['updateid'] = update['updateid'] + updateinfo['status'] = update['status'] + updateinfo['karma'] = update['karma'] + updateinfo['url'] = update['url'] + updateinfo['date_submitted'] = datetime.strptime(update['date_submitted'], '%Y-%m-%d %H:%M:%S') - updateinfo['bugs'] = [] - if len(update.bugs) > 0: - for buginfo in update.bugs: - updateinfo['bugs'].append(buginfo.bug_id) + updateinfo['title'] = update['title'] + updateinfo['request'] = update['request'] + updateinfo['stable_karma'] = update['stable_karma'] + updateinfo['bugs'] = [buginfo['bug_id'] for buginfo in update['bugs']] return updateinfo - def get_update(self, envr): - updates = self.bodhi_interface.query(package=envr) - return self.extract_information(updates['updates'][0]) - - def is_watched_bug(self, bugnum): - watched_bug = Bug.query.filter_by(bugid=bugnum).first() - if watched_bug: - return True - else: - return False - - def clean_updates(self, updates, relid): - """Remove updates for this release which are no longer related - to any blocker or freeze exception bug from the database. + def clean_updates(self, updateids: Iterable[str], release: Release) -> None: + """Remove all updates from the database which are related to a particular release and are + not listed among ``updateids``. """ - query_updates = Update.query.filter( - Update.release_id == relid, - ).all() - db_updates = set(update.title for update in query_updates) - bodhi_updates = set(update['title'] for update in updates) - unneeded_updates = db_updates.difference(bodhi_updates) - - for update in query_updates: - if update.title in unneeded_updates: - self.log.debug("Removing no longer relevant update %s" % - update.title) + db_updates: list[Update] = Update.query.filter_by(release=release).all() + db_updateids = set(update.updateid for update in db_updates) + unneeded_updateids = db_updateids.difference(updateids) + + for update in db_updates: + if update.updateid in unneeded_updateids: + self.log.debug("Removing no longer relevant %r", update) self.db.session.delete(update) - self.db.session.commit() + self.db.session.commit() - def search_updates(self, bugids, release_num): - # not all releases exist all the time (before branching, before bodhi - # activation point, etc), so drop those which Bodhi doesn't currently - # know of - known_releases = [rel['name'].lower() for rel in self.releases] + def search_updates(self, bugids: Sequence[int], release_num: int) -> list[dict[str, Any]]: + """Find all Bodhi updates in a particular release which fix one of the bugs specified. + + :param bugids: Bugzilla ticket numbers + :param release_num: the version of a release to query + :return: a list of update info dictionaries, as provided by ``extract_information()`` + """ query_releases = [ - 'f%d' % release_num, # standard repo + 'f%d' % release_num, # rpms 'f%df' % release_num, # flatpaks - 'f%dm' % release_num, # modularity + 'f%dm' % release_num, # modules + 'f%dc' % release_num, # containers ] + # not all releases exist all the time (before branching, before Bodhi activation point, + # etc), so drop those which Bodhi doesn't currently know of + known_releases = [rel['name'].lower() for rel in self.releases] for rel in query_releases.copy(): if rel not in known_releases: - self.log.warning("Release %s not found in Bodhi (might be " - "normal depending on release life cycle)" % rel) + self.log.debug("Release %s not found in Bodhi (might be normal depending on the " + "release life cycle)", rel) query_releases.remove(rel) - - queries_data = dict( - bugs=[str(bug_id) for bug_id in bugids], - release=query_releases, - limit=100, - ) - result = self.bodhi_interface.query(**queries_data) - - if u'status' in result.keys(): - raise ServerError('', 200, result['errors'][0].description) - - updates = {} - for update in result.updates: - updates[update.title] = update - - while result.page < result.pages: - result = self.bodhi_interface.query(page=result.page + 1, - **queries_data) - for update in result.updates: - updates[update.title] = update - - updates = updates.values() # updates without duplicates - self.log.info('found %d updates from bodhi for %d bugs in f%d' % - (len(updates), len(bugids), release_num)) + if not query_releases: + self.log.warning("No releases related to F%d found in Bodhi! Nothing to query.", + release_num) + return [] + + queries_data = { + 'bugs': [str(bug_id) for bug_id in bugids], + 'release': query_releases, + 'limit': 100, + 'status': ['pending', 'testing', 'stable'], + } + updates_dict = {} + # Bodhi counts pages from 1 + page = pages = 1 + while page <= pages: + result = self.bodhi.query(page=page, **queries_data) + + if 'status' in result: + raise ServerError('', 400, result['errors'][0]['description']) + + for update in result['updates']: + assert update['release']['version'] == str(release_num) + assert update['status'] in ['pending', 'testing', 'stable'] + updates_dict[update['updateid']] = update + + page += 1 + pages = result['pages'] + + updates = updates_dict.values() # updates without duplicates + self.log.info('Found %d updates in Bodhi for %d bugs in release F%d', len(updates), + len(bugids), release_num) return [self.extract_information(update) for update in updates] - def get_release_bugs(self, release): + def get_release_bugs(self, release: Release) -> list[Bug]: + """Get all open proposed/accepted bugs related to a certain release (i.e. to all its + active milestones). + """ buglist = [] - for milestone in release.milestones: - buglist.extend(milestone.bugs.filter_by(active=True).all()) + for milestone in release.milestones.filter_by(active=True): # type: ignore[attr-defined] + buglist.extend( + milestone.bugs.filter( # type: ignore[attr-defined] + Bug.active == True, # noqa: E712 + Bug.is_proposed_accepted == True, + ).all() + ) + return buglist - def sync_bug_updates(self, release, bugs): - starttime = datetime.utcnow() - bugs_ids = [bug.bugid for bug in bugs] - self.log.debug('searching for updates for bugs %s' % str(bugs_ids)) - try: - updates = self.search_updates(bugs_ids, release.number) - except ServerError as ex: - self.log.error( - 'f{r.number} sync updates failed: {e.code} {e.msg}'.format( - e=ex, r=release)) - return + def sync_updates(self, release: Release) -> None: + """Synchronize all updates for a particular release. That means pulling new updates from + Bodhi (related to all bugs which we track in the release), and removing no-longer-relevant + updates from the database. + """ + self.log.info('Syncing updates for release F%d ...', release.number) + + bugs = self.get_release_bugs(release) + self.log.debug('Found %d relevant bugs in release F%d', len(bugs), release.number) + + updateinfos = [] + synctime = datetime.utcnow() + if bugs: + bugs_ids = [bug.bugid for bug in bugs] + self.log.debug('Searching Bodhi for updates for bugs %s', bugs_ids) + try: + updateinfos = self.search_updates(bugs_ids, release.number) + except ServerError as ex: + self.log.error( + 'F{r.number} sync updates failed: {e.code} {e.msg}'.format(e=ex, r=release)) + return + else: + self.log.debug('Skipping Bodhi query due to no available bugs') # remove no longer relevant updates from the database - self.clean_updates(updates, release.id) - - for update in updates: - self.log.debug('running sync for update %s' % update['title']) - existing_update = Update.query.filter_by( - title=update['title']).first() - if existing_update: - self.log.debug( - 'syncing existing update %s' % existing_update.title) - existing_update.sync(update) + updateids = [u['updateid'] for u in updateinfos] + self.clean_updates(updateids, release) + + # update the existing Update objects or create new ones + for updateinfo in updateinfos: + oldupdate = Update.query.filter_by(updateid=updateinfo['updateid']).one_or_none() + if oldupdate: + self.log.debug('Updating existing %r', oldupdate) + oldupdate.sync(updateinfo) + self.db.session.add(oldupdate) else: - self.log.debug('creating new update %s' % update['title']) - existing_update = Update.from_data(update, release) - self.db.session.add(existing_update) - self.db.session.commit() + newupdate = Update.from_data(updateinfo, release) + self.log.debug('Created new %r', newupdate) + self.db.session.add(newupdate) - release.last_update_sync = starttime - self.db.session.add(release) + release.last_update_sync = synctime self.db.session.commit() - - def sync_updates(self, release): - bugs = self.get_release_bugs(release) - self.log.info( - 'found %d bugs in f%d release' % (len(bugs), release.number)) - if not bugs: - self.log.info( - 'no bugs in f%d release, skip update' % (release.number)) - release.last_update_sync = datetime.utcnow() - return - self.sync_bug_updates(release, bugs) diff --git a/testing/test_api.py b/testing/test_api.py index e473679..e1d1e5d 100644 --- a/testing/test_api.py +++ b/testing/test_api.py @@ -34,24 +34,25 @@ class TestRestAPI(object): bug1 = add_bug(9000, 'testbug1', cls.milestone) bug1.accepted_fe = True bug1.status = 'CLOSED' + bug1copy = add_bug(9000, 'testbug1', cls.milestone2) # different milestone than bug1 + bug1copy.accepted_fe = True + bug1copy.status = 'CLOSED' bug2 = add_bug(9002, 'testbug2', cls.milestone) bug2.accepted_blocker = False bug2.proposed_fe = True bug3 = add_bug(9003, 'testbug3', cls.milestone) bug3.accepted_blocker = False bug3.proposed_fe = True - bug4 = add_bug(9003, 'testbug3', cls.milestone2) # same bugid and summary as bug3 is intentional + bug4 = add_bug(9003, 'testbug3', cls.milestone2) # different milestone and proposals than bug3 bug4.accepted_blocker = True bug4.proposed_fe = False bug4.status = 'CLOSED' - cls.update_pending_stable = add_update(u'test-pending-stable.fc99', - u'stable', [bug1], cls.release, - [cls.milestone, cls.milestone2]) + cls.update_pending_stable = add_update('test-pending-stable.fc99', cls.release, 'testing', + [bug1, bug1copy]) cls.update_pending_stable.date_submitted = datetime(1990, 1, 1) - cls.update_pending_stable.pending = True - cls.update_testing2 = add_update(u'test-testing2.fc99', u'testing', - [bug2], - cls.release, [cls.milestone]) + cls.update_pending_stable.request = 'stable' + cls.update_pending_stable.title = 'mega fixer' + cls.update_testing2 = add_update('test-testing2.fc99', cls.release, 'testing', [bug2]) build = Build() build.koji_id = 33 build.nvr = 'libofx-0.9.9-1.fc20' @@ -124,15 +125,15 @@ class TestRestAPI(object): data = json.loads(resp.data) assert len(data) == 2 update = data[-1] - assert update['title'] == u'test-pending-stable.fc99' + assert update['updateid'] == u'test-pending-stable.fc99' + assert update['title'] == 'mega fixer' assert update['url'] == u'http://localhost/update' assert update['karma'] == 1 assert update['stable_karma'] == 3 - assert update['status'] == u'stable' - assert update['pending'] + assert update['status'] == 'testing' + assert update['request'] == 'stable' assert update['bugs'][0]['bugid'] == 9000 - assert set(update['bugs'][0]['type']) == set(('accepted_blocker', - 'accepted_fe')) + assert set(update['bugs'][0]['type']) == set(('accepted_blocker', 'accepted_fe')) assert update['release'] == 99 assert len(update['milestones']) == 2 assert {'version': 'final', 'release': 99} in update['milestones'] @@ -145,7 +146,7 @@ class TestRestAPI(object): data = json.loads(resp.data) assert len(data) == 1 update = data[0] - assert update['title'] == u'test-testing2.fc99' + assert update['updateid'] == u'test-testing2.fc99' def test_bad_bugtype_list_bugs(self): url = '/api/v0/milestones/99/final/updates?bugtype=foo&' diff --git a/testing/test_controllers.py b/testing/test_controllers.py index c5c9d9d..0fb976f 100644 --- a/testing/test_controllers.py +++ b/testing/test_controllers.py @@ -1,11 +1,11 @@ import datetime +import re from blockerbugs.models.milestone import Milestone from blockerbugs.models.release import Release from blockerbugs.models.bug import Bug from blockerbugs.models.update import Update -from blockerbugs import db -from blockerbugs import app +from blockerbugs import app, db from blockerbugs.controllers import main @@ -26,21 +26,32 @@ def add_milestone(release, version, blocker_tracker, fe_tracker, name, def add_bug(bugid, summary, milestone): - test_bug = Bug(bugid, - 'https://bugzilla.redhat.com/show_bug.cgi?id=%d' % bugid, - summary, 'NEW', 'testcomponent', milestone, True, True, 'John Doe') + test_bug = Bug(bugid=bugid, + url='https://bugzilla.redhat.com/show_bug.cgi?id=%d' % bugid, + summary=summary, + status='NEW', + component='testcomponent', + milestone=milestone, + active=True, + needinfo=True, + needinfo_requestee='John Doe') test_bug.accepted_blocker = True db.session.add(test_bug) db.session.commit() return test_bug -def add_update(title, status, bugs, release, milestones, - url='http://localhost/update'): - test_update = Update(title, url, 1, status, bugs, - release, milestones, - date_submitted=datetime.datetime.utcnow()) - test_update.stable_karma = 3 +def add_update(updateid, release, status, bugs=[]): + test_update = Update(updateid=updateid, + release=release, + status=status, + karma=1, + url='http://localhost/update', + date_submitted=datetime.datetime.utcnow(), + request=None, + title=None, + stable_karma=3, + bugs=bugs) db.session.add(test_update) db.session.commit() return test_update @@ -211,93 +222,82 @@ class TestSyncedMilestone(object): class TestGetFunctions(object): @classmethod - def setup_class(cls): + def setup_method(self): + self.client = app.test_client() db.session.rollback() db.drop_all() db.create_all() - cls.release = add_release(99) - cls.milestone = add_milestone(cls.release, 'beta', 100, 101, - '99-beta', True) - bug1 = add_bug(9000, 'testbug1', cls.milestone) - bug1.depends_on = [9090] - bug2 = add_bug(9001, 'testbug2', cls.milestone) - bug2.accepted_blocker = False + self.release = add_release(99) + self.milestone = add_milestone(self.release, 'beta', 100, 101, '99-beta', True) + self.bug1 = add_bug(9000, 'testbug1', self.milestone) + self.bug1.depends_on = [9090] + self.bug2 = add_bug(9001, 'testbug2', self.milestone) + self.bug2.accepted_blocker = False # a bug that fixes a Final blocker - cls.finalmile = add_milestone(cls.release, 'final', 200, 201, '99-final') - finalbug = add_bug(9002, 'finalbug', cls.finalmile) + self.finalmile = add_milestone(self.release, 'final', 200, 201, '99-final') + self.finalbug = add_bug(9002, 'finalbug', self.finalmile) # we add finalbug to this update for requests.txt testing: - # it should *not* show in the generated request, the update + # it should *not* show in the generated request for Beta, the update # will be in the request but it should only list bug1, not # finalbug, as bug1 is for Beta and finalbug for Final - cls.update_pending_stable = add_update(u'test-pending-stable.fc99', u'stable', - [bug1, finalbug], cls.release, [cls.milestone]) - cls.update_pending_stable.pending = True - cls.update_pending_stable.request = 'stable' - cls.update_stable1 = add_update(u'test-stable1.fc99', u'stable', [bug1], - cls.release, [cls.milestone]) - cls.update_stable2 = add_update(u'test-stable2.fc99', u'stable', [bug2], - cls.release, [cls.milestone]) - cls.update_pending_testing = add_update(u'test-pending-testing.fc99', u'testing', [bug1], - cls.release, [cls.milestone]) - cls.update_pending_testing.pending = True - cls.update_testing1 = add_update(u'test-testing1.fc99', u'testing', [bug1], - cls.release, [cls.milestone]) - cls.update_testing2 = add_update(u'test-testing2.fc99', u'testing', [bug2], - cls.release, [cls.milestone]) + self.update_pending_stable = add_update('test-pending-stable.fc99', self.release, 'testing', + [self.bug1, self.finalbug]) + self.update_pending_stable.request = 'stable' + self.update_stable1 = add_update('test-stable1.fc99', self.release, 'stable', [self.bug1]) + self.update_stable2 = add_update('test-stable2.fc99', self.release, 'stable', [self.bug2]) + self.update_pending_testing = add_update('test-pending-testing.fc99', self.release, + 'pending', [self.bug1]) + self.update_pending_testing.request = 'testing' + self.update_testing1 = add_update('test-testing1.fc99', self.release, 'testing', + [self.bug1]) + self.update_testing2 = add_update('test-testing2.fc99', self.release, 'testing', + [self.bug2]) # add an update that fixes a bug which is an accepted Beta FE # and a proposed Beta blocker, and another bug which is an # accepted Final blocker, for fix function tests - betafebug = add_bug(9003, 'betafebug', cls.milestone) - betafebug.accepted_blocker = False - betafebug.proposed_blocker = True - betafebug.accepted_fe = True - cls.update_complex = add_update(u'test-complex1.fc99', u'testing', - [betafebug, finalbug], - cls.release, [cls.milestone, cls.finalmile]) + self.betafebug = add_bug(9003, 'betafebug', self.milestone) + self.betafebug.accepted_blocker = False + self.betafebug.proposed_blocker = True + self.betafebug.accepted_fe = True + self.update_complex = add_update('test-complex1.fc99', self.release, 'testing', + [self.betafebug, self.finalbug]) # also an update that fixes a proposed Beta FE and an accepted # Final FE for the same purpose - propbetafebug = add_bug(9004, 'propbetafebug', cls.milestone) - propbetafebug.accepted_blocker = False - propbetafebug.proposed_fe = True - finalfebug = add_bug(9005, 'finalfebug', cls.finalmile) - finalfebug.accepted_blocker = False - finalfebug.accepted_fe = True - cls.update_complexfe = add_update(u'test-complexfe.fc99', u'testing', - [propbetafebug, finalfebug], - cls.release, [cls.milestone, cls.finalmile]) + self.propbetafebug = add_bug(9004, 'propbetafebug', self.milestone) + self.propbetafebug.accepted_blocker = False + self.propbetafebug.proposed_fe = True + self.finalfebug = add_bug(9005, 'finalfebug', self.finalmile) + self.finalfebug.accepted_blocker = False + self.finalfebug.accepted_fe = True + self.update_complexfe = add_update('test-complexfe.fc99', self.release, 'testing', + [self.propbetafebug, self.finalfebug]) db.session.commit() @classmethod - def teardown_class(cls): + def teardown_method(self): db.session.rollback() + db.session.close() db.drop_all() - def test_get_pending_stable_updates(self): - updates = main.get_milestone_pending_stable_updates(self.milestone) - assert len(updates) == 1 - update = updates[0] - assert self.update_pending_stable == update - - - def test_get_testing_status_and_has_bugs_updates(self): - updates = main.get_milestone_updates_testing(self.milestone) - assert len(updates) == 3 - expected = [self.update_testing1, self.update_complex, self.update_complexfe] - updates.sort(key=lambda x: x.title) - expected.sort(key=lambda x: x.title) - assert updates == expected - - def test_get_milestone_all_nonstable_blocker_fixes(self): - updates = main.get_milestone_all_nonstable_blocker_fixes(self.milestone) - assert len(updates) == 3 - # we should NOT find complex or complexfe here! - expected = [self.update_pending_stable, self.update_pending_testing, self.update_testing1] - updates.sort(key=lambda x: x.title) - expected.sort(key=lambda x: x.title) - assert updates == expected - - def test_get_milestone_all_nonstable_fe_fixes(self): - updates = main.get_milestone_all_nonstable_fe_fixes(self.milestone) + def test_get_milestone_updates(self): + updates = main.get_milestone_updates(self.milestone) + assert set(updates) == set([self.update_pending_stable, + self.update_stable1, + self.update_pending_testing, + self.update_testing1, + self.update_complex, + self.update_complexfe + ]) + + def test_get_updates_nonstable_blockers(self): + updates = main.get_updates_nonstable_blockers(self.milestone) + assert set(updates) == set([self.update_pending_stable, + self.update_pending_testing, + self.update_testing1 + ]) + + def test_get_updates_nonstable_FEs(self): + updates = main.get_updates_nonstable_FEs(self.milestone) # we should find complex (as it fixes an accepted Beta FE) but # NOT complexfe here! assert len(updates) == 1 @@ -306,13 +306,16 @@ class TestGetFunctions(object): def test_requests(self): # we also test the requests template generation here, as it's - # what the _fixes queries back and it makes sense to do it + # what the get_* queries provide and it makes sense to do it # with all these bits in place with app.app_context(): - beta_response = main.display_release_requests(num=self.release.number, - release_name=self.milestone.version) - final_response = main.display_release_requests(num=self.release.number, - release_name=self.finalmile.version) + beta_response = self.client.get( + f'/milestone/{self.release.number}/{self.milestone.version}/requests') + assert beta_response.status_code == 200 + final_response = self.client.get( + f'/milestone/{self.release.number}/{self.finalmile.version}/requests') + assert final_response.status_code == 200 + # === beta milestone === beta_requests = beta_response.get_data(as_text=True) print(beta_requests) @@ -350,7 +353,7 @@ class TestGetFunctions(object): compose_blockers, compose_fes, push_blockers, push_fes, deps = all_sections # check all the right updates and bugs are and are not in the correct sections - assert item_just_in("test-pending-stable", [], all_sections) + assert item_just_in("test-pending-stable", [compose_blockers, push_blockers], all_sections) assert item_just_in("test-stable1", [], all_sections) assert item_just_in("test-stable2", [], all_sections) assert item_just_in("test-pending-testing", [], all_sections) @@ -363,10 +366,25 @@ class TestGetFunctions(object): # bug 2 id assert item_just_in("9001", [], all_sections) # finalbug id - assert item_just_in("9002", [compose_blockers], all_sections) + assert item_just_in("9002", [compose_blockers, push_blockers], all_sections) # betafebug id assert item_just_in("9003", [], all_sections) # propbetafebug id assert item_just_in("9004", [], all_sections) # finalfebug id assert item_just_in("9005", [compose_fes], all_sections) + + def test_display_bug_updates(self): + with app.app_context(): + for bug in [self.bug1, self.bug2, self.finalbug, self.betafebug, self.propbetafebug, + self.finalfebug]: + rv = self.client.get(f'/bug/{bug.bugid}/updates') + assert rv.status_code == 200 + html = rv.get_data(as_text=True) + + # make sure the right updates are listed + for update in bug.updates.all(): + assert update.updateid in html + # make sure there are no additional updates listed + matches = re.findall(re.escape(''), html) + assert len(matches) == len(bug.updates.all()) diff --git a/testing/test_init.py b/testing/test_init.py new file mode 100644 index 0000000..8f2194c --- /dev/null +++ b/testing/test_init.py @@ -0,0 +1,87 @@ +'''Test blockerbugs.__init__''' + +import datetime + +import pytest + +from blockerbugs import db, updatetype +from blockerbugs.models.update import Update +from blockerbugs.models.release import Release +from blockerbugs.models.bug import Bug + + +class TestInit(object): + def setup_method(self, method): + db.session.rollback() + db.drop_all() + db.create_all() + + self.release = Release(99) + db.session.add(self.release) + self.bug1 = Bug(bugid='1', + url='http://bug1', + summary='summary 1', + status='NEW', + component='component 1', + milestone=None, + active=True, + needinfo=False, + needinfo_requestee=None) + db.session.add(self.bug1) + self.bug2 = Bug(bugid='2', + url='http://bug2', + summary='summary 2', + status='NEW', + component='component 2', + milestone=None, + active=True, + needinfo=False, + needinfo_requestee=None) + db.session.add(self.bug2) + self.update1 = Update(updateid='update_01', + release=self.release, + status='testing', + karma=0, + url='http://nowhere', + date_submitted=datetime.datetime.now(), + bugs=[self.bug1, self.bug2]) + db.session.add(self.update1) + db.session.commit() + + def teardown_method(self, method): + db.session.rollback() + db.session.close() + db.drop_all() + + @pytest.mark.parametrize('blocker', ['proposed_blocker', 'accepted_blocker', 'accepted_0day', + 'accepted_prevrel', None]) + @pytest.mark.parametrize('fe', ['proposed_fe', 'accepted_fe', None]) + @pytest.mark.parametrize('prio', ['prioritized', None]) + def test_updatetype(self, blocker, fe, prio): + print(blocker, fe, prio) + for attr in [blocker, fe, prio]: + if attr: + setattr(self.bug1, attr, True) + if blocker: + assert updatetype(self.update1) == 'Blocker' + elif fe: + assert updatetype(self.update1) == 'FreezeException' + elif prio: + assert updatetype(self.update1) == 'Prioritized' + + if blocker is fe is prio is None: + with pytest.raises(AssertionError): + updatetype(self.update1) + + def test_updatetype_second_bug_override(self): + '''A second bug must override the first one if it has a higher priority''' + self.bug1.proposed_fe = True + self.bug2.proposed_blocker = True + assert updatetype(self.update1) == 'Blocker' + + # the result shouldn't depend on the order + self.update1.bugs = [self.bug2, self.bug1] + assert updatetype(self.update1) == 'Blocker' + + def test_updatetype_no_input(self): + assert updatetype(None) == '' diff --git a/testing/test_updatemodel.py b/testing/test_updatemodel.py new file mode 100644 index 0000000..33e8a88 --- /dev/null +++ b/testing/test_updatemodel.py @@ -0,0 +1,128 @@ +'''Test blockerbugs.models.update''' + +import datetime +import copy + +from blockerbugs import db +from blockerbugs.models.update import Update +from blockerbugs.models.release import Release +from blockerbugs.models.bug import Bug + + +class TestUpdateModel(object): + def setup_method(self, method): + db.session.rollback() + db.drop_all() + db.create_all() + + self.release = Release(99) + db.session.add(self.release) + self.bug1 = Bug(bugid='1', + url='http://bug1', + summary='summary 1', + status='NEW', + component='component 1', + milestone=None, + active=True, + needinfo=False, + needinfo_requestee=None) + db.session.add(self.bug1) + self.bug2 = Bug(bugid='2', + url='http://bug2', + summary='summary 2', + status='NEW', + component='component 2', + milestone=None, + active=True, + needinfo=False, + needinfo_requestee=None) + db.session.add(self.bug2) + db.session.commit() + + self.updateinfo = { + 'updateid': 'update_01', + 'title': 'test title', + 'url': 'http://nowhere', + 'karma': 0, + 'stable_karma': 3, + 'status': 'testing', + 'request': 'stable', + 'date_submitted': datetime.datetime.now(), + 'bugs': [1, 2], + } + + def teardown_method(self, method): + db.session.rollback() + db.session.close() + db.drop_all() + + def create_update(self): + self.update = Update(updateid=self.updateinfo['updateid'], + release=self.release, + status=self.updateinfo['status'], + karma=self.updateinfo['karma'], + url=self.updateinfo['url'], + date_submitted=self.updateinfo['date_submitted'], + request=self.updateinfo['request'], + title=self.updateinfo['title'], + stable_karma=self.updateinfo['stable_karma'], + bugs=[self.bug1, self.bug2]) + db.session.add(self.update) + + def test_create_update(self): + self.create_update() + db.session.commit() + + updates = Update.query.all() + assert len(updates) == 1 + assert updates[0] is self.update + assert self.update.bugs == [self.bug1, self.bug2] + + def test_sync(self): + self.create_update() + db.session.commit() + updateinfo = copy.deepcopy(self.updateinfo) + updateinfo['title'] = 'new title' + updateinfo['karma'] = 3 + updateinfo['status'] = 'stable' + updateinfo['request'] = None + updateinfo['bugs'] = [1] + + self.update.sync(updateinfo) + db.session.commit() + + updates = Update.query.all() + assert len(updates) == 1 + assert updates[0] is self.update + for field in ['title', 'karma', 'status', 'request']: + assert getattr(self.update, field) == updateinfo[field] != self.updateinfo[field] + assert self.update.bugs == [self.bug1] + + def test_sync_handle_duplicate_bugs(self): + '''If the bug numbers are duplicated, we must handle it (deduplicate it)''' + self.create_update() + db.session.commit() + updateinfo = copy.deepcopy(self.updateinfo) + updateinfo['bugs'] = [1, 1] + + self.update.sync(updateinfo) + db.session.commit() + + updates = Update.query.all() + assert len(updates) == 1 + assert updates[0] is self.update + assert self.update.bugs == [self.bug1] + + def test_from_data(self): + update = Update.from_data(self.updateinfo, self.release) + db.session.add(update) + db.session.commit() + + updates = Update.query.all() + assert len(updates) == 1 + assert updates[0] is update + for key, value in self.updateinfo.items(): + if key == 'bugs': + assert update.bugs == [self.bug1, self.bug2] + else: + assert getattr(update, key) == value diff --git a/testing/test_updatesync_extract_information.py b/testing/test_updatesync_extract_information.py index 5c8f2f1..b37a32d 100644 --- a/testing/test_updatesync_extract_information.py +++ b/testing/test_updatesync_extract_information.py @@ -14,7 +14,9 @@ basicupdate = Munch( date_pushed=u'2012-09-13 16:40:28', date_submitted=u'2012-09-12 23:43:04', notes=u'Fix live install mounting /home and /boot and swap', - request=None, stable_karma=3, status=u'testing', + request=None, + stable_karma=3, + status=u'testing', submitter=u'bcl', title=u'anaconda-18.6.8-1.fc18', type=u'bugfix', @@ -82,101 +84,16 @@ class TestUpdateSyncExtractInformation(object): self.testupdate = deepcopy(basicupdate) self.testsync = UpdateSync(None, MagicMock()) - def test_extract_status_testing(self): - self.testupdate.status = u'testing' - - updateinfo = self.testsync.extract_information(self.testupdate) - - assert updateinfo['status'] == 'testing' - assert updateinfo['pending'] == False - - def test_extract_status_pending_stable(self): - self.testupdate.status = u'pending' - self.testupdate.request = u'stable' - - updateinfo = self.testsync.extract_information(self.testupdate) - - assert updateinfo['status'] == 'stable' - assert updateinfo['pending'] == True - - def test_extract_status_pending_testing(self): - self.testupdate.status = u'pending' - self.testupdate.request = u'testing' - - updateinfo = self.testsync.extract_information(self.testupdate) - - assert updateinfo['status'] == 'testing' - assert updateinfo['pending'] == True - - def test_extract_status_pending_nothing(self): - self.testupdate.status = u'pending' - self.testupdate.request = None - - updateinfo = self.testsync.extract_information(self.testupdate) - - assert updateinfo['status'] == 'undefined' - assert updateinfo['pending'] == True - - def test_extract_title(self): - ref_title = u'magic-fixall-update-1.1-0' - self.testupdate.title = ref_title - - updateinfo = self.testsync.extract_information(self.testupdate) - - assert updateinfo['title'] == ref_title - - def test_extract_karma(self): - ref_karma = 2 - self.testupdate.karma = ref_karma - + def test_extract_simple(self): updateinfo = self.testsync.extract_information(self.testupdate) - assert updateinfo['karma'] == ref_karma - - def test_extract_url(self): - updateinfo = self.testsync.extract_information(self.testupdate) - - assert (updateinfo['url'] == 'https://bodhi.stg.fedoraproject.org/%s' % self.testupdate.updateid) - - def test_extract_date_pushed_testing_intesting(self): - self.testupdate.status = u'testing' - self.testupdate.request = None - self.testupdate.date_pushed = u'2012-09-13 16:40:28' - - updateinfo = self.testsync.extract_information(self.testupdate) - - assert updateinfo['date_pushed_testing'] == datetime.strptime(self.testupdate.date_pushed, '%Y-%m-%d %H:%M:%S') - assert updateinfo['date_pushed_stable'] == None - - def test_extract_date_pushed_testing_pendingstable(self): - self.testupdate.status = u'pending' - self.testupdate.request = u'stable' - self.testupdate.date_pushed = u'2012-09-13 16:40:28' - - updateinfo = self.testsync.extract_information(self.testupdate) - - assert updateinfo['date_pushed_testing'] == datetime.strptime(self.testupdate.date_pushed, '%Y-%m-%d %H:%M:%S') - assert updateinfo['date_pushed_stable'] == None - - def test_extract_date_pushed_stable(self): - self.testupdate.status = u'stable' - self.testupdate.request = None - self.testupdate.date_pushed = u'2012-09-13 16:40:28' - - updateinfo = self.testsync.extract_information(self.testupdate) - - assert updateinfo['date_pushed_testing'] == None - assert updateinfo['date_pushed_stable'] == datetime.strptime(self.testupdate.date_pushed, '%Y-%m-%d %H:%M:%S') - - def test_extract_date_pushed_pending_testing(self): - self.testupdate.status = u'pending' - self.testupdate.request = u'testing' - self.testupdate.date_pushed = None - - updateinfo = self.testsync.extract_information(self.testupdate) - - assert updateinfo['date_pushed_testing'] == None - assert updateinfo['date_pushed_stable'] == None + assert updateinfo['updateid'] == self.testupdate['updateid'] + assert updateinfo['title'] == self.testupdate['title'] + assert updateinfo['url'] == self.testupdate['url'] + assert updateinfo['karma'] == self.testupdate['karma'] + assert updateinfo['stable_karma'] == self.testupdate['stable_karma'] + assert updateinfo['status'] == self.testupdate['status'] + assert updateinfo['request'] == self.testupdate['request'] def test_extract_bug_single(self): ref_bugs = [Munch(bug_id=123456, parent=False, security=False, @@ -214,10 +131,3 @@ class TestUpdateSyncExtractInformation(object): updateinfo = self.testsync.extract_information(self.testupdate) assert updateinfo['date_submitted'] == datetime.strptime(ref_date_submitted, '%Y-%m-%d %H:%M:%S') - - # check for handling of update data when there is no date_pushed info which - # occurs when a brand new package is submitted as a blocker or FE fix - def test_extract_pushed_date_newpackage(self): - self.testupdate.date_pushed = None - - self.testsync.extract_information(self.testupdate) diff --git a/testing/testfunc_bugmodel.py b/testing/testfunc_bugmodel.py index 15ebcd5..fb0579c 100644 --- a/testing/testfunc_bugmodel.py +++ b/testing/testfunc_bugmodel.py @@ -11,6 +11,7 @@ from blockerbugs import db def generate_release(number=99): return Release(number) + def generate_milestone(version, name, release, blocker_tracker=123456, accepted_tracker=234567): return Milestone(release, version, blocker_tracker, accepted_tracker, name) @@ -19,9 +20,16 @@ def generate_bug(bugid, summary, milestone): return Bug(bugid, 'https://bugzilla.redhat.com/show_bug.cgi?id=%d' % bugid, summary, 'NEW', 'testcomponent', milestone, True, True, 'John Doe') -def generate_update(title, status, bugs, release, milestones, url='http://localhost/update'): - return Update(title, url, 1, status, bugs, - release, milestones, date_submitted=datetime.datetime.utcnow() ) + +def generate_update(updateid, release, status, bugs, url='http://localhost/update'): + return Update(updateid=updateid, + release=release, + status=status, + karma=1, + url=url, + date_submitted=datetime.datetime.utcnow(), + title=updateid, + bugs=bugs) class TestfuncBugModel(object): @@ -90,9 +98,8 @@ class TestfuncBugModel(object): db.session.commit() ref_title = 'test-1.0-1.fc99' - ref_status = 'PENDING STABLE' - ref_update = generate_update(ref_title, ref_status, [ref_bug], - self.ref_release, [self.ref_milestone]) + ref_status = 'testing' + ref_update = generate_update(ref_title, self.ref_release, ref_status, [ref_bug]) db.session.add(ref_update) db.session.commit() @@ -104,7 +111,6 @@ class TestfuncBugModel(object): assert update.status == ref_status assert update.bugs[0] == ref_bug - def test_add_update_long_title(self): ref_bugid = 123456 ref_bug = generate_bug(ref_bugid, 'testbug1', self.ref_milestone) @@ -115,10 +121,8 @@ class TestfuncBugModel(object): ref_title = 'x' * 3000 ref_url = 'http://localhost/update/%s' % ref_title - ref_status = 'PENDING STABLE' - ref_update = generate_update(ref_title, ref_status, [ref_bug], - self.ref_release, [self.ref_milestone], - url=ref_url) + ref_status = 'testing' + ref_update = generate_update(ref_title, self.ref_release, ref_status, [ref_bug], ref_url) db.session.add(ref_update) db.session.commit() @@ -128,7 +132,6 @@ class TestfuncBugModel(object): assert update.title == ref_title assert update.url == ref_url - def test_add_update_multiple_bugs(self): ref_bugs = [generate_bug(123456, 'testbug1', self.ref_milestone), generate_bug(234567, 'testbug2', self.ref_milestone)] @@ -139,9 +142,8 @@ class TestfuncBugModel(object): db.session.commit() ref_title = 'test-1.0-1.fc99' - ref_status = 'PENDING STABLE' - ref_update = generate_update(ref_title, ref_status, ref_bugs, - self.ref_release, [self.ref_milestone]) + ref_status = 'testing' + ref_update = generate_update(ref_title, self.ref_release, ref_status, ref_bugs) db.session.add(ref_update) db.session.commit() @@ -152,7 +154,6 @@ class TestfuncBugModel(object): assert ref_bugs[0] in update.bugs assert ref_bugs[1] in update.bugs - def test_add_criterion(self): db.session.add(self.ref_release) db.session.add(self.ref_milestone) diff --git a/testing/testfunc_update_sync.py b/testing/testfunc_update_sync.py index 8d32448..5bbb3d9 100644 --- a/testing/testfunc_update_sync.py +++ b/testing/testfunc_update_sync.py @@ -29,18 +29,22 @@ base_update = Munch( Munch(bug_id=2001, parent=False, security=False, title=u'bodhi bug 2001 title')], release=Munch(dist_tag=u'f99', id_prefix=u'FEDORA', locked=True, - long_name=u'Fedora 99', name=u'F99'), + long_name=u'Fedora 99', name=u'F99', version='99'), updateid='FEDORA-2012-13902', url=u'https://bodhi.stg.fedoraproject.org/FEDORA-2012-13902', ) update_for_bugs_2000_2001 = copy(base_update) +update_for_bugs_2000_2001.update({ + 'updateid': 'update_for_bugs_2000_2001', +}) update1_for_bug_3000 = copy(base_update) update1_for_bug_3000.update({ 'title': u'libreport-2.1.3-2.fc19', 'bugs': [Munch(bug_id=3000, parent=False, security=False, title=u"anaconda can't report traceback to bugzilla")], + 'updateid': 'update1_for_bug_3000', }) update2_for_bug_3000 = copy(base_update) @@ -48,10 +52,11 @@ update2_for_bug_3000.update({ 'title': u'jboss-servlet-2.5-api-1.0.1-3.fc19,resteasy-2.3.2-12.fc19', 'bugs': [Munch(bug_id=3000, parent=False, security=False, title=u"anaconda can't report traceback to bugzilla")], + 'updateid': 'update2_for_bug_3000', }) -update3_for_bug_3100 = copy(base_update) -update3_for_bug_3100.update({ +update_for_bug_3100 = copy(base_update) +update_for_bug_3100.update({ 'title': u'flatpak-runtime-f32-3220200414150125.1 and flatpak-sdk-f32-3220200414150125.1', 'bugs': [Munch(bug_id=3100, parent=False, security=False, title=u"Update F32 flatpak runtime to include gtk3-3.24.18-1.fc32")], @@ -69,16 +74,10 @@ class FakeBodhiInterface(object): ('f99', '2000'): [update_for_bugs_2000_2001], ('f99', '2001'): [update_for_bugs_2000_2001], ('f99', '3000'): [update1_for_bug_3000, update2_for_bug_3000], - ('f99f', '3100'): [update3_for_bug_3100], + ('f99f', '3100'): [update_for_bug_3100], ('f99', '4000'): [], } - updates_time = { - update_for_bugs_2000_2001.title: datetime(2001, 1, 1), - update1_for_bug_3000.title: datetime(2002, 1, 1), - update2_for_bug_3000.title: datetime(2003, 1, 1), - } - def get_releases(self, **kwargs): return Munch(releases=self.releases, page=1, @@ -125,13 +124,31 @@ class TestfuncUpdateSync(object): db.session.add(self.test_milestone99alpha) db.session.add(self.test_milestone99beta) db.session.commit() - self.update_sync = UpdateSync(db, FakeBodhiInterface) + self.fbi = FakeBodhiInterface() + self.update_sync = UpdateSync(db, self.fbi) def teardown_method(self, method): db.session.rollback() db.session.close() db.drop_all() + def test_search_updates_two_bugs_one_update(self): + bug1 = add_bug(2000, 'testbug1', self.test_milestone99alpha) + bug2 = add_bug(2001, 'testbug2', self.test_milestone99alpha) + updates = self.update_sync.search_updates([bug1.bugid, bug2.bugid], + self.test_release99.number) + assert len(updates) == 1 # duplicates resolved in search + update = updates[0] + assert len(update['bugs']) == 2 + assert bug1.bugid in update['bugs'] + assert bug2.bugid in update['bugs'] + + def test_search_updates_no_known_releases(self): + bug1 = add_bug(2000, 'testbug1', self.test_milestone99alpha) + self.fbi.releases = [] + updates = self.update_sync.search_updates([bug1.bugid], self.test_release99.number) + assert len(updates) == 0 + def test_sync_single_bug_with_one_update(self): bug1 = add_bug(2000, 'testbug1', self.test_milestone99alpha) self.update_sync.sync_updates(self.test_release99) @@ -140,8 +157,8 @@ class TestfuncUpdateSync(object): update = updates[0] assert len(update.bugs) == 1 # second bug not in db assert update.bugs[0].bugid == bug1.bugid - assert update.status == u'testing' - assert update.pending + assert update.status == 'pending' + assert update.request == 'testing' def test_sync_two_bugs_with_one_update(self): bug1 = add_bug(2000, 'testbug1', self.test_milestone99alpha) @@ -155,17 +172,6 @@ class TestfuncUpdateSync(object): assert bug1.bugid in bugs_id assert bug2.bugid in bugs_id - def test_search_updates_two_bugs_one_update(self): - bug1 = add_bug(2000, 'testbug1', self.test_milestone99alpha) - bug2 = add_bug(2001, 'testbug2', self.test_milestone99alpha) - updates = self.update_sync.search_updates([bug1.bugid, bug2.bugid], - self.test_release99.number) - assert len(updates) == 1 # duplicates resolved in search - update = updates[0] - assert len(update['bugs']) == 2 - assert bug1.bugid in update['bugs'] - assert bug2.bugid in update['bugs'] - def test_sync_one_bug_with_two_updates(self): bug1 = add_bug(3000, 'testbug1', self.test_milestone99alpha) self.update_sync.sync_updates(self.test_release99) @@ -226,10 +232,10 @@ class TestfuncUpdateSync(object): update = updates[0] assert len(update.bugs) == 1 # second bug not in db assert update.bugs[0].bugid == bug1.bugid - assert update.status == u'testing' - assert update.pending + assert update.status == 'pending' + assert update.request == 'testing' - def test_no_updates_for_bug(self): + def test_sync_no_updates_for_bug(self): add_bug(4000, 'testbug1', self.test_milestone99alpha) self.update_sync.sync_updates(self.test_release99) updates = Update.query.all() @@ -244,18 +250,18 @@ class TestfuncUpdateSync(object): updates = Update.query.all() assert len(updates) == 0 - def test_release_not_found_error(self): + def test_sync_release_not_found_error(self): add_bug(4000, 'testbug1', self.test_milestone99alpha) - update_sync = UpdateSync(db, FakeErrorBodhiInterface) + update_sync = UpdateSync(db, FakeErrorBodhiInterface()) log_error = Mock(wraps=update_sync.log.error) update_sync.log.error = log_error update_sync.sync_updates(self.test_release99) assert log_error.call_count == 1 assert 'Unknown release' in log_error.call_args[0][0] - def test_release_server_error(self): + def test_sync_release_server_error(self): add_bug(4000, 'testbug1', self.test_milestone99alpha) - update_sync = UpdateSync(db, FakeRaisesBodhiInterface) + update_sync = UpdateSync(db, FakeRaisesBodhiInterface()) log_error = Mock(wraps=update_sync.log.error) update_sync.log.error = log_error update_sync.sync_updates(self.test_release99) @@ -271,7 +277,7 @@ class TestfuncUpdateSync(object): assert len(updates) == 2 # now we do a sync with only *one* update for bug #3000 - b = FakeBodhiInterface + b = FakeBodhiInterface() b.updates[('f99', '3000')] = [update1_for_bug_3000] update_sync = UpdateSync(db, b) update_sync.sync_updates(self.test_release99) @@ -291,3 +297,50 @@ class TestfuncUpdateSync(object): # no longer relates to an open blocker/FE bug updates = Update.query.all() assert len(updates) == 0 + + def test_sync_duplicate_updates(self): + '''If we receive the same update twice (e.g. due to pagination), we must be able to deal + with it correctly (deduplicate them)''' + bug1 = add_bug(3000, 'testbug1', self.test_milestone99alpha) + self.fbi.updates = { + ('f99', '3000'): [update1_for_bug_3000, update1_for_bug_3000], # the same update twice + } + + self.update_sync.sync_updates(self.test_release99) + updates = Update.query.all() + + assert len(updates) == 1 + update: Update = updates[0] + assert update.updateid == update1_for_bug_3000.updateid + assert update.title == update1_for_bug_3000.title + assert len(update.bugs) == 1 + assert update.bugs[0].bugid == bug1.bugid + + def test_sync_update_with_duplicate_bugs(self): + '''If Bodhi returns an update which lists the same bug bug twice, we should recover from + that situation (deduplicate them).''' + bug1 = add_bug(3100, 'testbug1', self.test_milestone99alpha) + update_for_bug_3100.bugs = update_for_bug_3100.bugs * 2 # duplicate the bug reference + + self.update_sync.sync_updates(self.test_release99) + updates = Update.query.all() + + assert len(updates) == 1 + update: Update = updates[0] + assert update.updateid == update_for_bug_3100.updateid + assert update.title == update_for_bug_3100.title + assert len(update.bugs) == 1 + assert update.bugs[0].bugid == bug1.bugid + + def test_sync_skip_inactive_milestones(self): + '''Inactive milestones shouldn't be synced''' + add_bug(3000, 'testbug1', self.test_milestone99alpha) + add_bug(2000, 'testbug2', self.test_milestone99beta) + self.test_milestone99alpha.active = False + + self.update_sync.sync_updates(self.test_release99) + + updates = Update.query.all() + assert len(updates) == 1 + assert updates[0].updateid == update_for_bugs_2000_2001['updateid'] + # updates for bug 3000 are not present, which is OK