This PR is transferred from: https://phab.qa.fedoraproject.org/D1208 The ticket is archived at: https://fedorapeople.org/groups/qa/phabarchive/differentials/phab.qa.fedoraproject.org/D1208.html The raw diff is archived at: https://fedorapeople.org/groups/qa/phabarchive/differentials/phab.qa.fedoraproject.org/raw/D1208.diff
Authored by jskladan on Jun 12 2017, 2:46 PM. Summary This changes the steps we use so they report their start/stop to Buildbot. There almost certainly is a nicer/better way of doing this like using factory-like code to produce the modified classes, or reimplementing the Bbots push notification code, but given the limited docs, this is the most straight-forward solution Test Plan Tested localy with docker setup (which has slightly different master.cfg), so there might be some copy-paste errors. But the overall idea is sound and working.
tflink:
I'd like to see some more comments to describe what's going on and why we're overriding some pretty base buildsteps but other than that, it looks good to me.
jskladan:
The patch overrides all the buildsteps we use - since we want every step to report on its progress, we ought to change how they behave, base or not. (I'm not sure I understand what you mean, though). Overall, the changes are the same for all buildsteps - we override the start and finished methods, to inject the report call into them. The report method just plain HTTP POSTs to an url given to it, and sends the provided data. All the steps have gained the execdb_url parameter in constructor, in order to store that url (which is constructed partialy in the jinja, and during runtime). The patch then sets the said execdb_url parameter in every addStep call, so the constructor can be called without exceptions raising. Does this answer the questions? If not, I'd probably need you to be a bit more specific :)
start
finished
report
HTTP POST
execdb_url
addStep
Yeah, that answers most of the questions - I just wanted them in the master.cfg file for future reference or if someone is looking at it for the first time
will do
Diff:
Index: inventory/group_vars/taskotron-dev =================================================================== --- inventory/group_vars/taskotron-dev +++ inventory/group_vars/taskotron-dev @@ -44,7 +44,7 @@ ############################################################ execdb_endpoint: execdb -execdb_statuspush: http://resultsdb-dev01.qa.fedoraproject.org/execdb/buildbottest +execdb_statuspush: http://resultsdb-dev01.qa.fedoraproject.org/execdb/api/v1/jobs execdb_server: http://resultsdb-dev01.qa.fedoraproject.org/execdb resultsdb_api_endpoint: resultsdb_api resultsdb_host: http://resultsdb-dev01.qa.fedoraproject.org/resultsdb_api/ Index: inventory/group_vars/taskotron-prod =================================================================== --- inventory/group_vars/taskotron-prod +++ inventory/group_vars/taskotron-prod @@ -37,7 +37,7 @@ ############################################################ execdb_server: http://resultsdb01.qa.fedoraproject.org/execdb -execdb_statuspush: http://resultsdb01.qa.fedoraproject.org/execdb/buildbottest +execdb_statuspush: http://resultsdb01.qa.fedoraproject.org/execdb/api/v1/jobs resultsdb_external_url: https://taskotron.fedoraproject.org/resultsdb/ resultsdb_fe_endpoint: resultsdb resultsdb_frontend_url: http://resultsdb01.qa.fedoraproject.org/resultsdb Index: inventory/group_vars/taskotron-stg =================================================================== --- inventory/group_vars/taskotron-stg +++ inventory/group_vars/taskotron-stg @@ -42,7 +42,7 @@ ############################################################ execdb_server: http://resultsdb-stg01.qa.fedoraproject.org/execdb -execdb_statuspush: http://resultsdb-stg01.qa.fedoraproject.org/execdb/buildbottest +execdb_statuspush: http://resultsdb-stg01.qa.fedoraproject.org/execdb/api/v1/jobs resultsdb_url: http://resultsdb-stg01.qa.fedoraproject.org/resultsdb_api/api/v2.0 resultsdb_frontend_url: http://resultsdb-stg01.qa.fedoraproject.org/ Index: roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2 =================================================================== --- roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2 +++ roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2 @@ -35,6 +35,7 @@ # the 'change_source' setting tells the buildmaster how it should find out # about source code changes. Here we point to the buildbot clone of pyflakes. +from buildbot.steps.slave import RemoveDirectory #as RemoveDirectory_orig #from buildbot.changes.gitpoller import GitPoller #c['change_source'] = [] #c['change_source'].append(GitPoller( @@ -171,37 +172,176 @@ # only take place on one slave. from buildbot.process.factory import BuildFactory -from buildbot.steps.source.git import Git -from buildbot.steps.shell import ShellCommand from buildbot.process.properties import Property, Interpolate -from buildbot.steps.slave import RemoveDirectory -from buildbot.steps.transfer import DirectoryUpload, FileUpload -from buildbot.steps.master import MasterShellCommand +from build.process import results as buildbot_results + +from buildbot.steps.source.git import Git as Git_orig +from buildbot.steps.shell import ShellCommand as ShellCommand_orig +from buildbot.steps.slave import RemoveDirectory as RemoveDirectory_orig +from buildbot.steps.transfer import DirectoryUpload as DirectoryUpload_orig +from buildbot.steps.transfer import FileUpload as FileUpload_orig +from buildbot.steps.master import MasterShellCommand as MasterShellCommand_orig + +import json +import urllib2 + +def map_result(result): + if result in [buildbot_results.SUCCESS, buildbot_results.WARNINGS]: + return 'COMPLETED' + if result in [buildbot_results.SKIPPED, buildbot_results.CANCELLED]: + return 'ABORTED' + if result == buildbot_results.EXCEPTION: + return 'CRASHED' + return 'FAILED' + +def report(url, data): + data = json.dumps(data) + req = urllib2.Request(url, data, {'Content-Type': 'application/json'}) + response = urllib2.urlopen(req) + data = json.load(response) + response.close() + return data + +class JobStartStopStep(ShellCommand_orig): + def __init__(self, execdb_url, execdb_args, **kwargs): + self.execdb_url = execdb_url + self.execdb_args = execdb_args + super(self.__class__, self).__init__(**kwargs) + + def start(self, **kwargs): + report(self.execdb_url, execdb_args) + return super(self.__class__, self).start(**kwargs) + +class Git(Git_orig): + def __init__(self, execdb_url, **kwargs): + self.execdb_url = execdb_url + self.execdb_step_id = None + super(self.__class__, self).__init__(**kwargs) + + def start(self, **kwargs): + data = report(self.execdb_url, {"name": self.name, "description": self.name}) + self.execdb_step_id = data['id'] + return super(self.__class__, self).start(**kwargs) + + def finished(self, results): + report("%s/%s" % (self.execdb_url, self.execdb_step_id), {"outcome": map_result(results)}) + return super(self.__class__, self).finished(results) + +class ShellCommand(ShellCommand_orig): + def __init__(self, execdb_url, **kwargs): + self.execdb_url = execdb_url + self.execdb_step_id = None + super(self.__class__, self).__init__(**kwargs) + + def start(self, **kwargs): + data = report(self.execdb_url, {"name": self.name, "description": self.name}) + self.execdb_step_id = data['id'] + return super(self.__class__, self).start(**kwargs) + + def finished(self, results): + report("%s/%s" % (self.execdb_url, self.execdb_step_id), {"outcome": map_result(results)}) + return super(self.__class__, self).finished(results) + +class RemoveDirectory(RemoveDirectory_orig): + def __init__(self, execdb_url, **kwargs): + self.execdb_url = execdb_url + self.execdb_step_id = None + super(self.__class__, self).__init__(**kwargs) + + def start(self, **kwargs): + data = report(self.execdb_url, {"name": self.name, "description": self.name}) + self.execdb_step_id = data['id'] + return super(self.__class__, self).start(**kwargs) + + def finished(self, results): + report("%s/%s" % (self.execdb_url, self.execdb_step_id), {"outcome": map_result(results)}) + return super(self.__class__, self).finished(results) + +class DirectoryUpload(DirectoryUpload_orig): + def __init__(self, execdb_url, **kwargs): + self.execdb_url = execdb_url + self.execdb_step_id = None + super(self.__class__, self).__init__(**kwargs) + + def start(self, **kwargs): + data = report(self.execdb_url, {"name": self.name, "description": self.name}) + self.execdb_step_id = data['id'] + return super(self.__class__, self).start(**kwargs) + + def finished(self, results): + report("%s/%s" % (self.execdb_url, self.execdb_step_id), {"outcome": map_result(results)}) + return super(self.__class__, self).finished(results) + +class FileUpload(FileUpload_orig): + def __init__(self, execdb_url, **kwargs): + self.execdb_url = execdb_url + self.execdb_step_id = None + super(self.__class__, self).__init__(**kwargs) + + def start(self, **kwargs): + data = report(self.execdb_url, {"name": self.name, "description": self.name}) + self.execdb_step_id = data['id'] + return super(self.__class__, self).start(**kwargs) + + def finished(self, results): + report("%s/%s" % (self.execdb_url, self.execdb_step_id), {"outcome": map_result(results)}) + return super(self.__class__, self).finished(results) + +class MasterShellCommand(MasterShellCommand_orig): + def __init__(self, execdb_url, **kwargs): + self.execdb_url = execdb_url + self.execdb_step_id = None + super(self.__class__, self).__init__(**kwargs) + + def start(self, **kwargs): + data = report(self.execdb_url, {"name": self.name, "description": self.name}) + self.execdb_step_id = data['id'] + return super(self.__class__, self).start(**kwargs) + + def finished(self, results): + report("%s/%s" % (self.execdb_url, self.execdb_step_id), {"outcome": map_result(results)}) + return super(self.__class__, self).finished(results) + factory = BuildFactory() +factory.addStep(JobStartStopStep( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/"), + execdb_args={"start": True}, + command=["true"], name="start_job", descriptionDone=['Start ExecDB job'])) + {% if deployment_type in ['local'] %} # clean out /var/tmp/taskotron (see T253) -factory.addStep(ShellCommand(command="rm -rf /var/tmp/taskotron/*", name="rm_tmp", descriptionDone=['Clean tmp'])) +factory.addStep(ShellCommand( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + command="rm -rf /var/tmp/taskotron/*", name="rm_tmp", descriptionDone=['Clean tmp'])) # clean the log (see T230) -factory.addStep(ShellCommand(command=["rm", "-f", "/var/log/taskotron/taskotron.log"], name="rm_log", descriptionDone=['Clean log'])) +factory.addStep(ShellCommand( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + command=["rm", "-f", "/var/log/taskotron/taskotron.log"], name="rm_log", descriptionDone=['Clean log'])) {% endif %} {% if deployment_type in ['dev', 'stg', 'prod'] %} # clean out /var/tmp/taskotron (see T253) -factory.addStep(ShellCommand(command=Interpolate("rm -rf /var/tmp/taskotron/%(prop:slavename)s/*"), name="rm_tmp", descriptionDone=['Clean tmp'])) +factory.addStep(ShellCommand( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + command=Interpolate("rm -rf /var/tmp/taskotron/%(prop:slavename)s/*"), name="rm_tmp", descriptionDone=['Clean tmp'])) {% endif %} # check out the source -factory.addStep(Git(repourl=Property('repo', default=Interpolate('{{ grokmirror_user }}@{{ buildmaster }}:/var/lib/git/mirror/fedoraqa/%(prop:taskname)s/')), +factory.addStep(Git( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + repourl=Property('repo', default=Interpolate('{{ grokmirror_user }}@{{ buildmaster }}:/var/lib/git/mirror/fedoraqa/%(prop:taskname)s/')), branch=Property('branch', default='{{ grokmirror_default_branch }}'), mode='full', method='clobber', shallow=True)) # run the runner -factory.addStep(ShellCommand(command=["runtask", +factory.addStep(ShellCommand( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + command=["runtask", '-i', Interpolate('%(prop:item)s'), '-t', Interpolate('%(prop:item_type)s'), '-a', Interpolate('%(prop:arch)s'), @@ -224,26 +364,36 @@ {% endif %} -factory.addStep(ShellCommand(command=Interpolate('testcloud instance remove --force taskotron-%(prop:uuid)s; true'), - descriptionDone=['Make sure the minion is removed'])) +factory.addStep(ShellCommand( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + command=Interpolate('testcloud instance remove --force taskotron-%(prop:uuid)s; true'), + descriptionDone=['Make sure the minion is removed'])) # create artifacts dir on master -factory.addStep(MasterShellCommand(command=["mkdir", '-m', '0755', Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s')], - descriptionDone=['Create artifacs dir'])) +factory.addStep(MasterShellCommand( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + command=["mkdir", '-m', '0755', Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s')], + descriptionDone=['Create artifacs dir'])) # copy artifacts to master -factory.addStep(DirectoryUpload(slavesrc=Interpolate('/var/lib/taskotron/artifacts/%(prop:uuid)s/'), - masterdest=Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s/task_output'))) +factory.addStep(DirectoryUpload( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + slavesrc=Interpolate('/var/lib/taskotron/artifacts/%(prop:uuid)s/'), + masterdest=Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s/task_output'))) # gzip artifacts -factory.addStep(MasterShellCommand(command=Interpolate('gzip -r {{ public_artifacts_dir }}/%(prop:uuid)s/task_output/*'), - descriptionDone=['gzip artifacs dir content'])) +factory.addStep(MasterShellCommand( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + command=Interpolate('gzip -r {{ public_artifacts_dir }}/%(prop:uuid)s/task_output/*'), + descriptionDone=['gzip artifacs dir content'])) {% if deployment_type in ['local'] %} # copy taskotron log to master -factory.addStep(FileUpload(slavesrc='/var/log/taskotron/taskotron.log', - masterdest=Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s/taskotron.log'), - mode=0644)) +factory.addStep(FileUpload( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + slavesrc='/var/log/taskotron/taskotron.log', + masterdest=Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s/taskotron.log'), + mode=0644)) {% endif %} # move the artifacts to the correct dir on the master @@ -255,35 +405,56 @@ return datetime.datetime.now().strftime("%Y%m%d") # move artifacts dir -factory.addStep(MasterShellCommand(command=Interpolate('mkdir -p -m 0755 {{ public_artifacts_dir }}/%(kw:today)s && mkdir -p -m 0755 {{ public_artifacts_dir }}/all && mv {{ public_artifacts_dir }}/%(prop:uuid)s/ {{ public_artifacts_dir }}/%(kw:today)s/ && ln -s {{ public_artifacts_dir }}/%(kw:today)s/%(prop:uuid)s {{ public_artifacts_dir }}/all/', today=today), - descriptionDone=['Move artifacs dir'])) +factory.addStep(MasterShellCommand( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + command=Interpolate('mkdir -p -m 0755 {{ public_artifacts_dir }}/%(kw:today)s && mkdir -p -m 0755 {{ public_artifacts_dir }}/all && mv {{ public_artifacts_dir }}/%(prop:uuid)s/ {{ public_artifacts_dir }}/%(kw:today)s/ && ln -s {{ public_artifacts_dir }}/%(kw:today)s/%(prop:uuid)s {{ public_artifacts_dir }}/all/', today=today), + descriptionDone=['Move artifacs dir'])) + +factory.addStep(JobStartStopStep( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/"), + execdb_args={"outcome": '_COMPUTED_'}, + command=["true"], name="end_job", descriptionDone=['End ExecDB job'])) {% if deployment_type in ['dev', 'stg', 'prod'] %} # this is for package-specific tasks distgit_factory = BuildFactory() +distgit_factory.addStep(JobStartStopStep( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/"), + execdb_args={"start": True}, + command=["true"], name="start_job", descriptionDone=['Start ExecDB job'])) + + # clean out /var/tmp/taskotron (see T253) -distgit_factory.addStep(ShellCommand(command=Interpolate("rm -rf /var/tmp/taskotron/%(prop:slavename)s/*"), name="rm_tmp", descriptionDone=['Clean tmp'])) +distgit_factory.addStep(ShellCommand( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + command=Interpolate("rm -rf /var/tmp/taskotron/%(prop:slavename)s/*"), name="rm_tmp", descriptionDone=['Clean tmp'])) {% if deployment_type in ['dev', 'prod'] %} # check out the source from prod dist-git -distgit_factory.addStep(Git(repourl=Property('git_repo', default=Interpolate('http://pkgs02.phx2.fedoraproject.org/rpms/{}/'.format("%(prop:item)s".split('-')[0]))), +distgit_factory.addStep(Git( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + repourl=Property('git_repo', default=Interpolate('http://pkgs02.phx2.fedoraproject.org/rpms/{}/'.format("%(prop:item)s".split('-')[0]))), branch=Property('git_branch', default='master'), mode='full', method='clobber', shallow=True)) {% else %} # check out the source from stg dist-git -distgit_factory.addStep(Git(repourl=Property('git_repo', default=Interpolate('http://pkgs01.stg.phx2.fedoraproject.org/rpms/{}/'.format("%(prop:item)s".split('-')[0]))), +distgit_factory.addStep(Git( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + repourl=Property('git_repo', default=Interpolate('http://pkgs01.stg.phx2.fedoraproject.org/rpms/{}/'.format("%(prop:item)s".split('-')[0]))), branch=Property('git_branch', default='master'), mode='full', method='clobber', shallow=True)) {% endif %} # run the runner -distgit_factory.addStep(ShellCommand(command=["runtask", +distgit_factory.addStep(ShellCommand( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + command=["runtask", '-i', Interpolate('%(prop:item)s'), '-t', Interpolate('%(prop:item_type)s'), '-a', Interpolate('%(prop:arch)s'), @@ -301,26 +472,44 @@ )) -distgit_factory.addStep(ShellCommand(command=Interpolate('testcloud instance remove --force taskotron-%(prop:uuid)s; true'), - descriptionDone=['Make sure the minion is removed'])) +distgit_factory.addStep(ShellCommand( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + command=Interpolate('testcloud instance remove --force taskotron-%(prop:uuid)s; true'), + descriptionDone=['Make sure the minion is removed'])) # create artifacts dir on master -distgit_factory.addStep(MasterShellCommand(command=["mkdir", '-m', '0755', Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s')], - descriptionDone=['Create artifacs dir'])) +distgit_factory.addStep(MasterShellCommand( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + command=["mkdir", '-m', '0755', Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s')], + descriptionDone=['Create artifacs dir'])) # copy artifacts to master -distgit_factory.addStep(DirectoryUpload(slavesrc=Interpolate('/var/lib/taskotron/artifacts/%(prop:uuid)s/'), - masterdest=Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s/task_output'))) +distgit_factory.addStep(DirectoryUpload( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + slavesrc=Interpolate('/var/lib/taskotron/artifacts/%(prop:uuid)s/'), + masterdest=Interpolate('{{ public_artifacts_dir }}/%(prop:uuid)s/task_output'))) # gzip artifacts -distgit_factory.addStep(MasterShellCommand(command=Interpolate('gzip -r {{ public_artifacts_dir }}/%(prop:uuid)s/task_output/*'), - descriptionDone=['gzip artifacs dir content'])) +distgit_factory.addStep(MasterShellCommand( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + command=Interpolate('gzip -r {{ public_artifacts_dir }}/%(prop:uuid)s/task_output/*'), + descriptionDone=['gzip artifacs dir content'])) {% endif %} # move artifacts dir -distgit_factory.addStep(MasterShellCommand(command=Interpolate('mkdir -p -m 0755 {{ public_artifacts_dir }}/%(kw:today)s && mkdir -p -m 0755 {{ public_artifacts_dir }}/all && mv {{ public_artifacts_dir }}/%(prop:uuid)s/ {{ public_artifacts_dir }}/%(kw:today)s/ && ln -s {{ public_artifacts_dir }}/%(kw:today)s/%(prop:uuid)s {{ public_artifacts_dir }}/all/', today=today), - descriptionDone=['Move artifacs dir'])) +distgit_factory.addStep(MasterShellCommand( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/steps"), + command=Interpolate('mkdir -p -m 0755 {{ public_artifacts_dir }}/%(kw:today)s && mkdir -p -m 0755 {{ public_artifacts_dir }}/all && mv {{ public_artifacts_dir }}/%(prop:uuid)s/ {{ public_artifacts_dir }}/%(kw:today)s/ && ln -s {{ public_artifacts_dir }}/%(kw:today)s/%(prop:uuid)s {{ public_artifacts_dir }}/all/', today=today), + descriptionDone=['Move artifacs dir'])) + +distgit_factory.addStep(JobStartStopStep( + execdb_url=Interpolate("{{execdb_statuspush}}/%(prop:uuid)s/"), + execdb_args={"outcome": '_COMPUTED_'}, + command=["true"], name="end_job", descriptionDone=['End ExecDB job'])) + + + ####### RESOURCE LOCKS # # This is a set of resource locks to make sure that we don't have too many things
Metadata Update from @kparal: - Issue assigned to jskladan - Issue priority set to: Normal - Issue tagged with: infra
Login to comment on this ticket.