Add a buildstep to the taskotron buildmaster to retrieve the task artifacts (/var/lib/taskotron/artifacts/<uid>/) from the buildslave following exection of the task.
/var/lib/taskotron/artifacts/<uid>/
The taskotron.log file from the slave (and output from stderr/stdout, if sanely possible) should also be retrieved from the buildslave.
This ticket is a duplicate of https://pagure.io/taskotron/issue/80
This ticket had assigned some Differential requests: D290
Attaching the patch for ansible playbooks below, I'd like to get this reviewed before trying it out on dev, thanks. @tflink
Regarding stdout/stderr, I asked at #buildbot and they said that we'd need to hack a ShellCommand to get that since buildbot does not keep separate copy of stdout/stderr. Do we want to spend time on this or having taskotron.log would suffice?
diff --git a/roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2 b/roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2 index 4ac0ec9..7f4b8ee 100644 --- a/roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2 +++ b/roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2 @@ -142,6 +142,7 @@ from buildbot.steps.source.git import Git from buildbot.steps.shell import ShellCommand from buildbot.process.properties import Property, Interpolate from buildbot.steps.slave import RemoveDirectory +from buildbot.steps.transfer import DirectoryUpload, FileUpload factory = BuildFactory() @@ -161,14 +162,19 @@ factory.addStep(ShellCommand(command=["runtask", '-i', Interpolate('%(prop:item_type)s'), '-a', Interpolate('%(prop:arch)s'), '-j', Interpolate('%(prop:buildername)s/%(prop:buildnumber)s'), + '--uuid', Interpolate('%(prop:uuid)s'), Interpolate('%(prop:taskname)s.yml')], descriptionDone=[Interpolate('%(prop:taskname)s on %(prop:item)s')], name='runtask', logfiles={'taskotron.log': {'filename': '/var/log/taskotron/taskotron.log', }})) -# capture the taskotron log -factory.addStep(ShellCommand(command=["cat", "/var/log/taskotron/taskotron.log"], name="cat_log", - descriptionDone=['Store log'])) +# copy artifacts to master +factory.addStep(DirectoryUpload(slavesrc=Interpolate('/var/lib/taskotron/artifacts/%(prop:uuid)s/'), + masterdest=Interpolate('/srv/taskotron/artifacts/%(prop:uuid)s'))) + +# copy taskotron log to master +factory.addStep(FileUpload(slavesrc="/var/log/taskotron/taskotron.log", + masterdest=Interpolate('/srv/taskotron/artifacts/%(prop:uuid)s/taskotron.log'))) ####### RESOURCE LOCKS diff --git a/roles/taskotron/buildmaster/tasks/main.yml b/roles/taskotron/buildmaster/tasks/main.yml index 5244240..b9bd3ac 100644 --- a/roles/taskotron/buildmaster/tasks/main.yml +++ b/roles/taskotron/buildmaster/tasks/main.yml @@ -23,6 +23,17 @@ notify: - restart httpd +- name: copy artifacts httpd config + template: src=artifacts.conf.j2 dest=/etc/httpd/conf.d/artifacts.conf owner=root group=root + notify: + - restart httpd + +- name: create artifacts directory + file: path={{ item }} state=directory owner=buildmaster group=buildmaster mode=0775 setype=httpd_sys_content_t + with_items: + - /srv/taskotron + - /srv/taskotron/artifacts + - name: ensure buildmaster database is created when: buildmaster_db_host != 'localhost' delegate_to: "{{ buildmaster_db_host }}" diff --git a/roles/taskotron/buildmaster/templates/artifacts.conf.j2 b/roles/taskotron/buildmaster/templates/artifacts.conf.j2 new file mode 100644 index 0000000..268da83 --- /dev/null +++ b/roles/taskotron/buildmaster/templates/artifacts.conf.j2 @@ -0,0 +1,6 @@ +Alias /artifacts /srv/taskotron/artifacts +<Directory "/srv/taskotron/artifacts"> + Options +Indexes + AllowOverride None + Require all granted +</Directory>
@mkrizek: I wondered whether we can use Differential reviews even for projects and repos not configured in Phab (i.e. ansible scripts) and it seems we can! You can create a manual revision in Differential tab. I copied the patch above into D290, so that it's easier to comment on that code. Does that work for you?
! In #401#4876, @mkrizek wrote: Regarding stdout/stderr, I asked at #buildbot and they said that we'd need to hack a ShellCommand to get that since buildbot does not keep separate copy of stdout/stderr. Do we want to spend time on this or having taskotron.log would suffice?
I'm afraid that's a problem, because taskotron.log only contains libtaskotron logging output, it does not contain task output, which is something we definitely want to show. For example, compare rpmlint output: https://taskotron.fedoraproject.org/taskmaster//builders/x86_64/builds/40129/steps/runtask/logs/stdio with taskotron.log contents: https://taskotron.fedoraproject.org/taskmaster//builders/x86_64/builds/40129/steps/runtask/logs/taskotron.log
taskotron.log
I'd say skip the stdout/stderr for now, but create a ticket to figure out how to extract that from buildbot in the future.
For now, I'm OK with having separate links to stderr/stdout from execution. It'd be nice to have but I'd rather avoid hacking on buildbot too much for now
! In #401#4883, @tflink wrote: I'd say skip the stdout/stderr for now, but create a ticket to figure out how to extract that from buildbot in the future. Ticket created: this ticket
! In #401#4883, @tflink wrote: I'd say skip the stdout/stderr for now, but create a ticket to figure out how to extract that from buildbot in the future.
Ticket created: this ticket
Deployed on dev for testing. Closing.
Login to comment on this ticket.