#220 Add buildmaster steps to retrieve artifacts from buildslaves
Closed: Fixed None Opened 9 years ago by tflink.

Add a buildstep to the taskotron buildmaster to retrieve the task artifacts (/var/lib/taskotron/artifacts/<uid>/) from the buildslave following exection of the task.

The taskotron.log file from the slave (and output from stderr/stdout, if sanely possible) should also be retrieved from the buildslave.


This ticket had assigned some Differential requests:
D290

Attaching the patch for ansible playbooks below, I'd like to get this reviewed before trying it out on dev, thanks. @tflink

Regarding stdout/stderr, I asked at #buildbot and they said that we'd need to hack a ShellCommand to get that since buildbot does not keep separate copy of stdout/stderr. Do we want to spend time on this or having taskotron.log would suffice?

diff --git a/roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2 b/roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2
index 4ac0ec9..7f4b8ee 100644
--- a/roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2
+++ b/roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2
@@ -142,6 +142,7 @@ from buildbot.steps.source.git import Git
 from buildbot.steps.shell import ShellCommand
 from buildbot.process.properties import Property, Interpolate
 from buildbot.steps.slave import RemoveDirectory
+from buildbot.steps.transfer import DirectoryUpload, FileUpload

 factory = BuildFactory()

@@ -161,14 +162,19 @@ factory.addStep(ShellCommand(command=["runtask", '-i',
                                         Interpolate('%(prop:item_type)s'),
                                         '-a', Interpolate('%(prop:arch)s'),
                                         '-j', Interpolate('%(prop:buildername)s/%(prop:buildnumber)s'),
+                                        '--uuid', Interpolate('%(prop:uuid)s'),
                                         Interpolate('%(prop:taskname)s.yml')],
                              descriptionDone=[Interpolate('%(prop:taskname)s on %(prop:item)s')],
                              name='runtask',
                              logfiles={'taskotron.log': {'filename': '/var/log/taskotron/taskotron.log', }}))

-# capture the taskotron log
-factory.addStep(ShellCommand(command=["cat", "/var/log/taskotron/taskotron.log"], name="cat_log",
-                            descriptionDone=['Store log']))
+# copy artifacts to master
+factory.addStep(DirectoryUpload(slavesrc=Interpolate('/var/lib/taskotron/artifacts/%(prop:uuid)s/'),
+                                masterdest=Interpolate('/srv/taskotron/artifacts/%(prop:uuid)s')))
+
+# copy taskotron log to master
+factory.addStep(FileUpload(slavesrc="/var/log/taskotron/taskotron.log",
+                           masterdest=Interpolate('/srv/taskotron/artifacts/%(prop:uuid)s/taskotron.log')))


 ####### RESOURCE LOCKS
diff --git a/roles/taskotron/buildmaster/tasks/main.yml b/roles/taskotron/buildmaster/tasks/main.yml
index 5244240..b9bd3ac 100644
--- a/roles/taskotron/buildmaster/tasks/main.yml
+++ b/roles/taskotron/buildmaster/tasks/main.yml
@@ -23,6 +23,17 @@
   notify:
     - restart httpd

+- name: copy artifacts httpd config
+  template: src=artifacts.conf.j2 dest=/etc/httpd/conf.d/artifacts.conf owner=root group=root
+  notify:
+    - restart httpd
+
+- name: create artifacts directory
+  file: path={{ item }} state=directory owner=buildmaster group=buildmaster mode=0775 setype=httpd_sys_content_t
+  with_items:
+  - /srv/taskotron
+  - /srv/taskotron/artifacts
+
 - name: ensure buildmaster database is created
   when: buildmaster_db_host != 'localhost'
   delegate_to: "{{ buildmaster_db_host }}"
diff --git a/roles/taskotron/buildmaster/templates/artifacts.conf.j2 b/roles/taskotron/buildmaster/templates/artifacts.conf.j2
new file mode 100644
index 0000000..268da83
--- /dev/null
+++ b/roles/taskotron/buildmaster/templates/artifacts.conf.j2
@@ -0,0 +1,6 @@
+Alias /artifacts /srv/taskotron/artifacts
+<Directory "/srv/taskotron/artifacts">
+    Options +Indexes
+    AllowOverride None
+    Require all granted
+</Directory>

@mkrizek: I wondered whether we can use Differential reviews even for projects and repos not configured in Phab (i.e. ansible scripts) and it seems we can! You can create a manual revision in Differential tab. I copied the patch above into D290, so that it's easier to comment on that code. Does that work for you?

! In #401#4876, @mkrizek wrote:
Regarding stdout/stderr, I asked at #buildbot and they said that we'd need to hack a ShellCommand to get that since buildbot does not keep separate copy of stdout/stderr. Do we want to spend time on this or having taskotron.log would suffice?

I'm afraid that's a problem, because taskotron.log only contains libtaskotron logging output, it does not contain task output, which is something we definitely want to show. For example, compare rpmlint output:
https://taskotron.fedoraproject.org/taskmaster//builders/x86_64/builds/40129/steps/runtask/logs/stdio
with taskotron.log contents:
https://taskotron.fedoraproject.org/taskmaster//builders/x86_64/builds/40129/steps/runtask/logs/taskotron.log

I'd say skip the stdout/stderr for now, but create a ticket to figure out how to extract that from buildbot in the future.

For now, I'm OK with having separate links to stderr/stdout from execution. It'd be nice to have but I'd rather avoid hacking on buildbot too much for now

! In #401#4883, @tflink wrote:
I'd say skip the stdout/stderr for now, but create a ticket to figure out how to extract that from buildbot in the future.

Ticket created: this ticket

Deployed on dev for testing. Closing.

Login to comment on this ticket.

Metadata