Since yesterday depcheck crashes every time. Example on dev: https://taskotron-dev.fedoraproject.org/taskmaster/builders/x86_64/builds/2737/steps/runtask/logs/stdio and stg: https://taskotron.stg.fedoraproject.org/taskmaster/builders/x86_64/builds/9948/steps/runtask/logs/stdio
However, I received no emails about it, even though these things should be sent to qa-taskotron-admin-members@fedoraproject.org . The email error reporting seems to be broken.
I just started looking into this and it appears to be a configuration issue on the buildmaster. I'm seeing the following traceback in the master logs:
2014-08-12 08:37:58+0000 [-] Unhandled Error Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/buildbot/status/build.py", line 307, in buildFinished w.callback(self) File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 368, in callback self._startRunCallbacks(result) File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 464, in _startRunCallbacks self._runCallbacks() File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 551, in _runCallbacks current.result = callback(current.result, *args, **kw) --- <exception caught here> --- File "/usr/lib/python2.7/site-packages/buildbot/status/builder.py", line 535, in _buildFinished w.buildFinished(name, s, results) File "/usr/lib/python2.7/site-packages/buildbot/status/mail.py", line 468, in buildFinished return self.buildMessage(name, [build], results) File "/usr/lib/python2.7/site-packages/buildbot/status/mail.py", line 700, in buildMessage results, builds, patches, logs) File "/usr/lib/python2.7/site-packages/buildbot/status/mail.py", line 598, in createEmail 'builder': builderName, exceptions.KeyError: 'prop:taskname'
@mkrizek, does this look obvious to you? I can take a look if you're otherwise busy but this might be easier for you to look at.
diff --git a/roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2 b/roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2 index 8882506..a28a5b8 100644 --- a/roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2 +++ b/roles/taskotron/buildmaster-configure/templates/taskotron.master.cfg.j2 @@ -278,7 +278,7 @@ def html_message_formatter(mode, name, build, results, master_status): mn = MailNotifier(fromaddr='taskotron@fedoraproject.org', sendToInterestedUsers=False, - subject="%(prop:taskname) %(result)s in %(projectName)s on %(builder)s", + subject="%(prop:taskname)s %(result)s in %(projectName)s on %(builder)s", mode=('failing', 'exception', 'warnings'), extraRecipients=['qa-taskotron-admin-members@fedoraproject.org'], relayhost="bastion.phx2.fedoraproject.org",
Does this look like a fix?
yeah, that looks like it could fix the problem. Let's give it a try on dev
The fix has been deployed on dev.
So that didn't work. Deploying other possible fix to dev...
So the problem with putting taskname into the mail subject is that we need to use Interpolate('%(prop:taskname)s') to generate taskname into the subject="%(result)s in %(projectName)s on %(builder)s" but Interpolate doesn't have str(). @tflink any ideas?
Interpolate('%(prop:taskname)s')
subject="%(result)s in %(projectName)s on %(builder)s"
Interpolate
str()
Can you post some more specific TBs that show the issue?
Traceback (most recent call last): File "/bin/buildbot", line 4, in <module> runner.run() File "/usr/lib/python2.7/site-packages/buildbot/scripts/runner.py", line 731, in run sys.exit(subcommandFunction(subconfig)) File "/usr/lib/python2.7/site-packages/buildbot/scripts/checkconfig.py", line 53, in checkconfig return _loadConfig(basedir=basedir, configFile=configFile, quiet=quiet) File "/usr/lib/python2.7/site-packages/buildbot/scripts/checkconfig.py", line 24, in _loadConfig basedir, configFile) --- <exception caught here> --- File "/usr/lib/python2.7/site-packages/buildbot/config.py", line 149, in loadConfig exec f in localDict File "/home/buildmaster/master/master.cfg", line 266, in <module> subject=Interpolate("%(prop:taskname)s %(result)s in %(projectName)s on %(builder)s"), File "/usr/lib/python2.7/site-packages/buildbot/process/properties.py", line 416, in __init__ self._parse(fmtstring) File "/usr/lib/python2.7/site-packages/buildbot/process/properties.py", line 539, in _parse d, kw, repl = self._parseSubstitution(key) exceptions.TypeError: 'NoneType' object is not iterable Configuration Errors: invalid Interpolate substitution without selector 'projectName' error while parsing config file: 'NoneType' object is not iterable (traceback in logfile)
So we found no way to easily include a taskname in the mail subject. I revert the change and deployed on dev and stg, email reports should work now.
Since my inbox is now overflowing with reports again, I think that we can close this. The issue about having more/better information in the failure reports can be filed in another ticket
Metadata Update from @kparal: - Issue tagged with: infrastructure
Login to comment on this ticket.