#149 Add buildstep to cat relevant sections of taskotron.log after task completion
Closed: Fixed None Opened 9 years ago by tflink.

As we start sending more information to taskotron.log instead of stdout/stderr, there are going to be holes in the information presented to users.

To work around this, modify the builders in buildbot to cat the sections of taskotron.log which were created during task execution.

There have been a couple methods suggested, more ideas welcome:

=== Use /tmp/ logs ===
By default, taskotron will log to /var/tmp if /var/log/taskotron is not writable. We could prevent /var/log/taskotron from being writable and just cat the logs from /var/tmp at the end of the task

Advantages:
don't have to deal with time
don't need to trim log before execution

Disadvantages:
have to find the log file
taskotron logs are spread out over many /var/tmp directories

=== Use section of /var/log/taskotron/taskotron.log ===
This would note the start time and end time of the task. Using those times (plus buffer on each side), /var/log/taskotron/taskotron.log would be parsed and any log messages in the selected time period would be printed.

Advantages:
simple apperance
easier to reconstruct and debug later

Disadvantages:
have to know start and end time of tasks, make it accessible to build step
potential to blend log messages and confuse users

=== Trim /var/log/taskotron/taskotron.log ===
Instead of messing with log parsing, clear the log before task execution and cat the entire thing. To prevent data loss, we could append the logs to be cleared to the end of a more permanent log file or change logging configuration to use multiple file logs.

Advantages:
simple to implement
difficult to mix up or blend logs from different tasks.

Disadvantages:
potential for data loss
doesn't provide window before/after (probably not a big deal for our current use cases)


I think that the trim method would be easiest to implement and the scanning-with-buffer method might be the best option. I don't really like the /var/tmp option here.

Considering our time constraints, I'm leaning toward the trim method with either multiple log copies or appending data to a more permanent log before deletion.

Once we have disposable test clients, the taskotron.log will be always empty (missing) on task start. So that's the same approach that the //trim method// emulates for persistent clients.

@tflink, can you elaborate on its disadvantages? I don't really understand any of this:

potential for data loss
doesn't provide window before/after (probably not a big deal for our current use cases)

Since I don't understand the disadvantages, I also think this is the best approach to take :-)

Also, I don't see the reason behind appending the current log to some permanent location before trimming it. Of course, there seems to be no harm in it.

! In #230#8, @kparal wrote:
Once we have disposable test clients, the taskotron.log will be always empty (missing) on task start. So that's the same approach that the //trim method// emulates for persistent clients.

We're probably going to need some sort of centralized logging solution before moving to disposable clients and that's somewhat likely to make this buildstep less useful.

@tflink, can you elaborate on its disadvantages? I don't really understand any of this:

potential for data loss
doesn't provide window before/after (probably not a big deal for our current use cases)

Since I don't understand the disadvantages, I also think this is the best approach to take :-)

I might be misunderstanding the upcoming taskotron.log changes but I'm under the impression that there will be data in taskotron.log which is not stored anywhere else. My concern is in triaging failures - specifically, debugging taskotron when something goes wrong. By deleting the contents of taskotron.log before/after every task, getting a somewhat consistent stream of information output from taskotron will be more difficult.

Granted, I expect that most issues will be debuggable without needing a non-interrupted taskotron.log but I'd like to avoid throwing out that information or splitting it up if reasonably possible.

! In #230#9, @kparal wrote:
Also, I don't see the reason behind appending the current log to some permanent location before trimming it. Of course, there seems to be no harm in it.

It preserves a more coherant stream of log information. While I don't see it as absolutely critical, I don't see a good reason not to preserve it as a coherent log.

I might be misunderstanding the upcoming taskotron.log changes but I'm under the impression that there will be data in taskotron.log which is not stored anywhere else.

taskotron.log will differ from the check standard output, it will contain debug messages not displayed on the stdout by default.
At the end of the task, the whole log file will be copied to buildbot logs by catting it (this ticket).
After that, there should be no information in taskotron.log that's not available somewhere else (in buildbot logs).

My concern is in triaging failures - specifically, debugging taskotron when something goes wrong.

That probably depends on buildbot. If the check crashes, is buildbot going to execute the following steps (catting taskotron.log)? If it is, then we should be safe - no information lost. If the crashed step would also mean that no following steps are performed, then yes, we wouldn't retrieve the debug log messages and that would be very unfortunate. If this is the case, your suggestions make a lot of sense. But I hoped that we'd be able to run the cat taskotron.log step always, no matter whether the check exits correctly or crashes. We need to check Buildbot's behavior/configuration options.

It preserves a more coherant stream of log information. While I don't see it as absolutely critical, I don't see a good reason not to preserve it as a coherent log.

Yes, I don't see any harm, I just wasn't sure why it was requested. Makes more sense now. Let's hope Buildbot is able to execute the following steps always and make our life easier.

pull request accepted and merged into ansible-playbooks. closing this task

Login to comment on this ticket.

Metadata