#51 Remote Logging for Taskotron Machines
Closed: Invalid None Opened 9 years ago by tflink.

As we make a move towards disposable clients, it's going to become very important to gather the logs produced on those clients while the data is available so that it isn't lost when that instance is destroyed.

While the minimum is to gather logs from the Taskotron client instances, it would be a shame to ignore the other support processes in Taskotron when collecting all logs in a similar spot could be very helpful. Using buildbot to slurp client machine logs at the end of task execution would be a minimally functional solution.

This is a tracking ticket for the implementation of a remote log collection (and possibly processing/analysis) system for Taskotron.


I've been looking around at other options after talking to the beaker devs about log files and storage. It might be worth looking into using some remote, distributed fs for logging that can handle artifacts like tarballs, images etc.

The shortlist of stuff I'm interested in looking at, in order of interest/likelyhood of actually working:
* XtreemFS
* OrangeFS
* CephFS (probably not ready for use)
* OpenStack Swift (may be too basic)

I have been working on this as time allows. As an update:

  • elasticsearch + kibana look pretty good to me. yes, they're java but the searchability they provide would open a lot of doors for new features down the road.
  • it remains to be seen if the added functionality is worth the required memory/cpu overhead

  • Most of the log shippers are going to be a pain to use, at best. Fortunately, it looks like we can use rsyslog to parse the libtaskotron and testcloud logs for shipping to elasticsearch or some other log sink

Metadata Update from @tflink:
- Issue marked as depending on: #52
- Issue tagged with: infrastructure

6 years ago

Login to comment on this ticket.

Metadata