The State of Open Source Logging

Create your own real time, interactive, searchable and sharable dashboards and analytics interfaces on top of Elasticsearch (http://elasticsearch.org), and do it from the comfort of a browser. Kibana 3 (http://three.kibana.org) is an open source, Apache licensed, project for creating analytics interfaces to machine generated (or people generated!) data.

As the speed of machine generated data accumulation increases as does the importance of having up to the moment access to real time (or near real time) analytics of that data. Dashboards convey huge amounts of information in a concise, aggregated, meaningful and actionable way. And we're going to build a bunch of them!

The tool chain

We'll discuss the pluses and minuses of existing tools, what they're getting right and what we've learned from using them. We'll spend time talking about machine generated data: Logs and events from infrastructure and the tool chain (e.g. Logstash http://logstash.net) for collecting them. Shay Banon, founder of Elasticsearch, will join me to discuss why Elasticsearch works so well for storing and searching machine generated data.

Live demos

After that we'll move to real world use cases and how we can easily build interfaces to machine data that suit operations, development and management in a matter of minutes. We'll also play with some fun data: social feeds and our very own fictional company: MonkeyCo, who tracks the progress of an infinite number of monkeys distributed around the world, producing the collected works of Shakespeare

This talk will skew heavily towards live demonstration. We'll build interfaces from scratch on top of multiple real time data sets and show how they can be saved, searched and shared within an organization.

Slides: 

Logging.pdf