Jeff Longland

Relax, don't worry – have a home brew!

#OSCON – Wrangling Logs with Logstash and ElasticSearch

leave a comment »

  • Nate Jones, David Castro from Media Temple
  • Per week, approximately 1.8 TB uncompressed log data for their mail servers
  • Need to make log access easy for front-line support and ops team
  • Architecture: logstash-agent on each host pushes to RabbitMQ, then to elasticsearch, and searched using Kibana
  • logstash groks the logs, then mutates to JSON
  • Prebuilt patterns allow you to extract more than you’d necessarily get with regex
  • elasticsearch head helps monitor performance of the shards and allows you to browse the data directly, but it’s not front-line support friendly
  • Kibana provides a user friendly front-end for elasticsearch
  • logstash can output gelf for Graylog2.  But not the best approach for Media Temple since they use RHEL and package dependencies can be a PITA.
  • Kibana support streaming, allowing for real-times searching/monitoring – ie. as a user is doing something, support can be watching the events come in.
  • logstash hasn’t done much to reduce the size of logs, in fact, they’ve increased by ~50% – but it’s worth it for all the benefits.
  • No more grepping logs for hours. Couple of minutes and you have everything you need.
  • Using statsd to push log metrics into Graphite for visualization
  • Keep ~7 days data (for mail).  Compressed copies of raw logs, should they ever need something historical.
  • Have kindly provided a VM to kickstart everyone with logstash, rabbitmq, kibana, etc: logwrangler.mtcode.com
Advertisements

Written by jlongland

July 20, 2012 at 11:01 am

Posted in OSCON 2012

Tagged with

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: