George Starcher

November 13, 2009: 2:27 pm: George StarcherAdmin Tricks

Building a logging VM – syslog-ng and Splunk

Recently I wanted to build a log collection virtual machine. I settled on a combination of syslog-ng and splunk. Syslog-ng lets you do filtering, message rewriting and routing to multiple destination types. Splunk v4 gives you a nice ability to search the gathered logs. So you can follow my two documents. The roll your own covers the building of the vm. The getting started covers doing the last setup tweak to use and collect certain event types I decided would make a good stating example set.

We use Ubuntu sever 32bit 9.10 with syslog-ng v3 and splunk v4 in this tutorial. I built mine in vmware fusion on my mac. But you should be able to adapt to your own box/virtualization of choice.

http://twitter.com/cabodj Dave Jones

Excellent documentation! So I assume the goal of this configuration was to be able to forward events via syslog or Snare from the remote servers rather than installing Splunk in forwarding mode?

Have you run into any limitations by taking this approach vs Splunk as a forwarder?
http://www.georgestarcher.com georgestarcher

I mainly wanted the syslog-ng for the rewriting to compensate on forwarded syslog events where you cannot run by policy or get to work the spoof based forwarding. However the advantage to hitting it first is the filtering and other log destination options you have should you suddenly want to do that. It becomes just changing syslog-ng config files instead of having to redo all your forwarding setup to wedge it in later. So yeah splunk itself can receive all those events on any ports you define. But you wont get the extra routing, filtering and relogging with precision.

I do actually have snort with splunk on the same box. Then I just make that splunk forward but not index to my real splunk vm I want to do my searches at.

In the end you can get way more complex later. But I wanted a nice solid getting started framework for anyone else fighting the learning curve. From here we can get into forwarding logs from macs, forwarding snort via a second splunk install etc.
shelston

George -

Simon from Splunk here. Just wanted to clarify a few points. The first is that Splunk can in fact index Windows Event Logs and any sort of “perfmon” metrics via WMI out of the box and very easily. Splunk can also index and “tail” a Windows Registry but that requires Splunk to be running on the box itself.

Second, tegarding Syslog-ng. Splunk can pretty much do everything you describe. Splunk can obviously index Syslog directly on UDP/514. It can also send Syslog back out to other systems if you wanted to. See: http://www.splunk.com/base/Documentation/4.0.6/…

Splunk can also route data in other formats to other kinds of systems/apps based on pretty much anything in the data. You specific the “sourcetype” and its routing parameters and voila! Check out: http://www.splunk.com/base/Documentation/4.0.6/…

Combining this with Splunk deployment server (centralized config manager) would probably simplify your deployment by leaps and bounds. Remember that our licensing is based on how much data you index in a 24 hour period. We don't count servers, agents, data sources, etc. This is just about leveraging Splunk out of the box awesomeness! ;)

Hope this helps.
Simon
http://www.georgestarcher.com georgestarcher

Hi Simon,

I knew about the wmi. But in some scenarios folks might not want to get into admin level credentials and using snare to send via syslog eliminates that. Plus it further controls what is sent and thus how much data is indexed by splunk.

I still think using syslog-ng is useful since it can rewrite messages before they get indexed by Splunk and forward to sql. After someone gets comfortable with splunk I can see digging into the routing and forwarding syslog native to it for more options.

Thanks for the feedback. I really made this to help folks completely unfamiliar with logging have a starting point on both syslog-ng and splunk. Plus target a simple solution for smaller shops.
John Telford

Sounds great. Would you be willing to share your Logging VM?
http://www.georgestarcher.com georgestarcher

I am not comfortable redistributing software I dont have a right too like splunk. But follow the PDF and you can whip up the vms really quickly from scratch.
John Telford

Sounds great. Would you be willing to share your Logging VM?
http://www.georgestarcher.com georgestarcher

I am not comfortable redistributing software I dont have a right too like splunk. But follow the PDF and you can whip up the vms really quickly from scratch.
Whatupjack

Hello George,

Great write up on this config. I’ve been trying to get syslog-ng to work on a CentOS 5.6 install. your write up is what I’ve been looking for — I’ve been very confused onto what I need to configure. I do have one question though, regarding the change to the syslog-ng.conf to point to the splunk.conf file. The document states to include “/opt/syslog-ng/etc/
splunk.conf”. Could you please be so kind to show how exactly it should be included in the Source section.

Thanks,
Jack
http://www.georgestarcher.com georgestarcher

You are misreading it. You should add the exact text: include “/opt/syslog-ng/etc/splunk.conf” as a line under the sources section in the syslog-ng.conf file

This is like an include header in a snort rules or even a programming language file in c where you do something like

Thoughts, tricks and tirades on technology today.

Archived Entry

Categories

Archives

About Me

Favorite Links