Getting started with Splunk and my favorite starter applications.

Getting Started

I am often asked how to start looking at Splunk when someone gets interested. This is the same thing I do for myself.

  1. Get the latest build of Splunk and install it on a machine you can test with. Usually this is your daily use laptop or desktop.
  2. Consider your license options. Splunk licensing is based on how much data per day you index into Splunk for searching. The free license will let you index up to 500MB per day. One thing many Splunk administrators do is to get a development license for their personal workstation. This will let you index up to 10GB per day and unlock all the enterprise features. This is great for prototyping and testing your parsing, apps etc on your workstation before moving it to your production system.
  3. Change your default admin password on Splunk once you login for the first time. The last thing you want is to be in a coffee shop and have someone poking into data you have indexed into Splunk that you might not want to share.
  4. Change the web interface to use https. Sure it is the default Splunk SSL certificate but it is better than no encryption at all. Just enable it under Settings->System Settings->General Settings

If you do not end up using a development license or your demo license runs out be sure to firewall Splunk from being accessed outside your local machine. Reference back to my someone in a coffee shop digging through your data comment.


Before I give a laundry list of applications we need to talk about what a Splunk application IS. A Splunk application is a bundle of configurations, scripts, saved searches, visualizations etc as they are objects within Splunk. Sure an application can have code in the form of scripts such as python, shell, powershell etc. But it is generally more than that and yet can be as simple as nothing but as a single configuration file. That is something I do often for defining log file monitoring for specific application types. I make an \”app\” that is a single inputs.conf file for picking up apache logs outside of /var/log on a *nix system that might be common across a cluster of web application servers. Then I assign that app to just those systems and boom, those logs are picked up and shoved into the desired index within Splunk.

There also is a convention of app \”types\” for where you have to put them. Let\’s talk about the Splunk on Splunk application for our example. You do not run into having to know this if you run Splunk with all iS\’s roles on a single server instance and just index local logs. Where it becomes the most important is when you start picking up logs from remote systems via the universal forwarder agent or start diving up Splunk across different servers for different roles. Indexers, search heads, deployment server and license server.

Splunk on Splunk also referred to as the SoS application is something you install onto your search head (the web interface where you search at) or just into Splunk if it is your single instance on your personal laptop. If you are running Splunk solely as a single instance then you do not need to worry about what I say next just to use it. SoS then has associated applications called TAs (technology add-on). TAs typically hold the data input and log parsing configuration without all the GUI and search related components to an application. You put TAs onto the systems that have the log/data you want to collect and index other than where the main application is installed. In some cases you put them onto indexers as well though this depends on the application.

There are two TAs associated with SoS.

  1. Splunk on Splunk TA for Unix and Linux – You install this onto a Splunk server running on the *nix OS. It will collect various performance, error etc information about the health of that particular Splunk server.
  2. Splunk on Splunk TA for Windows – This is the same as above but if you are running your Splunk server instances on Windows. I won\’t lie. I hate doing that. My personal feeling is always run Splunk server functions on *nix operating systems. I treat Windows servers as just log sources via the universal forwarder and I use the Windows TA for collection.

You might also see components called SAs. These are search head add-ons. Simply they are TAs that SA tells us we only need to put them on our search heads. Not indexers or remote systems where log collection occurs. A good example is the Splunk Support for Active Directory aka SA-ldapsearch written by @DaGryph. Sure we all beat him up in the Splunk IRC channel about needing to update it, but that is a whole other discussion.

There are a few apps I always install when I setup a new Splunk instance.

Informational Aid Apps

  1. Splunk Common Information Model – This application has to do with the Common Information Model that is used for abstracting original source data to types of events via eventtyping and tagging. If you follow the model and build all your searching, alerting, dashboard etc on it, then as you add new sources the data just bubbles up into it. For now you don\’t have to understand it, just install the app. You will eventually need it. It is also the base behind Splunk\’s paid apps for PCI and Enterprise Security.
  2. IP Reputation – This is a fun app that looks up ip addresses against the Honeypot Project database. I like to run my Web site logs against it. Warning: to get the dashboards populating correctly you will have to assign the eventtypes the searches are looking for to the log sources you want to map such as your apache log sourcetype (often that sourcetype is combined_access).
  3. Google Maps – But George, Splunk v6 has built in mapping now why would I need this? Both use the maxmind free database. But for visualization the Google Maps app will let you actually display counts within the location markers on the map. The built in map display won\’t do that. Also the strength of the built in mapping is you can easily embed it in dashboards. The weakness? You have to bloody edit the dashboard xml to get it to display it as a map. I had that discussion with Splunk at .conf2013. It will eventually be fixed so you don\’t have to edit xml code by hand.
  4. Timewrap written by @davidcarasso – Ok let me say this. David is not going to have to worry about bar tabs at the next .conf if he shows up. EVERYONE loves this new app. Previously you had to make a horribly complex _time search eval to get time period over time period graphs. We will be lining up to buy his drinks because it is now a simple custom search command like \”|timewrap 1w\”. See his Splunk Blog post on the new tool. Keep in mind you need to make sure your search time period window is wide enough to return all the events for the time period you want to wrap on. It is easy to search only previous week when trying to timewrap for week over week and wonder why you see only one line on the graph.

Splunk Administration Apps

  1. Splunk on Splunk – Just install this. If you are in a distributed environment make sure to install the TAs appropriately as well. When you start having any performance, curious about license usage metrics etc this is where you start. SoS started it\’s life as the internal support application that Splunk would load your collected diag results into when opening support cases. You can save yourself a support ticket sometimes by looking for yourself. There is always a booth for this application at .conf where you can ask questions etc on using it.
  2. Deployment Monitor – If you use Splunk Deployment Server to push out your apps to your log collection sources then this app is a MUST have. When folks ask me for help with their Splunk setup, if they aren\’t at least running deployment server I tell them to call me back. I hate waiting on folks to edit configs manually across LOTS of hosts. Of course if you use other configuration management like puppet etc that works too. Keep in mind Deployment Server role is a enterprise feature. Meaning you can only use it with a paid license.
  3. Firebrigade – This application is all about metrics, health etc that a Splunk admin needs to know about his indexes holding all the data. This one is fairly new. Give it time to run. It uses various scheduled searches to build the dashboards so you won\’t necessarily see a lot of data right after installing it. There is a TA-firebrigade for your indexers and you install the main application on your search head. Just as we covered for the SoS application. If are in a Splunk v5 environment you will need the older v1 firebrigade app and it\’s TA.

My next blog post will be about how I like to make a simple admin review dashboard for my Splunk instance. Sure we have Deployment Monitor and Splunk on Splunk. But for a quick daily glance each morning I prefer a custom dashboard for operational review just to know nothing has broken due to patches, stopped log sources of importance etc. Have fun trying out Splunk!