Splunk – Metrics and a Poor Man’s mcollect

Strangely the metrics feature in Splunk v7 is missing a command you would think we would have — mcollect. A command to take search results and collect it into a metric store index. Similar but way better for tracking things like license or index size over time using the normal stats and collect commands together.

Duane and I were hanging out and decided to make a poor man’s mcollect.

  1. Make a metric index, mine is called testmetrics
  2. Using inputs.conf in the app context of your choice setup a batch input
  3. Make your search to format results compatible with the metrics_csv sourcetype
  4. Graph your mstats for fun!

Inputs.conf Example:

Note we used a name convention of metrics_testing* so that we could easily target only the csv files we will be exporting using outputcsv soon. Start with the word metrics then second word is related to the index we are going to put it into then a wildcard. You can see we use the sinkhole policy to ensure the file is removed after indexing to avoid filling the disk if you do a lot of metrics this way.

Note that indexing data into a metrics index is counted as 150 bytes per metric event against your Splunk License.

Metrics Gen Search:

In this search we gather the index total size per index, reformat fields to match metrics_csv format.

Then we use an Evil subsearch trick to generate a filename with a timestamp to outputcsv to.

You could easily schedule this search to collect your stats hourly, daily etc. And adapt all this for license usage.

Viewing your New Stats:

The graphing search:

And a sample from our nice new graph!

splunk mstats