Splunk Getting Extreme Part Two

Part one gave us a walk through of a simple anomalous search. Now we need to go over foundational knowledge about search construction when building extreme search contexts.

Comparing Search Methods

Traditional Search

This is what we did in part one. We ran a normal SPL search across regular events then used a bucket by _time and stats combination to get our statistics trend over time. This is handy when your event data is not tied to an accelerated Data Model.

Context Gen Search Pattern:

search events action=failure | bucket _time span=1h | stats count by _time, src | stats min, max etc | XS Create/Update

Search Speed:

tag=authentication action=failure

“This search has completed and has returned 8,348 results by scanning 14,842 events in 7.181 seconds”

tstats Search

Splunk is great at the “dynamic schema” aka search time extractions. This flexibility comes at the cost of speed when searching. An Accelerated Data Model is a method to give a step up in performance by building an indexed map of a limited set of fields based on that data. This is much faster to search at the trade off of only being able to specify fields that are mapped in the Data Model. Tstats means tsidx stats. It functions on the tsidx indexing files of the raw data plus it runs the equivalent to “ | datamodel X | stats Z” to catch data that is not accelerated already. This is a middle ground between accelerated and non accelerated only data searching.

Context Gen Search Pattern:

| tstats count from datamodel=…. by _time… span=1h | stats min, max etc | XS Create/Update

Search Speed:

| tstats count from datamodel=Authentication where nodename=Authentication.Failed_Authentication

“This search has completed and has returned 1 results by scanning 12,698 events in 1.331 seconds”

tstats summariesonly=true Search

Using summaries only with tstats tells Splunk to search ONLY the data buckets that have had their Data Model map acceleration build completed. It leaves off the attempt to even check for non accelerated data to return. This does mean you can miss data that has not yet been accelerated. Or you can miss data if something happens where acceleration data has to be rebuilt. This often happens in an index cluster after a rolling restart.

Ball park, the accelerated data copy is going to consume an extra 3.4x storage the size of the indexed data. We are trading that storage for speed for the index of the data. So keep that in mind when you decide how much data to accelerate.

Context Gen Search Pattern:

| tstats summariesonly=true count from datamodel=…. by _time… span=1h | stats min, max etc | XS Create/Update

Search Speed:

| tstats count from datamodel=Authentication where nodename=Authentication.Failed_Authentication

“This search has completed and has returned 1 results by scanning 10,081 events in 0.394 seconds”

Summary:

We can see significant speed increases in the progression across how we constructed the searches.

  1. Traditional Search took 7.2 seconds

  2. tstats took 1.3 seconds

  3. tstats summariesonly=true took 0.4 seconds.

This tells us that when we want to generate stats trends for Extreme Search contexts over large data sets we should use tstats, and with summariesonly=true where we can. That often makes it trivial even in multi TB/day deployments to generate and update our XS search contexts quickly, even over months of data. That is handy when you are trying to “define normal” based on the existing data. All the above speeds are just using Splunk on my late 2012 MacBook Pro. Real indexers etc will perform even better. The point is to show you the gains between the base search methods when building your XS contexts.

The next posts in our series will focus on actual search use cases and the different XS context types.

Share