Consider a “traditional” starting splunk deployment by a security group. You get the IT group to install the universal forwarder sending you logs. Up front they aren’t interested in more than making you go away so they can work the next support ticket. Later, they find out how much access to their own logs in splunk can help operations succeed. Everything is all mixed together; your IDS, mail logs and web logs. Maybe a lot they don’t need to see.
Splunk will put data into the index named “main” by default. Everyone with a login to splunk can see this index. There is no simple move command once data is in an index to shift it into a new one.
It gets to be a bigger mess when start installing apps. Some like the *nix app put everything into an index called “os”.
You should setup different indexes as early on as possible in a new deployment. Above all use a naming convention. Sticking with the default retention period is ok. It’s six years, so you have time to shrink it later.
I follow this naming convention.
- I use underscores in index names.
- This type of index is for OS related logs so it starts with os_.
- The first and often only groupname is the IT or organizational group that owns the systems and provides the logs.
- Optionally what if you have a system developers and IT admins need to share log access. That is where I add _secondgroup name to it and send events for just those systems to this index.
Why do I follow this convention?
As mentioned indexes in Splunk are the control mechanism for access control and data retention. This is all set by index for user roles, then time periods for retention set for the index as well.
Searching with wildcards. Using this scheme you can setup a dashboard that leverages searches like
If you save that search or build it into a dashboard then if one group has access to the dashboard they see only their logs that match. The next group sees only theirs with the same dashboard. You get to see ALL events if as the security staff you have permissions to all the indexes. This also works well for eventtyping. Since eventtypes are defined by searches you can ensure an eventtype for only certain windows events run only across those indexes but ALL of them via the wildcard.
The downside shows up when you are not using the default index and you are new to splunk. There is a tendency to install some given Splunk app and expect it to just show data. Often these apps are coded to search just default indexes or their own. You will have to dig into their code and find where you have to replace the app searches etc with your wildcard naming scheme to get it wired up. It is still worth the effort and saves you from a lot of pain as your deployment matures.
For more about indexing be sure to read through the Splunk manual on Managing Indexes and Clusters.