Splunk Alert Scripts – Automating Control

A big thanks to the members of the @SplunkDev team that were helpful and patient with my questions while I pulled this together. Thanks Guys: @gblock, @damiendallimore‎ and David Noble

In Splunk circles, you often hear about the holy grail of using Splunk to actively control other systems. It can be hard to find details or good examples on HOW to do it. I am always working on something new that deepens my technical skills. I had not previously dealt with REST APIs or Splunk alert scripts and this post is the result. Used well you can replace manual daily operations tasks; changing Splunk from a tool into a team member.

We will cover a working example of using Splunk alert results to update a Google Spreadsheet via the Drive Python SDK. Once you understand how it works, you can make you own controls of any system that supports REST API calls such as an Intrusion Prevention System to block a list of IP addresses using a scheduled Splunk alert.

We will leverage a Splunk blog post on saving credentials in a Splunk App to avoid leaving our Google credentials hard coded and exposed in the alert script. It turns out alert scripts work in the same way but it is not well documented. I built a Python class for retrieving those credentials from Splunk so you could re-use the code across many alert scripts. The scripts can all be found in the supporting GitHub repo. You will be able to use these as a framework for your own alert scripts to drive actions in other systems. I will not be stepping through the code itself as it is fairly well commented. There are plenty of moving parts to this so you need to be an experienced Splunk administrator to get it working. The benefit is that once you get one working you can just make new variants with little effort.


How this works

Let’s step through the concept of how this works.

  1. Let’s presume you have the myadmin app from my earlier post on an Admin Dashboard/App
  2. Follow the the Splunk blog post to add a setup.xml to our myadmin app default folder
  3. Use the now visible setup option for the app within Splunk
  4. Save the appropriate credentials
  5. At this point Splunk saves the encrypted credentials within the app.conf file within the Splunk application’s local folder.
  6. We will create a role that has ‘admin_all_objects’ capability and permission to all data indexes we want to run alerts for
  7. We will create a new alert script user account to own our alert knowledge objects and give it the role in our alert role
  8. You should make your scheduled alerts that run scripts to use this service account and we will discuss why shortly
  9. When a schedule search fires and has results to send to your alert script it passes arguments and a session ID via standard input/output
  10. Our alert script inherits the path to the search results from the arguments, and it uses the session ID which is keyed to the alert script user account to make a REST API call back into Splunk to request the saved credentials we want via my Python class
  11. Our alert script then uses the retrieved credentials to append the parsed search results up into the Google Spreadsheet, again we use the home-brew Google Spreadsheet Python class that I put together for this exercise

Strategic Security Decision

You could argue that leaving credentials in scripts on a locked down server is not high risk. However you can avoid a lot of compliance auditor questions if you can just say up front credentials are stored encrypted. It also avoids issues of a Unix IT admin who maintains the OS of your Splunk server from having easy access to the credentials that control your IPS and can shut down your network. It is a strategy decision you have to make. I took the complicated path of using the encrypted stored credentials.

Any account that has the ‘admin_all_objects’ capability can access the rest api endpoint to retrieve stored credentials. The downside is that capability is what it sounds like. Extensive access. It even trumps Splunk app permissions. That means your alert account can retrieve credentials saved inside apps even when it does not have app level permissions.

By default any Splunk user that is in the Admin role can access these credentials due to the capability. This gives us another Splunk administration concern. It is common to use centralized authentication via LDAP to Active Directory. If one of the AD based user accounts is put into the Splunk Admin role it can retrieve the stored credentials just by accessing the URL like:


An account that does not have the ‘admin_all_objects’ capability gets a response like the follow.
This XML file does not appear to have any style information associated with it. The document tree is shown below.
In handler 'passwords': You (user=alert-account) do not have permission to perform this operation (requires capability: admin_all_objects).

If an AD account with Admin role is compromised it would take only a quick scan by the attacker or pen-tester to hit the Rest endpoint URL above and then they have the credentials that can control your IPS or other gear controlled by Splunk.

The lesson is, ONLY use a Splunk local account with the Admin role or in making an alert service account where you create a new role and give it the ‘admin_all_objects’ capability. The secondary reason for using an alert service account is that when a user is removed from Splunk all their owned objects are removed unless the objects were shared. This includes being removed from an AD group setup via LDAP authentication in Splunk. You avoid losing your scheduled searches and alerts from staff turn over if a service account owns them. One other note about the REST API. If you use the Duo Security two factor authentication for the Splunk web interface it does NOT apply to the REST API. Use a good password, use a local Splunk account and restrict access to the REST interface via firewall rules to minimize your risk.

I had a great conversation about these issues with the Splunk Dev team. They will be turning in a feature request for a new more limited role just for password retrieval that also honors app level permissions. This will help limit exposure from an account compromise. I also suggested adding a transfer owner option for Splunk knowledge objects to help avoid accidental loss due to removal of user accounts.

Take a minute to make a new Splunk role, I called mine splunk-alert-service. I did not base it on any existing roles, as for now the ‘admin_all_objects’ capability you assign to the role cover the access it needs to run searches etc. Make sure to add any indexes you need it to be able to search or just use all non internal indexes. Then make a user account called svc-alert and put it in the group. That is the account I make and schedule my alerts under when I want to use these alert scripts.

Installing the alert scripts and wiring it together

The alert script code

You will need to download the alert script code from my GitHub repository. These scripts must be placed in the $SPLUNK_HOME/bin/scripts/ folder. You can read more details in the repo’s readme. The readme explains the functional Google Spreadsheet components and alert. I also provide an example alert script if you had your own IPS python class built.

The Python scripts, ‘alert_to_ghseet.py’ and ‘alert_script.py’ have the first line (hashbang) set to the typical Splunk Unix install path for Python. If you are doing this on Windows, OSX or a custom install location you will need to change these paths as well. If you don’t, you will find Splunk runs your scheduled search, may get results but the alert script will fail to run at all.

Sure it would never be a good idea to upload sensitive production data to a Google spreadsheet. I just prefer to provide a functional example.

Your credentials

  1. You will need a valid Google credential that will write to the spreadsheet you want the results to go to.
  2. Edit a setup.xml in the $SPLUNK_HOME/etc/apps/myadmin/default/ path
  3. Go into Splunk->Manage Apps, choose the now visible setup link by our admin app.
  4. Enter your google credentials. I recommend specifying the realm ‘googledocs’ or something similar so you can retrieve the right type of credential with the matching username.
  5. Edit in the matching realm and username in the ‘alert_to_gsheet.py’ alert script. It is a commented section: ‘# Define the source in Splunk for the stored credential’

Google Setup

  1. Create a target spreadsheet in Google Drive, my script assumes you will use a name of ‘Splunk Test’ If you use something else, be sure to put the matching name in the spradsheet_name variable in the ‘alert_to_gsheet.py’ script
  2. Give the spreadsheet the column headers: Date, IP, Post
  3. Those headings are going to match the example search we will schedule shortly. They also match the dictionary keys defined in the ‘gsheet.py’ file. If you go on to make your own spreadsheet and searches you will have to know to change the script to reference the new column headings
  4. You MUST download the Google Drive Python SDK
  5. Move BOTH the ‘atom’ and ‘gdata’ folders from the SDK /src folder into $SPLUNK_HOME/lib/Python2.7/site-packages/
  6. That is the equivalent to installing a Python package into the Splunk’s Python instance. Our home-brew Google spreadsheet class ‘gsheet.py’ requires it to work. You might have to reinstall to that location if you perform a major Splunk software update as it might replace that folder completely.

Scheduled Search

In my case I have my Web site logs going into my Splunk developer instance. I ensure they are tagged with ‘web’. below is the search which shows me which of my blog posts Splunk HQ folks are looking at. We should log in under the Splunk alert service account we mentioned earlier. If you do not get results in your search and you know you have data, check the alert role you created and check what indexes it has access to.

tag=web clientip= p=* | table _time, clientip, p | dedup _time, clientip, p | rename clientip AS src_ip | rename p AS post

We schedule the search to run as often as we want and we enable the ‘run a script’ option and specify the script name ‘alert_to_gsheet.py’

Splunk Search Results
Splunk Search Results

That should do it. If you have any questions let me know in the comments below or catch me at the Atlanta Splunk Live on March 13th.