In Splunk circles, you often hear about the holy grail of using Splunk to actively control other systems. It can be hard to find details or good examples on HOW to do it. I am always working on something new that deepens my technical skills. I had not previously dealt with REST APIs or Splunk alert scripts and this post is the result. Used well you can replace manual daily operations tasks; changing Splunk from a tool into a team member.
We will cover a working example of using Splunk alert results to update a Google Spreadsheet via the Drive Python SDK. Once you understand how it works, you can make you own controls of any system that supports REST API calls such as an Intrusion Prevention System to block a list of IP addresses using a scheduled Splunk alert.
We will leverage a Splunk blog post on saving credentials in a Splunk App to avoid leaving our Google credentials hard coded and exposed in the alert script. It turns out alert scripts work in the same way but it is not well documented. I built a Python class for retrieving those credentials from Splunk so you could re-use the code across many alert scripts. The scripts can all be found in the supporting GitHub repo. You will be able to use these as a framework for your own alert scripts to drive actions in other systems. I will not be stepping through the code itself as it is fairly well commented. There are plenty of moving parts to this so you need to be an experienced Splunk administrator to get it working. The benefit is that once you get one working you can just make new variants with little effort.