Splunk and Apple Swift for Alert Scripting

This week, I attended my first Nashville Cocoaheads meeting in a few years. Cocoaheads is basically an Apple programming club. The topic was “Scripting with Swift” It was a great presentation by Blake Merryman. Blake works for Griffin Technology and they were our awesome hosts for the meeting. Griffin is one of my favorite accessory makers. Plus there are right here in Nashville.

Of course I immediately wondered. If I can treat Apple Swift as a shell scripting language… Splunk alerts written in Apple Swift! Oh yeah why not?!?

And… The short answer is that it works for Splunk running on OSX. This means you could make data sent from Splunk via alerts to OSX code that has full native library access to anything on the Mac.

Here is a very simple example thanks again to Blake’s example code.

Alert Script Setup

You must have Xcode installed on the Mac as well as Splunk. Xcode provides the Swift language installation.

You need to make a shell wrapper script in the $SPLUNK_HOME/bin/scripts/ folder. Splunk won’t be able to call a .swift script directly. You will need to add the executable flag to both script files. Then you simply setup the desired Splunk alert to call alertSwift.sh which is just a shell command wrapper script to call the Swift script. The key on the swift script is the hashbang that tells the system how to execute Swift.

Our Swift code example below is a simple call of Apple Script to make the Mac say “Splunk Alert.” This is a very simple example. But it shows that if you use all the normal tricks of pulling in the alert script arguments you could pull in a search results zip file and take actions in Mac OSX based on the data. It could be anything including Notification Center alerts. Enjoy the possibilities.

Alert Script Code

Example alertSwift.sh: (Make sure you do a chmod +x alertSwift.sh)

Example alertSwift.swift: (Make sure you do a chmod +x alertSwift.swift)

Splunk sessionKeys and Python Indentions

On sessionKeys to the Kingdom:

I started making a scripted input app to pull in logs from the LastPass Enterprise API. Everything was progressing nicely until I found I could not retrieve the encrypted API key from Splunk where I saved it. I was going to be smart and re-use my credentialsFromSplunk.py class that I created for alert scripts. That is when I beat my head on the wall. Scripted Inputs get sent a sessionKey if you set the passAuth value for your script in the inputs.conf stanza. Stdin is also how sessionKeys are sent to alert_scripts. So I figured my existing code would work great.

I kept getting authentication failures on the API. It turns out, I had not put in a logDebug event for the original sessionKey as it came in. So I had not noticed an inconsistency. SessionKeys sent for scripted inputs do NOT have the “sessionKey=” string at the front of the key sent by Splunk to alert scripts. Thus re-using my existing code that clips off those eleven characters broke the sessionKey value. So I share it here in case you are learning new Splunk features that depend on the sessionKey. Check your original values if you get authentication errors on the API. The sessionKey could contain extra text you have to remove or is URL encoded.

Remember that you can submit feature enhancement requests through the Splunk support portal. It is important when you find such inconsistencies to submit it so Splunk developers can eventually fix it. I did on this one.

On Indention in Python Scripts:

I used to give no thought to using tabs in my Python script as long as it indented right. Now I am firmly in the “four spaces” in place of a tab camp. I have gone through and updated all the code in my git repo to be four spaces based. I found I was getting unexplained bugs where code may or may not execute when expected. It turns out it was all indention related. Code was running only when an IF stanza above it was true. A pesky hidden tab was causing it. So if you edit in vi/vim make sure to change your config to use four spaces. I found these two great links on doing that and putting it into your .vimrc file for persistance.

Splunk a DNS Lookup for Abuse Contacts

Let’s follow up on our DNS theme of the last post. I have used my alert scripting to block attackers in the past such as those scanning heavily against SSH. Now I want to start considering emulating the complaint notification one can get from using fail2ban. So let’s start with just adding a simple external command lookup for getting the abuse contact for a given IP address. We will actually use the method found in the fail2ban complain module. So big thanks to them!

We want to have a search like this:
tag=authentication action=failure | stats count values(user) by src_ip | lookup abuseLookup ip AS src_ip

Once you add the transforms and python script below the command should work in Splunk. Keep in mind like the dnsLookup this has to happen on any search heads that will need it. I also have not yet worked on making this handle ipv6 which abusix.com can do with the lookups. The new abuseLookup will return a field to your events called abusecontact. Then you can use that how you want in reporting events.

First edit your transforms.conf to add this stanza:

Now create the python script abuseLookup.py in $SPLUNK_HOME/etc/system/bin/

Splunk Alert Scripts – Automating Control

A big thanks to the members of the @SplunkDev team that were helpful and patient with my questions while I pulled this together. Thanks Guys: @gblock, @damiendallimore‎ and David Noble

In Splunk circles, you often hear about the holy grail of using Splunk to actively control other systems. It can be hard to find details or good examples on HOW to do it. I am always working on something new that deepens my technical skills. I had not previously dealt with REST APIs or Splunk alert scripts and this post is the result. Used well you can replace manual daily operations tasks; changing Splunk from a tool into a team member.

We will cover a working example of using Splunk alert results to update a Google Spreadsheet via the Drive Python SDK. Once you understand how it works, you can make you own controls of any system that supports REST API calls such as an Intrusion Prevention System to block a list of IP addresses using a scheduled Splunk alert.

We will leverage a Splunk blog post on saving credentials in a Splunk App to avoid leaving our Google credentials hard coded and exposed in the alert script. It turns out alert scripts work in the same way but it is not well documented. I built a Python class for retrieving those credentials from Splunk so you could re-use the code across many alert scripts. The scripts can all be found in the supporting GitHub repo. You will be able to use these as a framework for your own alert scripts to drive actions in other systems. I will not be stepping through the code itself as it is fairly well commented. There are plenty of moving parts to this so you need to be an experienced Splunk administrator to get it working. The benefit is that once you get one working you can just make new variants with little effort.


Continue reading “Splunk Alert Scripts – Automating Control”

Splunk Presenting Data in statusboard on iPad

Splunk is a great tool for digging into data and presenting the results. Sometimes, you just want a status board of results that comes to you without having to log into a web application. A wonderful app for this is the iPad app statusboard by Panic software.

You always could create a panel on your statusboard that links to a URL of a file for presentation. However, this means your data is not protected by authentication. Panic added Dropbox support so you can now make a panel that pulls from a csv or json file. You can also airplay to an AppleTV or direct connect the iPad to a TV to present the dashboard on a large display.

In this post I will cover how I combined a Splunk alert script in python, dropbox and statusboard to get the result below. I am displaying the number of failed login attempts against my wordpress blog by country code for the previous 7 days. Keep in mind this is a Splunk instance running on my laptop with minimally sensitive information. I would never run dropbox directly on a work related production Splunk server. An alternative method would be to run a scheduled script that pulls the results out of Splunk via the REST api and write it out to a csv in the dropbox folder. I will do that version of this post in the future.

WordPress Logins Statusboard

Continue reading “Splunk Presenting Data in statusboard on iPad”

Splunk Alert Script – OSX Notification Center

I want to start making some custom alert scripts. As usual, I like to practice by using a live example. I have SSH remote access and Apache enabled on my laptop. When at work I keep a map up in Splunk on my laptop showing the source ip location of any attempts to connect to my laptop. If you start beating on my laptop it results in an instant ban hammer in the network IPS.

I sometimes miss seeing the map updates when busy. If I had an alert history that is quickly accessible it would be easier to handle the scanning systems. I decided on this alert to test the hits on apache that runs every 15 minutes. These logs just happen to go into an index called os_osx. I tagged the combined_access source type as “web”.

index=os_osx tag=web | stats count by clientip

Now the fun part. I am working on my python skills so I did the alert script in python. This required me to call the OSX shell command osascript in order to execute the Apple Script that generates the actual Notification Center message. It took a minute of experimentation to get the right combination of escaped quotes to build the Apple Script command.

We get a result like this:


And here is the alert script that I saved as osx-alert.py in the /Applications/splunk/bin/scripts folder on my laptop. That is the script I chose to call on the search above when saved as an alert.

Splunking the Laundry

I have been working on learning Python lately.  One of the best ways to learn is to pick small goals and achieve them.

The laundry room of my apartment building uses  a service called Laundryview.com to let people see the status of the washer and dryer units including time remaining.  I have my raspberrypi handy.  So I set out to put together a python script to scrape the machine status every fifteen minutes and push the data into splunkstorm.com.  This is so I can actually trend the machine usage to determine what days of the week and times are most available.   Plus I wanted to see if I could do something new. Below is a sample graph from splunkstorm showing the in use pattern for the washing machines.

Washing Machine status graph

If you want to see the python script just click more.  Warning it is down and dirty.  I could have made things more elegant but it works and I have not had time to polish it up.  You will see I use lxml to parse the mobile version of the site for the machine status from a table.

Continue reading “Splunking the Laundry”

Need to open a password protected zip on Mac OSX?

Recently we were sent a password protected zip file at work. If you have ever tried to double click open a protected zip file you know that you will get an error not a password prompt.

If we ignore third party applications to open these files we are left with running the unzip command from terminal. You can use the option “-P password” where you replace the word password with actual password used to protect the file. An example command might look like:

This may have been good enough for me. But we have less technically inclined folks in our group who needed access to the provided files. So I made an Automator that everyone in the group could reuse for even future files. The Automator needed several features.

  1. Provide a GUI browser selection box to choose the protected zip file.
  2. Prompt for the password needed to unzip the file.
  3. Send the content to the user’s desktop regardless of whom ran it.

Here is how we build the Automator. The key components are the use of variables within Automator and a shell script object that takes the file and password as arguments that were provided by the user.

Continue reading “Need to open a password protected zip on Mac OSX?”