Splunk Auto Location Lookup by Host

You might be lucky enough that you have all your log reporting hosts properly resolving to fully qualified domain names (FQDN) (e.g. splunk.cal01.georgestarcher.com). If you are really lucky part of your fqdn is a location code (e.g. cal01 = San Francisco). This can be useful if the location code is in the hostname of your wifi gear logs. You can use an auto lookup against a location table to match wifi mac address activity to lat/lon based on the equipment’s site code in it’s hostname.

First, you need to create a location csv file for the lookup to use. In our example we will use the following gs-location-lookup.csv. As a disclaimer; I do not have anything located at Splunk HQ. It is just a public address to use to demonstrate this example. We will place this file in /$SPLUNK_HOME/etc/system/local/lookups. However you could place this sort of lookup in an app that you distribute to all your search heads. If you only know the address locations of your organization sites, just use Google Maps to find out the lat/lon for the address.

Next, we define both the lookup table and the host field site code extraction in transforms.conf. We do make the assumption our site location is the component of the FQDN just before our domain name.

Last, we add the automatic lookup in our props.conf to apply to any host that has a value ending in georgestarcher.com. You probably noticed that I made the lookup command output the fields all to start with host. This is because we might do other lookups against the site code. We will know specifically this location information is tied to the host name. Not a value for siteCode that might come up in our logged data that we also wish to lookup. After all, a syslog.cal01.georgestarcher.com might collect logs that have a site code in them like cal02. Now you can search for logs based on their site location.

Here is a bonus. If you wanted to map the events based on the host site location just add this geostats command to your searches:
| geostats latfield=hostSiteLat longfield=hostSiteLon count


Splunk Updating the GeoIP Database

In the “old days” we had to install the Google Maps App for Splunk to get IP geolocation lookups. Splunk added the built in iplocation command in v6. The maxmind free database is used by both the Maps app and Splunk natively.

It is very convenient and fun to make searches like:

tag=authentication action=failure | stats count values(user) by src_ip | iplocation ip AS src_ip

The issue we run into is that IP information changes often. Spunk does not provide any automatic direct update for the database. You only seem to get a new copy when you install a version release (e.g. upgrading v6 to v6.1.2). The documentation does not even detail where the database is located within Splunk. Lastly, you might have some good reason for not upgrading a release the moment it comes out just so you can have more current ip location information. You might not want to risk breaking something in your deployment until you can test it.

Here is how you can replace the database manually. You can use the free one that Maxmind updates monthly or you might pay for the commercial copy.

  1. Download the current database from http://dev.maxmind.com/geoip/geoip2/geolite2/ You will want the city binary gzipped version.
  2. Copy it to your Splunk search head server.
  3. Expand the gizipped file to get the file GeoLite2-City.mmdb
  4. Overwrite the copy in $SPLUNK_HOME/share/

That is it. You have updated the existing copy with the currently available one. You should update it monthly or after you patch Splunk as it too will overwrite the copy in that location.


Geo-location Sunday

Today I spent a bit playing with Yahoo’s new Fire Eagle location service. It has some pretty decent privacy controls and it is taking off fast as a junction point for location aware applications. If you sign up for Fire Eagle you can get an automatic invite to Bright Kite which has good sms and email mechanisms for updating your location. It also has decent privacy controls. Such as only close friends see your exact location and everyone else gets the city.

So I tied them together and then tied Brightkite to my twitter location. While I was doing this I was surprised to see how many of my twitter followers have their exact longitude and latitude coordinates updating from their iPhone. I would wager a lot of them did not give a real thought to the privacy concerns. Or that it tells a lot of people when you are definitely not home. Worse, imagine your kids with iPhones and twitter. Raises cyber bullying to a whole new level if the bully can go straight to where they really are.

I would recommend disabling location updates and wipe the current location. Or use something like Fire Eagle/Brightkite to mask your location to a city level where it has value to you.