Splunk Bringing in Data – Minecraft the Model Method

I like to take more than traditional IT and security logs into Splunk. You can enhance your production data in creative ways. I am a firm believer the best way to learn is to practice on something out of the norm. The game Minecraft is a fun source of log data if you find out how to extract the information. I am a bit of a closet Minecraft Let’s Play video fan. At the last Splunk user conference the gaming room was setup with a local Minecraft server and logging to Splunk. That was the public debut of the Splunk Minecraft App. It was fun to see the live information about what types of resources had been collected etc.

The Splunk Minecraft App relies on a plugin for a variant build of Minecraft called Bukkit, which makes it easy to run Minecraft with modifications. The problem is that the Log To Splunk plugin has not been updated to keep up with java versions. Yeah, Minecraft is written in java. Over the holiday I wanted to play with some minecraft logs in Splunk v6 so I had to find another solution. After all, it is a good way to practice on parsing logs, event typing and tagging them. There is an old blog post that predates the Splunk Minecraft App that tells how to use a Minecraft plugin called PlayerLogger to do this. You can find the original post over on Robert Jordan’s blog.

Bringing in the Data

Robert’s post is enough to get logs collecting on your Minecraft server but he does not cover parsing the logs into Splunk. Let’s talk about those steps.

  1. Follow the steps on Robert’s blog about Bukkit and PlayerLogger.

  2. We need to make one change to step 3 on his blog about modifying the config file for PlayerLogger. Under the Log: section change the DateFormat line to the following.
    DateFormat: yyyy-MM-dd HH:mm:ss

That will give us a log timestamp in a ISO Timestamp format that Splunk will automatically recognize. That saves us the trouble of defining where in the log events the timestamp occurs and how to map it out. if you question the value of the ISO format be sure to review the obligatory XKCD ISO Date reference graphic.

XKCD ISO 8601

  1. Setup Splunk to collect the logs that PlayerLogger outputs. Now you can install Splunk on the same machine or use the Universal Forwarder. I went with the Splunk UF since I am using an old 1st generation Intel Mac mini as my Minecraft Bukkit server. What follows are my inputs.conf stanzas to pickup the main Minecraft server logs and the plugin logs. You also need to make a new index on your Splunk indexer to receive the logs. I called mine app_minecraft. You will notice I did not spend any time parsing the primary Minecraft server logs (minecraft_app), only the ones from the logging plugin (minecraft_play). That is because it has the same events plus all the user in game activity. I still wanted to collect the primary logs just in case of any system errors etc I might need to search through later. Note that I installed bukkit in the Applications folder on my old Mac. So that is where the monitor path is coming from.

inputs.conf

4. We need to define the source type and field extractions for the logs coming from PlayerLogger. Here are my props.conf and transforms.conf stanzas. We use several calculated (eval) fields in the sourcetype (props) definition for our logs coming in. My regular expressions for the field extractions could be more elegant but they work in almost all cases common in the logs.

props.conf

transforms.conf

  1. I am a big fan of the Splunk Common Information Model so in addition to the fields we did above here are my eventtypes.conf and tags.conf stanzas. We don’t have anything for authentication failures since Mojang handles the user accounts. There is still more room for assigning event types but I have not taken the time to explore the blacklist, whitelist functions to see what one of those events looks like. Sounds like a good practice exercise for you to enhance all this.

eventtypes.conf

tags.conf

Example Searches

We took the time above to eventtype and tag log data. This makes it much easier to search without having to be the Splunk administrator who knows what indexes, sourcetypes etc are defined. We can just search mostly via tags. That is the point of the Common Information Model, to abstract out log details to a language we can remember and work with. If you have questions on Splunk you can often find myself and those who love to help over in the Splunk IRC Channel.

Active Logged in Users

Search all time or at least a period longer than you expect users to stay logged in.
tag=minecraft tag=session | transaction keepevicted=true startswith=Joined endswith=Quit user | search duration=0 NOT action=Quit | stats dc(user)

Users (past 24 hours)

tag=minecraft tag=session earliest=–24h latest=now | transaction keepevicted=true startswith=Joined endswith=Quit user | stats dc(user)

Where are the Users Logging In From

tag=minecraft tag=session | iplocation src_ip | stats values(user) AS Users by Country,Region,City

Total Play Time (all time)

This makes a good pie chart for a small number of players.
tag=minecraft tag=session | transaction keepevicted=true startswith=Joined endswith=Quit user | eval duration=round(duration/60) | eval minutes=duration | eval duration=if(duration=0,“active”,duration+“ mins”) | search NOT (duration=active action=Quit) | table _time, user, minutes | stats sum(minutes) as TotalTime by user

Block Activity

tag=minecraft block=* | timechart span=1h count(block) by operation

Blocks Used Over Time

Least Placed Blocks

tag=minecraft block=* | rare block
RareBlocks

Most Placed Blocks

tag=minecraft operation=Placed block=* | top block
TopBlocks

Share