Splunk uLimits and You

Most folks are familiar with the concept of file descriptors in Unix/Linux. It gets mentioned in the Splunk docs for system requirements under the section “Considerations regarding file descriptor limits (FDs) on *nix systems” and for troubleshooting.

I run a very high volume index cluster on a daily basis. Complete with Splunk Enterprise Security. One thing I have seen is if you have timestamps off you can get a VERY LARGE number of buckets for low overall raw data size. If you see nearly 10000 buckets for only several hundred GB of data then you have that problem. Keep in mind that is a lot of file descriptors potentially in use. You should check your incoming logs and you will likely find some nasty multi line log file having a line breaking issue where some large integer is getting parsed as an epoch time and causing buckets with timestamps way back in time.

It got me thinking about the number of open files though. Especially, when also being concerned with all the buckets for data model accelerations to be built for supporting the Enterprise Security application. Maybe FD limits have been interfering with my data model acceleration bucket builds.

Then we had a couple of indexers spontaneously crash their splunkd processes. With an error indicating file descriptor limit problems.

I discussed it with my main Splunk partner in crime, Duane Waddle. He explained that if a process starts on it’s own without a user session that Linux might not honor ulimits from limits.conf. So even though we had done the right things accounting for ulimits, Transparent Huge Pages etc that we were still likely getting hosed.

Such as this example from /etc/security/limits.conf using a section like below for a high volume indexer in a cluster:

You might be getting the 4096 default if Splunk is kicking off via the enable boot-start option.

You can test this by logging into your server then do the following:

Check the results looking for the Max open files.

Duane suggested editing the Splunk init file. My coworker Matt Uebel ran with that and came up with the follow quick commands to make that edit. Use the following commands substituting your desired limits values.

Now when your system fully reboots and Splunk starts via enable-bootstart without a user session you should still get the desired ulimits values.


Review – M3D Micro 3D Printer

I wanted a 3D printer for a while. So, I have been watching Noe and Pedro Ruiz with Adafruit. They have a great show on the Adafruit YouTube channel called 3D Thursdays. Originally, I was holding out for the Flash Forge Creator Pro. Then Adafruit added the retail version of the M3D to their store. At just under $500, that fit a gift card I had been holding onto. It was also a simpler printer for someone getting started.

I am a digital guy. So this whole real world 3D printing thing is new to me beyond watching the Adafruit team. What follows are the things I ran into from the point of view of a complete rookie in this area. I had to have concepts and terms click that experienced folks with printers take for granted.

Buying the printer

I mentioned I had a gift card. It was the typical visa type card. It was enough to cover the printer. But, I wanted some other items from Adafruit when I ordered. The purchasing system will not let you specify multiple credit cards and how much to apply to which card. Some creativity let me work around that limitation. I purchased an Adafruit gift certificate with the Visa gift card. It promptly came via email. I then applied that “to my Adafruit account creating a store balance. That allowed the printer to deduct from that balance and overflow costs to my credit card for the extra items like filament, etc just like I wanted. One of those items was a good digital calipers tool. Critical later when you want to print your own items.

I also made sure to wait till a Thursday to order. This let me use the 3D Thursday discount code they give out on the show good to midnight. Awesome that it saves me money, but it also lets Adafruit know the sale is because of Noe and Pedro’s hard work. PS saving me money really meant I ordered more to compensate. I had a budget I had set so I used it all.

Out of the box

There are plenty of unboxing videos out there for the micro 3D. It was well packaged. Just be sure to follow their directions step by step. Do not forget to remove all the tape, foam inserts and gantry clips before hooking it to power and USB.

I made sure to have a flat stable table with room for a filament spool stand next to the printer.

Videos I found useful:

Mac vs Windows

I am a Mac person. The current version works but keep in mind the Windows version is ahead of the Mac version in features and firmware. Whenever you start up the M3D software regardless of platform it will check the firmware version. The software and firmware versions are intertwined best I can tell. If I update firmware to print from the Windows version when I go back to the Mac side then the firmware must downgrade before I can use the Mac M3d software. Same going back the other way.


My particular printer does not print center of the bed when on the Mac firmware (2015-10-23-03) despite what the M3D software shows before printing. The print head can go to the center when told. I had even tried the full system recalibrate. The problem goes away when doing the exact same print from Windows with newer firmware (Beta 2016-01-08-12).

You can get scared you bricked your printer if the update gets interrupted. So far I have been able to just go back to the Mac side force a fresh downgrade to recover. There is a tech note on firmware updating in the tech support pages.

Filament and feeding

The biggest problem I had with printing was getting my head around good filament feeding to the printer. Most of the time the internal feed path from under the print bed worked reliably. At times it would still catch. You know when you have filament binding/friction issues because your print will skew as it builds. Drag causes higher layers to be off compared to where they should be.

Remember I said I’m a digital guy? Yeah.. I was dumb and just put the external PLA spool on a hatch box spool holder I got from Amazon. Without what is called a spool bearing. Meaning that it didn’t fit centered and thus did not rotate feeding filament when gently pulled. That gave me most of my skew problems. When Pedro via Twitter pointed out skew means friction I went after fixing that. Did I mention these guys are great about sharing their knowledge? And without making me feel dumb for not seeing the obvious.

I customized and printed two of this bearing from thingiverse. Remember those digital calipers? Came in handy here. But only after I read this Make article on how to use them. When I first took them out, I had an image of Noe and Pedro dressed as wizards waving around digital calipers like a magic wand. I had to measure the hole in the PLA spool and the tube on the holder then customize the bearing print accordingly. They are not a perfect fit because I’m a noob. They probably need to be a hair bigger or have some sort of locking washer to hold them in. Still good enough for me and now I can gently tug with two fingers on the filament and see it turn the spool without catching.

Another thing I learned. Not to be afraid of the emergency stop or abort print buttons. Several times I had not calibrated after changing filament or bumped the print head taking out a print. I could tell I was getting skew or bad layer bonding early. Just be sure if you use emergency stop to use the set bed clear button before you can print again. Calibrating the bed position again isn’t bad idea either. It is better than wasted print time and filament. And this unit is SLOW, but seriously what did you expect for such an easy to use printer for $500?

PLA vs Flex (tough 3D)

So far the best prints I have gotten from the M3D have been with their own PLA 3D ink filament. I have some blue PLA I got from Adafruit and it works, but not as well for when comparing my best prints.

Flexible aka tough filament can be Ninjaflex that I got from Adafruit or the new “tough 3d ink” that I got from Micro3D directly. I haven’t opened my Ninjaflex roll yet. But, I have tried the tough ink. You will get absolutely miserable layer bonding if there is any skew at all due to filament binding. It’s obviously because it ends up a big spaghetti mess instead of the object you expected.

I seem to get way better results printing the tough ink filament from Windows with the updated software and firmware that “knows” the filament cheat codes for the new tough ink. On the Mac version, you have to trick it and setup a custom filament profile. That is another reason I wish they would keep the Mac and Windows in sync.


I have been around IT a long time. The concept of a printer language was not new to me. So the slicing/gcode thing didn’t throw me for a loop.

Slicing is where software takes our 3D object and turns it into printer language (gcode). That gcode are the actual actions the printer takes to put the filament where it needs to go to create our object. The M3D software does a good starting job at this. I did buy a copy of Simplify3D to get more efficient prints with better support structure. The only downside to using Simplify3d is that you cannot just hit print when ready and have the printer start up. The M3D uses some special serial port communications protocol that prevents Simplify3D from talking directly to the printer. So you have to print “tool path to file” then use the “add spool job” in the M3D spooler engine to print that file. Similar to what you see in this gcode to M3D YouTube video. I found that I have better control over support structures and overall printing speed seems better due to Simplify3D being smarter than the M3D software itself. Another great feature of Simplify3D is that it lets you animate a preview of how the object will print. so you can look for problems before spending an hour or more on a print.

I do need to spend more time setting up established print profiles in Simplify3D for quality and filament types I want to use often.


This is not an option in M3D software, but is something you can have enabled in Simplify3D. The benefit to me so far is that it gives the printer a chance to purge out filament as it gets warmed up to print my object. That leaves the excess filament off to the side instead of on my object or throwing the raft out of whack.


Raft? I almost always print one on the M3D. At first filament adhesion to the print bed was not an issue. It did get worse over time with many prints. So the printing of a raft gives the print a more level footing. The downside is that often the raft is harder to break away at higher print resolutions for me. I could probably improve this if I get my head around what all the numbers mean in the Simplify3D settings. That is again something you have little to no control over in the M3D software natively.

The rafts on my first M3D software based prints when the printer was new broke away great. Seems both Simplify3D and M3D generated rafts have fused more with the objects than they did at first. I suspect either operator error or all the knocking around. Or maybe it’s print quality related. The higher the settling the more heat that gets to small area on this printer.


The Adafruit guys love Octoprint. The idea of using an idle Raspberry Pi2 for a print server is certainly attractive. It would save me from leaving my laptop attached to the printer for hours when I’d rather take it with me to Starbucks. You can even turn on mobile interface for your phone or tablet.

I tried using Octoprint from my raspberry pi 2. It was unbearably slow on my B+ so just stuck with my Pi2. I simply could not get it working with my M3D when starting from the Mac firmware. Octoprint wanted it’s own firmware update of the M3D to let it communicate. Even after letting the firmware update my printer, nothing would work. I kept having to downgrade back to the Mac firmware version.

Next, I tried using Octoprint with the beta windows firmware. It let me communicate to the printer and did not prompt to upgrade firmware via Octoprint. I could move the head around. When I tried to print a gcode file that I previously ran with M3D spooler; the print head tried to go up out the top of the printer. So I figured I needed to calibrate from within Octoprint. That bought me a small burn hole into the front left as it moved the printhead outside the bed area. I would NOT be messing with Octoprint and M3D if you are a rookie like me. I am giving up on it until better step by step tutorials are out by experienced folks.

Updated There is a M3D-Octoprint tutorial on Adafruit that mentions leveling each corner manually. It is all on me for not reading that tutorial over again before messing with Octoprint.


I will make one comment about M3D support. When I first started having issues I was worried indicated printer hardware alignment in the first few days I sent in a support ticket. They are either so busy or so understaffed I have only received automated ticket email on it. That is even days later. I have emailed back asking them to close the ticket. If fast technical support on the retail version is a concern, you should take that into consideration before buying.

I love my M3D as someone new to 3D printing. I have learned a lot and made some mistakes. Hopefully, if you are as new as I am to 3D printing you can learn from my experience so far. It will continue to be good for portable printing and small lower detail parts. I expect in a few months I’ll graduate to the Flash Forge Creator Pro unless something better comes out for 2016.


Splunk, Adafruit.io, and MQTT

I have been enjoying the Splunk HTTP Event Collector (HEC) since it’s introduction in Splunk v6.3. You can check out a python class I made for it over on the Splunk Blog. That got me started back on data collection from my Raspberry Pi. I can just send data straight into Spunk using the HEC. But what if I wanted data from a remote Raspberry Pi?


That brought me back to messing around with my Beta Adafruit.io account. This is a data bus service being made by Adafruit perfect for your DIY Internet of Things projects. You can find a lot of their learning tutorials on it in the Adafruit LMS. I did some minor playing over the holiday. Then Lady Ada went and made a tutorial specifically on MQTT.

MQTT and Splunk:

I remember seeing a modular input for MQTT in Splunkbase. Why not try it out with Adafruit.io? Well the answer was… Its java dependent. I love Damien’s work which is awesome as always. But, the Splunk admin hat side of me cannot stand having to install Java to make a feature work. He is trying to convince me to made a Python based version myself. We shall see if I can make the time. Was there an alternative? Why… yes there is. That is how we come back full circle to the the HTTP Event Collector and my python class.

Mixing Chocolate and Peanut Butter:

I took the Adafruit Python class for adafruit.io and it’s example code. Just import in my HEC class and mod the Adafruit code just a little. Now we have a bridge between the Adafruit MQTT client example and sending it into Splunk via the HEC. This let me take the feed value posted to a give MQTT feed on Adafruit.io and send it into Splunk with a single listening Raspberry Pi running a python script local to my Splunk instance.

The code I used was the MQTT Client example. Just add import and creation of an HEC object at the top of the script right before the Adafruit_IO import section

Next we add the following to the bottom of the message method in the Adafruit code.

That is it. Now as long as the script is running it takes the value from a monitored Adafruit.io MQTT feed and kicks it over into Splunk via the HEC. Enjoy!


Splunk TA-Openphish

Perhaps I should have waited till Friday to release something related to Phishing. Yeah bad humor, Phish Fryday…

I want to test things a little more before putting this to apps.splunk.com. However, you can find the TA-Openphish over on my Git Repo. It indexes the feed that Openphish provides you. The readme gives you all the items to consider and setup. I provided a way to filter what gets indexed based on ASN or Brand. You can even combine them for an ADD type filter. However, the Openphish feed is fairly small so I recommend at least starting out to index the whole thing unfiltered.

I also provided Splunk Enterprise App modular inputs for threatlist correlation integration. As I do not have ES here at home I have not recently tested that.  Jack Coates of Splunk did test my initial threat list for the IPs over the weekend and said it worked fine. Big thanks Jack! I appreciate getting a slice of your very busy time.

I also want to look at expanding this to Critical Stack processed feeds. Maybe, I can normalize Phishtank and Openphish feeds together through it for more coverage on brand protection information going into Splunk.

 ** Note March 5, 2014: corrected Critical Stack link from Threat Stack link.**


Fishing for Phishers

Earlier today I saw @averagesecguy tweet a Python script for submitting random credentials to a phishing site. This got my attention as I have manually done this to some of my phishing group “BFF”s before.

It can be entertaining to submit a honey token credential to a phishing campaign against your organization. Follow up with a Splunk alert on the credential to monitor sources, maybe even take an Active Defense approach to them.

It got me thinking. How could I glue this together for a sit back and enjoy experience?

I have been working on a Splunk TA (technology add on) for openphish.com feeds. I’ve done automated response before in Splunk. I bet you see where this is headed.

The Idea:

  1. Take in the Openphish.com feed.
  2. Alert in Splunk on your Brand.
  3. Have the alert submit a random credential leveraging @averagesecguy’s script.
  4. Have the credential add to a Splunk KV store table for used honey credentials.
  5. Setup alerts and active response in Splunk based on any authentication hits on the KV store lookup.
  6. Grab the tartar sauce and enjoy.


  1. Maybe have the random honey credential submissions generate a modest number of submissions per phishing link. Only one and the bad guys might not use it amongst real ones obtained from your organization. Too many and they might notice and filter those out such as from same source IP.
  2. Conform the honey credentials to your organizations naming and password credentials. This will make them appear real compared to genuine credentials they capture for your organization.
  3. Make the submission mechanism use one or more appropriate source IPs for your Org. If its traceable to one single source IP the bad guys could filter on it.
  4. Make sure your pool of random credentials do not contain valid usernames of real users so your alert/automation don’t hit folks you care about.
  5. If you get into automating defensive action be sure to whitelist source IPs appropriately. It would be unpleasant if the bad guys tricked your defenses into shutting down traffic to things you care about.
  6. As we evolve our code maybe take into account the time discovered on phishing pages and don’t submit to all of them or if they are too fresh. This could reduce chances the Phishers are making a new site and seeing if the security team finds and hits it before they’ve had a chance to send it in a real phishing email blast.
  7. Account for source IPs for successful two factor associated logins for your employees. You might use Duo Security with Last Pass Enterprise as an example. That gives you source IPs you have high confidence are indeed your employees. You can tailor response to alerting vs active defense accordingly.

We know phishers use poor grammar to target the users most likely to fall for phishing. We can use this as a similar strategy. Target the less sophisticated phishers with some simple automation and alerting. You could spice it up by adding auto abuse reporting on the hosting of the phishing sites hitting our brand.

I will be trying out some coding on this. If I get it working reasonably well it will go up into my git repo as usual.


Splunk Dry Harder – Splunking the Laundry 2

I was originally going to call this revisit of my old Splunking the Laundry post, Heavy Duty Cycle. My former coworker Sean Maher instead suggested Dry Harder and I could not pass that up as the sequel. So we return to playing with Laundryview data. This is a fun service used in campuses and apartment buildings to let the residents track when there are available machines, check status of their wash and get alerts when done.

The original code was very primitive python scraping a specific laundry view page for my apartment building laundry room. It formatted the data like syslog. From there that went into Splunk.

I decided remaking the code as a modular input was in order if I could make it scrape all shown machines from the page automatically. It works and you can find the TA-laundryview on my Github account. The readme does point out you need to know the laundry room (lr) code found in the URL you normally visit to see a room’s status.

Splunk can pull in any textual information you feed it. Whether that is data generated by small devices like a RaspberryPi 2 or scraping a site like Laundryview and benefitting from the existing machine data. Let’s explore a day’s data from UoA.

So here is the example I have been collecting of the University of Alabama laundry rooms. Note that I have defined an input for each laundry room on campus. The interval I set is every 15 minutes to the index=laundry, sourcetype=laundry, host=laundryview.com. I have found that the 15 minute time frame is enough resolution to be useful without hammering the sites too hard.

UoA Laundry Rooms

UoA Laundry Rooms

A pair of stacked column graphs gives us a fun trend of washers and dryers in use for the entire campus population.

> index=laundry site_name=”UNIVERSITY OF ALABAMA” type=washer | timechart span=15m count by inUse

UoA Washers

> index=laundry site_name=”UNIVERSITY OF ALABAMA” type=dryer | timechart span=15m count by inUse

UoA Dryers

Next we make a Bubble chart panel to bring out the machines in an error status. We define that as Laundryview reporting an offline or out of service status. You will find I defined an eventtype for that.

> index=laundry | stats dc(uniqueMachineID) AS totalMachines by room_name, type | append [search index=laundry eventtype=machine_error | stats dc(uniqueMachineID) AS inError by room_name, type] |  stats sum(totalMachines) AS totalMachines sum(inError) AS inError by room_name, type | eval failure=inError/totalMachines*100

UoA Machine Errors

Here we show it again but this time with a stats table below it helping spot the laundry rooms with the most unavailable machines.

UoA Machine Error Table

You can see we could run all sorts of trends on the data. Want to bet the laundry room usage plummets around the football game schedule? How about throwing machine errors on a map? I actually did make a lookup table of laundry room name to lat/long information. That is when I found out the default map tiles in Splunk do not have enough resolution to get down to a campus level. It gets you down to about the city level. Tuscaloosa in this case. So it was not worth showing.

Other questions you could answer from the data might be:

  1. Do we have any laundry rooms with functioning washers or dryers with none of the other type? Imagine how ticked students would be stuck with a wet bunch of clothes and have to carry it to the next closest laundry room to dry it.

  2. How about alerting when the ratio of machines in an error state hits a certain level compared to the population available in a given laundry room.

  3. Could the data help you pick which housing area you want to live in at school?

  4. How long and how often do machines sit in an Idle status? This maps to a machine that has finished it’s cycle but no one has opened the machine door to handle the finished load. (eventtype=laundry_waiting_pickup)

The possibilities are quite fun to play with. Enjoy!



Splunk and Apple Swift for Alert Scripting

This week, I attended my first Nashville Cocoaheads meeting in a few years. Cocoaheads is basically an Apple programming club. The topic was “Scripting with Swift” It was a great presentation by Blake Merryman. Blake works for Griffin Technology and they were our awesome hosts for the meeting. Griffin is one of my favorite accessory makers. Plus there are right here in Nashville.

Of course I immediately wondered. If I can treat Apple Swift as a shell scripting language… Splunk alerts written in Apple Swift! Oh yeah why not?!?

And… The short answer is that it works for Splunk running on OSX. This means you could make data sent from Splunk via alerts to OSX code that has full native library access to anything on the Mac.

Here is a very simple example thanks again to Blake’s example code.

Alert Script Setup

You must have Xcode installed on the Mac as well as Splunk. Xcode provides the Swift language installation.

You need to make a shell wrapper script in the $SPLUNK_HOME/bin/scripts/ folder. Splunk won’t be able to call a .swift script directly. You will need to add the executable flag to both script files. Then you simply setup the desired Splunk alert to call alertSwift.sh which is just a shell command wrapper script to call the Swift script. The key on the swift script is the hashbang that tells the system how to execute Swift.

Our Swift code example below is a simple call of Apple Script to make the Mac say “Splunk Alert.” This is a very simple example. But it shows that if you use all the normal tricks of pulling in the alert script arguments you could pull in a search results zip file and take actions in Mac OSX based on the data. It could be anything including Notification Center alerts. Enjoy the possibilities.

Alert Script Code

Example alertSwift.sh: (Make sure you do a chmod +x alertSwift.sh)

Example alertSwift.swift: (Make sure you do a chmod +x alertSwift.swift)