Splunk DNS Lookup Performance and Caching with dnsmasq

I use dnsmasq, a light weight DNS caching server, at home on a raspberry pi to log dns traffic when testing things (just uncomment the log-queries option and pull the logs into Splunk). But, what about helping performance of some DNS related activity with Splunk itself?

It is very common to do a LOT of DNS lookups when using Splunk for security purposes. This can create a metric ton of lookup requests to the DNS servers your Splunk server normally points at. That traffic load can cause unforeseen issues at times. I like to setup dnsmasq for local DNS caching on my Splunk search head to help reduce that load when IP lookups are repetitive.

Let’s say your normal network dns server is Rather than have your Splunk server query it or go directly to the root servers we will setup dnsmasq on the server and point it to the normal server you use. This will let Splunk get DNS requests locally to the system if they have been recently cached. We will also lock it down to only work for the local server so other systems on your network do not try and use your Splunk server as a DNS server. Now in my testing it seems dnsmasq seems to re-forward requests in fairly short order despite a large cache value (number of host names). This still should provide some protection if you run a poorly placed dnsLookup command in a Splunk search. Just think of how many times the same IP can come up in searches with large numbers of events. This has to be done for each of the Splunk search heads where the DNS lookups occur.

This is an example of a search that will potentially generate repetitive lookups for the same ip address:

tag=authentication action=failure | lookup dnsLookup ip AS src_ip

This is a better placement of the lookup so you only get one lookup per ip address:

tag=authentication action=failure | stats count by src_ip | lookup dnsLookup AS src_ip

Let’s walk through adding dnsmasq to help reduce the traffic caused by the first search and lookup example.

Continue reading “Splunk DNS Lookup Performance and Caching with dnsmasq”


Windows Domain Login Script

I don’t think I ever posted this before.  If you need a login script to map drives and network printers based on Windows domain group membership for users try the below.  Put it in a vbs file like login.vbs.  Edit “domainname” to be your Windows domain name, and edit the permission group names appropriately.  It also has example of removing existing drive mounts before trying to mount by group.

Continue reading “Windows Domain Login Script”


Setting up SSH Alerts to iPhone

This is sort of a follow up to my SSH screencast series for remote access to your Mac.  Maybe you are paranoid like me and want to know when a connection has been made to your mac, when a wrong user name has been tried or even a failure to login on a good username.  You also want to know this no matter where you are.

I was inspired by the script written by Whitson Gordon, over at Macworld on automating turning off your wireless Airport interface.  Note what I have below has only been tested on my Snow Leopard setup.  I leave it up to you if you are on Leopard or even Tiger.  BTW update your system if you are as far back as Tiger. C’mon join the modern world.

You will have to have Growl installed, also install growlnotify and last you need a Growl to push notification service like Prowl.  Then have the Prowl app on your iPhone or iPad.

Read on for the scripts and how to get it all working.

Continue reading “Setting up SSH Alerts to iPhone”


Scripting Acrobat Reader Updates – nmap and psexec

The latest round of adobe patches are a pain for IT staff to implement.   If you allow automatic updates then many machines updating the full reader installer from Adobe is likely to knock out your wan or Internet links.  Too much traffic.

Manually running around and installing the update is also a pain for IT and consumes a lot of man hours.  So I love to make script packs for them to automate things.

To use these scripts you need to do several prep things.

  1. Download and put nmap binaries for windows in the folder you will run the scripts from.
  2. You will need to install the winpcap driver for the nmap scans to work.
  3. Download psexec from the Microsoft Sysinternals site and put it in the script folder too.
  4. Download the adobe reader installer and put it on a network share.
  5. Create a toss off domain user account that simply can map to the network share of the acrobat. I put it in a subfolder of that share called acro93 for the version I am installing.  Because if you have your domain setup reasonably well you want only authenticated users to connect to shares etc.  You will delete this account once done.

Next come the scripts.  We have the master script we call acrobat.bat.  This script pushes a second bat file into each target host.  You need to put your target hosts into a text file in a format that would be accepted by nmap.  A subnet, indvidiual ips, hostnames your pc can resolve.

Continue reading “Scripting Acrobat Reader Updates – nmap and psexec”


Logging – Collecting Mac logs to your Logging VM

Perhaps you have made yourself a logging vm, or even a logging machine out of an old laptop using my pdf instructions.  At home I actually turned a real old IBM Thinkpad A22m into a unbuntu logging machine.  Just like my directions only no vmware.

I send all my network hardware logs via syslog to the machine.  BUT I also did one simple change to the syslog.conf on every mac in my house.  Now all my mac logs collect into my machine for searching in Splunk.

  1. Just open Terminal on your mac.
  2. sudo vi /etc/syslog.conf
  3. edit the file and add the following line, substituting your own logging machine IP address.
    *.*               @loggingmachineipaddress
  4. Make sure to use an actual ip address in place of loggingmachineipaddress.  I tried using the bonjour or mdns name like logger.local and my macs never consistently sent logs.  So changing to IP address it seemed to work after that.
  5. Next if you are in Leopard you can do the following two terminal commands to restart syslog and pick up the config change.  Otherwise you could also just reboot your mac.
  6. sudo launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist
  7. sudo launchctl load /System/Library/LaunchDaemons/com.apple.syslogd.plist

Building a logging VM – syslog-ng and Splunk

Recently I wanted to build a log collection virtual machine.  I settled on a combination of syslog-ng and splunk.  Syslog-ng lets you do filtering, message rewriting and routing to multiple destination types.  Splunk v4 gives you a nice ability to search the gathered logs.  So you can follow my two documents.  The roll your own covers the building of the vm.  The getting started covers doing the last setup tweak to use and collect certain event types I decided would make a good stating example set.

We use Ubuntu sever 32bit 9.10 with syslog-ng v3 and splunk v4 in this tutorial.  I built mine in vmware fusion on my mac.  But you should be able to adapt to your own box/virtualization of choice.


Logging – syslog-ng rewrite kiwiSyslog forwards

Lately I have been working on making a vmware virtual machine for combining syslog-ng version 3 and splunk.  I wanted to leverage syslog-ng for routing of messages and for rewriting messages from an existing kiwisyslog server.

Let’s say you have all your network gear sending events to an existing kiwisyslog install.  You can add an action to foward the messages and include the original source IP.  The problem is that the original IP becomes part of the message.  When it reaches splunk you would rather it see the messages as having come from the original host so you get the best mapping to host fields in splunk searches.

So we use syslog-ng to receive the forwarded messages then rewrite the message before it is picked up by splunk.  We tell syslog-ng to listen on udp port 3514.  This is the port we tell kiwisyslog to forward events to.  Next we tell syslog-ng to write the events to a fifo linux queue while applying the rewrite.  It is easy from there to tell splunk to pull events from the fifo.

So click more to see the config I used in syslog-ng to make this work.  The solution is a combination of telling syslog-ng to NOT parse the incoming messages then to apply the rewrite rule.  I do plan on writing a pdf guide on building the logging vm from scratch soon.  But for now you can check out the config below.

Continue reading “Logging – syslog-ng rewrite kiwiSyslog forwards”


Wireshark + OSX Leopard

Normally I just run a sudo tcpdump at a command line.  But I wanted to play around in the wireshark gui of the latest build 1.0.8 for OSX Leopard.

So I downloaded the latest DMG for Wireshark 1.0.8 for Intel Leopard.  Dragged the Wireshark app to my Applications folder and ran it.  Wireshark would not see any network interfaces.

What I found is that I need to do the following then wireshark can see the interfaces.  BTW no: sudo open “Applications/Wireshark.app” would not work either.  I suspect because its an x11 app.

sudo -S chown username /dev/bpf*

Note you substitute your short username for the “username” field above.  But who wants to do that every time you reboot?  Even if you script it.  So I of course made an automator.

  1. Drag over the “Ask for Text” object.  Use a prompt like “Enter Password:”
  2. Drag over Run Shell Script.  put in the sudo chown from above.  Also change the pass input to: to stdin
  3. Lastly drag over Launch Application.  Choose the Wireshark.app. 

Save it as an automator application.  Maybe on your desktop.  And now you have a simple double click method to perform the chown of the network interfaces so Wireshark.app can see them.  It will prompt you for your user password (assuming you are an admin user or added your account to sudoers using the visudo command) and pass it to the sudo statement for you then launch wireshark.

One last thing.  Seems Leopard and where wireshark thinks some mibs are disagree.  I found a great blog post by Josh Fuller on fixing it with a couple of symbolic links.  You may have to put sudo in front of his commands. It worked for me.