Splunk plus TOR = Splunkion: forwarding logs over TOR

A fun crazy experiment:

Some weekends I just pick a couple of lego blocks of technology and click them together to see what happens. I was thinking over the concept of TOR hidden services. It turns out you can run a Splunk Universal Forwarder (UF) with an outputs.conf pointing to your indexer while it listens for inputs from other UFs as a TOR hidden service. You can then make a UF running on something like a raspberrypi send it’s logs back over TOR like a dynamic vpn.

Why would you want to? Because it was neat to do. Here is how to repeat the proof of concept.

Splunk forwarding over TOR
Splunk forwarding over TOR


How do we make it work?

The Universal Forwarder TOR to Indexer Relay:

  1. Install the Splunk Universal Forwarder
  2. Install TOR:  sudo apt-get install tor
  3. Setup TOR to listen on 9997 as a hidden service by editing the /var/tor/torrc file
  4. Restart TOR:  sudo service tor restart
  5. Get the server’s .onion address: sudo vi /var/lib/tor/other_hidden_service/hostname
  6. Setup $SPLUNK_HOME/etc/system/local/inputs.conf to listen on 9997
  7. Setup $SPLUNK_HOME/etc/system/local/outputs.conf to send data to your existing Splunk Indexer. The below example is setup for SSL so replace with what yours uses.

The Remote Forwarding Log Source:

  1. Install the Splunk Universal Forwarder
  2. Install TOR:  sudo apt-get install tor
  3. Install socat:  sudo apt-get install socat
  4. Setup $SPLUNK_HOME/etc/system/local/outputs.conf to send logs to localhost:9998
  5. Ensure socat is running to bounce 9998 to 9997. This is how we torrify the Splunk forwarder to Indexer traffic. We need to use it to tunnel the Splunk TCP traffic through TOR. You will want to work up how to make that auto start on reboot and run in background. But here is the command you can run manually to test it. Note in this command you have to know the .onion address of the UF we will use as our TOR to Splunk indexer gateway on the receiving end.
  6. Set Splunk to pickup logs etc via the normal inputs.conf methods.

 Final Comments:

That is it and you have torrified Splunk forwarder to Indexer traffic. It would let you collect data from remote sources without exposing to them the actual destination address of your Indexing system.

Keep in mind that TOR itself encrypts the traffic so you could stick with the unencrypted “9997” outputs.conf style setup. Or you could still go all out and generate a new SSL Certificate Authority with ECC certificates and do all the normal certificate root and name validation that you should when setting up SSL for Splunk. If you want to learn more on how to do that come see a talk I am giving with a friend at Splunk .conf 2014 this year.