This post continues [More Splunk: Part 1] Monitor specific processes on remote servers.
So far, I have a simple shell script called
counters.sh that will return two pieces of information I want fed into my Splunk indexer server:
- MySQL CPU usage
- Count of Apache web server processes
It’s going to return a result that looks something like:
2012-11-20 14:34:45-08:00 MySQLCPU=23.2 ApacheCount=1
Install the Forwarder
For each server I need a Splunk agent called a Forwarder installed. The Forwarder’s purpose is to send the data collected on the local server to a remote Splunk server for indexing and reporting. Splunk offers three types of Forwarders but I want the one with the lightest weight and overhead—a Universal Forwarder. For my testing I downloaded the Mac OS X 10.7 installer and installed it onto OS X 10.8 without any noticeable issues.
At this point the Forwarder service hasn’t been started yet. I first want to add my script and a couple of configuration files. The configuration files are necessary because the Universal Forwarder has no web interface to facilitate point and click configuration.
Create and populate the app directory
First, I want to create a folder for my “app”. An app is a directory of scripts and configuration files. By creating my own app directory I can control the behavior of its contents, overriding preset server defaults if I choose.
Inside my app folder I’ll create two more called
bin folder is a Splunk security requirement. Any executable, such as a script, must reside in this folder. This is where I’ll place my
counters.sh script and make it executable using
local folder will contain two plain text configuration (
inputs.conf is the configuration file that controls executing the script and getting its data into the Splunk Forwarder. And
outputs.conf is the configuration file that controls sending the data out to the indexing server or “Splunk Receiver”. These files can be very simple or very complex depending on the needs. I like simple.
Contents of inputs.conf
disabled = false
interval = 60.0
.conf file tells the Splunk Forwarder where to find the script to execute and then executes it every 60 seconds.
Contents of outputs.conf
.conf file tells the Splunk Forwarder to send its collected script data to a specific IP address on port
9997 where the Splunk Receiver is listening.
Configure the Splunk Receiver to listen
All that’s left to do is configure the Splunk Receiver to listen for data coming in from Splunk Forwarders on port
9997 via its web interface and start the Splunk Forwarder’s service via its command line utility.
On the Splunk Receiver server, the server accepting all the data for searching later, click the Manager link in the upper right corner and then click Forwarding and receiving. Click on Configure receiving and then click the New button to create a new listening port. Enter
9997 or another port number not commonly used. Click the Save button.
On each Splunk Forwarder the necessary files are already in place. The only task left is to start the Forwarder’s service.
sudo /Applications/splunkforwarder/bin/splunk start
If this is the first time running the
start command then press the spacebar repeatedly to read the license agreement or press “q” to quit and immediately accept the agreement.
To test that the Forwarder is working run the
sudo /Applications/splunkforwarder/bin/splunk list forward-server
If prompted for credentials use Splunk’s defaults:
Splunk username: admin
It should return something that looks like this:
Configured but inactive forwards:
Searching on the Splunk Receiver should also return results from the Forwarders. Search for
Now that remote server data is flowing into the Splunk indexer machine the last step is to search for it and turn it into meaningful reports.