Posts Tagged ‘logging’

Manually delete data from Splunk

Thursday, December 27th, 2012

By default Splunk doesn’t delete the logging data that it’s gathered. Without taking some action to remove data it will continue to process all results from Day 1 if doing an All time search. That may be desirable in some cases where preserving and searching the data for historical purposes is necessary, but when using Splunk only as a monitoring tool the older data becomes superfluous after time.

Manually deleting information from Splunk is irreversible and doesn’t necessarily free disk space. Splunk users should only delete when they’ve verified their search results return the information they expect.

Enabling “can_delete”

No user can delete data until he’s been provided the can_delete role. Not even the admin account in the free version of Splunk has this capability enabled. To enable can_delete:

  1. Click the Manager link and then click the Access Controls link in the Users and authentication section.
  2. Click the Users link. If running the free version of Splunk then the admin account is the only account available. Click the admin link. Otherwise, consider creating a special account just for sensitive procedures, such as deleting data, and assigning the can_delete role only to that user.
  3. For the admin account move the can_delete role from the Available roles to the Selected roles section.
    Enable can_delete
  4. Click the Save button to keep the changes.

Finding data

Before deleting data be sure that a search returns the exact data to be deleted. This is as simple as performing a regular search using the Time range drop down menu to the right of the Search field.

The Time range  menu offers numerous choices for limiting results by time including Week to date, Year to date and Yesterday. In this case, let’s search for data from the Previous month:

Search time range

Wait for the search to complete then verify the results returned at the results that need deleting.

Deleting data

Deleting the found data is as simple as performing a search and then piping it into a delete command:

Search and delete

 

This runs the search again, deleting found results on the fly, which is why searching first before deleting is important. Keep in mind that a “delete and search” routine takes as long or longer to run as the initial search and consumes processing power on the Splunk server. If deleting numerous records proceed with one search/delete at a time to avoid overtaxing the server.

Tags: , ,
Posted in General Technology, IT Management | Comments Off

Introducing Splunk: Funny name, serious logging

Thursday, November 15th, 2012

So, my boss says:

“Write an article called ‘Getting Started with Splunk.’”

I reply:

“What, you think I know all this stuff? This really would be a getting started article.”

But here it is and WOW is Splunk cool!

My only experience with Splunk up to a couple days ago was seeing a T-shirt with “Log is my copilot”. I knew it had something to do with gathering log files and making them easier to read and search. In about an hour I had gone to Splunk’s website to research the product, downloaded and installed it, and started viewing logs from my own system. The Splunk folks have made getting their product into their customer’s hands easy and getting started even easier.

What is Splunk?

Simply put, Splunk can gather just about any kind of data that goes into a log (system logs, website metrics, etc.) into one place and make viewing that data easy. It’s accessed via web browser so it’s accessible on any computer or mobile device such as an iPad.

What do I need to run Splunk?

Practically any common operating system today can run Splunk: Mac OS X, Linux, Windows, FreeBSD and more.

How much does Splunk cost?

Don’t worry about that right now. Download and install the free version. It takes minutes to install and is a no-brainer. Let’s get started.

Getting Splunk

IT managers and directors may be interested in watching the introductory and business case videos with the corporate speak (“operational intelligence” anyone?) and company endorsements. Techs will be interested in getting started. Right on their home page is a big green Free Download button. Go there, click it and locate the downloader for your OS of choice. I downloaded the Mac OS X 10.7 installer to test (and installed it on OS X 10.8 without any issues).

Splunk home

This does require a sign-up to create an account. It takes less than a minute to complete. After submitting the information the 100 MB download begins right away.

While waiting for the download…

When the download is on its way the Splunk folks kindly redirect to a page with some short videos to watch while waiting. Watch this first one called Getting data into Splunk. It’s only a few minutes and this is the first thing to do after getting into Splunk.

Installing and starting Splunk

The download arrives as a double-clickable Apple Installer package. Double-click and install it. Toward the end it opens a simple TextEdit window with instructions for how to start, stop and access the newly installed Splunk site.

Install done

Files are installed in /Applications/splunk and resemble a UNIX file system.

Splunk application folder

Open the Terminal application found in /Applications/Utilities and run the command /Applications/splunk/bin/splunk start. If this is the first time running Splunk it prompts to accept its license agreement. Tap the spacebar to scroll through and read the agreement or type “q” to quit and agree to the license.

EULA

Accepting the agreement continues to start Splunk where it displays some brief setup messages.

Starting Splunk

The setup then provides the local HTTP address for the newly installed Splunk site. Open this in a web browser to get to the login screen. The first login requires that the administrator account password be reset.

Splunk login

Following along with the Getting data into Splunk video, Splunk will need some information. Mac OS X stores its own log files. Let’s point to those.

Click the Add Data link to begin.

New Splunk home

Since Mac OS X’s log files are local to the machine, click A file or directory of files.

Add files

Click Next to specify local files.

Add local logs

This opens a window that exposes not only Mac OS X’s visible folders but its invisible folders as well. Browse to /var/log/system.log and click the Select button.

Browse logs folder

For now, opt to skip previewing the log file and click Continue.

Path to system.log

Now, let’s opt to monitor not only the system.log file but the entire /var/log folder containing dozens of other log files as well. Note that Splunk can watch rotated and zipped log files too. Click Save to finish adding logs.

Add /var/log folder

Let’s start searching!

Succes, start searching

The Search window initially displays a list of all logs Splunk is monitoring. To narrow the search change the time filter drop down menu to Last 60 minutes. This will make the results a little easier to see on a system that’s only been running a short while.

Last 24 hours

Now, search for install*. Splunk will only search for the word “install” without providing the asterisk as a wildcard character. Splunk supports not only wildcard searches but booleans, parentheses, quotes, etc. It will return every instance recorded in the logs that matches the search criteria. It also creates an interactive bar chart along the top of the page to indicate the number of occurrences found for the search at particular times.

Search for install

To further refine the search, Option+click most any word in the log entries below and Splunk will automatically add the necessary syntax to remove an item. In this case the install* search returned install, installer and installd. Option+clicking installd changed the search criteria to install* NOT installd.

Modified search

Now what?

Continue exploring the videos to understand Splunk’s possibilities and take advantage of its Splunk Tutorial, which is available online as well as in PDF format for offline viewing. They do a great job leading users through setup and creating reports.

Still asking about price? Good.

The free version remains free but doesn’t include many features that really make it sing such as monitoring and alerts, multiple user accounts and support beyond the Splunk website. Cost depends primarily on the amount of data you want to suck into Splunk and have it watch. It’s not cheap but for an enterprise needing to meet certain service level requirements it beats browsing through multiple servers trying to find the right log with the right information.

FYI, putting together this 1,000-word article probably took me 10 times longer than performing the Splunk install itself and beginning to learn it. It’s really well-done and easy to use. Splunk makes getting started simple.

Tags: , , , , , , ,
Posted in General Technology, IT Management, Kerio, Linux, Mac OS X, Mac OS X Server, Mass Deployments, Network Architecture, Scripts, Web Development, Wireless Networking | Comments Off

Installing a SonicWALL ViewPoint Virtual Machine

Monday, April 2nd, 2012

When installing a Viewpoint VM machine you will need to download three items.

First is the SonicWALL_ViewPoint_Virtual_Appliance_GSG.pdf available from mysonicwall.com
This will be you step by step instruction manual for installing the Viewpoint VM.
Next you will need to identify which version VXI host and then download the same version client as your VXI host.
Lastly you will need log into mysonicwall.com and download the sw_gmsvp_vm_eng_6.0.6022.1243.950GB.ova from mysonicwall.com

When you have all three of these downloaded open the SonicWALL_ViewPoint_Virtual_Appliance_GSG and start going through the step by step instructions.
You will first install the VM client and may run into the first gotcha. Depending on machine setup the .exe may be blocked from running.
The download will look like this: VMware-viclient-all-4.1.0-345043.exe.zip, get properties on this file and unblock if blocked.
After the install of the VM client follow the instructions in the PDF till you get to page 18 step 2.

2. When the console window opens, click inside the window, type snwlcli at the login:
prompt and then press Enter. Your mouse pointer disappears when you click in the
console window. To release it, press Ctrl+Alt

Here is where you will run into the biggest gotcha.

You will be ask to log into with name and password, on first login use name of: snwlcli no password,
Then use the default name and password and continue.

Tags: , , , , , , , , , ,
Posted in IT Management, Network Architecture | Comments Off

Monitoring Xsan with Nagios and SNMP

Monday, December 12th, 2011

Monitoring a system or device using SNMP (a SonicWALL, for instance) is simple enough, provided you have the right MIB. XSNMP is an Open Source project that provides a simple Preference Pane to manage SNMP on OS X, and it also includes an MIB developed by LithiumCorp. This MIB provides OS X’s SNMP agent to gather and categorize information relating specifically to Mac OS X, Mac OS X Server, and Xsan.

XSNMP-MIB can be downloaded from GitHub, or directly from Lithium.

Download the XSNMP-MIB.txt file and put it in /usr/share/snmp/mibs. You can verify that the MIB is loaded by running snmpwalk on the system, specifying the XSNMP Version OID. If snmpwalk returns the version, the MIB is installed correctly. If it returns an error about an “Unknown Object Identifier”, then the MIB isn’t installed in the right spot.

bash$ snmpwalk -c public -v 1 my.server.address XSNMP-MIB::xsnmpVersion
XSNMP-MIB::xsnmpVersion.0 = Gauge32: 1

The fact that the MIB was developed by Lithium doesn’t stop us from using it with Nagios, though. You can define a Nagios service to gather the free space available on your Xsan volume by adding the following to a file called xsan_usage.cfg. Put the file in your Nagios config directory.

define service{
host_name xsan_controller
service_description Xsan Volume Free Space
check_command check_snmp!-C public -o xsanVolumeFreeMBytes.1 -m XSNMP-MIB
}

The host_name should match the Nagios host definition for your Xsan Controller. The service_description can be any arbitrary string that makes sense and describes the service.

The check_command definition is the actual command that’s run. The -C flag defines the SNMP community string, the -m flag defines which MIB should be loaded (you can use “-m all” to just load them all), and the -o flag defines which OID we should return. “xsanVolumeFreeMBytes.1″ should return the free space, in MB, of the first Xsan volume.

Tags: , , , , , , , , , ,
Posted in IT Management, Mac OS X, Mac OS X Server, Scripts, Xsan | Comments Off

Provisioning TelePacific iNOC On A SonicWALL

Friday, January 7th, 2011

1. Login to SonicWALL

2. Check to see if SNMP is already in use on WAN IPs by checking under Network > Firewall.

ALERT: Enabling SNMP Management on the SonicWALL will cause issues with the SNMP firewall rules. You can ONLY have SNMP SonicWALL Management OR SNMP firewall port forwarding. Not both. This was confirmed with SonicWALL Tech Support.

3. Go to System > Administration

4. Scroll down and put a check mark for “Enable SNMP”

5. Click on Configure

6. Put in whatever you want for System Name, System Contact, System Location. You can leave Asset Number blank. Ask TPAC for their monitoring WAN IP and put that in the “Host 1″ field.

7. Go to Network > Interfaces

8. Click on the Configure icon for the Interface that you want monitored.

9. Put a check mark next to SNMP

10. Click OK

11. You can confirm SNMP is listening by using snmpwalk. On a Mac, the command can be:

snmpwalk -c private -v 2c “wanipaddress of SonicWALL”

or

snmpwalk -c private -v 1 “wanipaddress of SonicWALL”

The SonicWALL utilizes version 1 and 2c for SNMP.

Tags: , , , , , , , , , , , ,
Posted in IT Management, Security | Comments Off

Adding Windows Services Monitoring in Zenoss

Thursday, July 22nd, 2010

1. Under devices find the server
2. Go to Configuration Properties
3. Scroll down until you find zWinUser and zWinPassword, and enter in admin username and password.
4. Click on the first item under Components on the left hand side
5. Click on the “+” Sign
6. Click Add Win Service
7. Choose the service from the drop down menu.
8. Click on Service if status says “Unknown”
9. Find server under Display
10. Change Set Local value to Yes
11. Click SAVE (from light testing, this seems to only have to be done once per service).

Tags: , , , , , , , , , ,
Posted in General Technology, IT Management, Network Architecture | Comments Off

Installing Zenoss

Wednesday, December 30th, 2009

To monitor a device over the WAN, there needs to be a 1 to 1 Firewall Rule. There needs to be a firewall rule, allowing SNMP traffic from the WAN to a device on the lan. For multiple devices, then each device will need a dedicated WAN IP with the firewall rule. SNMP runs on UDP on port 161

Install SNMP service in components will require I386 for .dll
Download and install additional SNMP dll files provided by SNMP Informant, http://www.snmp-informant.com
Once installed right click on SNMP click properties and go to the Agents tab:
Contact: (e.g. support@318.com)
Location: (e.g. 830 Colorado Ave. Santa Monica, CA)
Check all services below that
Move to next tab traps:
Community Name:(e.g. 318zenoss)
Click Add to list
Then click add and enter the Zenoss server address
Move to next tab Security:
Make sure send authentication trap is checked
Add community name 318zenoss read only
And check SNMP packets from any host
Click Apply and Ok

Restart the Service.

Add two firewall rules allow traffic from the Device (LAN) to the WAN zenoss address of the Zenoss Server

Next Add device in zenoss:
Log in as user
Click Add Device
Enter Device IP WAN IP Address for Device Name
SNMP Community: 318zenoss
Select the Server Class:
/Servers/Windows – Windows Server
/Servers/Darwin – Mac Server
/Servers/Unix – Linux/Unix Server

Add or select Location Path

Add Or Select Client Name as Location

Select Your Team As Group

Tags: , , , , , , , ,
Posted in IT Management, Network Architecture | Comments Off

Configuring IPS to Deny P2P Traffic On a SonicWALL

Thursday, May 28th, 2009

1. Login to SonicWALL
2. Go to Application Firewall
3. Go to Application Objects
4. “Add New Object”
5. In the next window, name the object
a. Under “Application Object Type” select “Signature List”
b. Under “IDP Category” select P2P
c. Under “IDP Signature” select each one, and add it to the list
NOTE: I tried using Signature Category List, assuming that this would be the same thing as choosing Signature List, and then Selects all of the IDP Signatures. I did not get good results, YMMV.
d. Click OK
6. Go to Policies
a. “Add New Policy”
b. Name the Policy
c. For “Policy Type”, choose “Dynamic Content”
d. For “Application Object” choose the name of the Application Object that you created initially.
e. For Action, choose “Reset/Drop”
f. Select “Enable Logging”
g. Ensure “Log Redundancy Filter” is selected.
h. Click OK
7. Ensure that the Policy is enabled.
8. Check the little bar graph next to the policy, called the Policy Statistics. This will tell you how many times it was used to block traffic.
9. Check the logs to see the blocking in effect, it will most likely be highlighted in yellow.

Tags: , , , , , , , , , , ,
Posted in General Technology, IT Management, Network Architecture, Security | Comments Off

Using LCR for Exchange 2007 Disaster Recovery

Thursday, April 16th, 2009

Local Continuous Replication (LCR) is a high availability feature built into Exchange Server 2007.  LCR allows admins to create and maintain a replica of a storage group to a SAN or DAS volume.  This can be anything from a NetApp to an inexpensive jump drive or even a removable sled. In Exchange 2007, log file sizes have been increased, and those logs are copied to the LCR location (known as log shipping) and then used to “replay” data into the replica database (aka change propagation).

LCR can be used to reduce the recovery time in disaster recovery scenarios for the whole database, instead of restoring a database you can simply mount the replica.  However, this is not to be used for day-to-day mailbox recovery, message restores, etc.  It’s there to end those horrific eseutil /rebuild and eseutil /defrag scenarios.  Given the sizes that Exchange environments are able to get in Exchange 2003 R2 and Exchange 2007, this alone is worth the drive space used.

Like with many other things in Windows, LCR can be configured using a wizard.  The Local Continuous Backup wizard (I know, it should be the LCR wizard) can be accessed using the Exchange Management Console.  From here, browse to the storage group you would like to replicate and then click on the Enable Local Continuous Backup button.  The wizard will then ask you for the path to back up to and allow you to set a schedule.  Once done, the changes will replicate, but the initial copy will not.  This is known as seeding and will require a little PowerShell to get going.  Using the name of the Storage Group (in this example “First Storage Group”) you will stop LCR, manually update the seed, then start it again, commands respectively being:

Suspend-StorageGroupCopy –identity “First Storage Group”

Update-StorageGroupCopy –identity “First StorageGroup”

Resume-StorageGroupCopy –identity “First StorageGroup”

Now that your database is seeded, click on the Storage Group in the Exchange Management Console and you should see Healthy listed in the Copy Status column for the database you’re using LCR with.  Loop through this process with all of your databases and you’ll have a nice disaster recovery option to use next time you would have instead done a time consuming defrag of the database.

Tags: , , , , , , , , , , , , , , , , , ,
Posted in Kerio, Microsoft Exchange, Network Architecture, Windows | Comments Off

New article on Xsan Scripting by 318

Saturday, April 11th, 2009

318 has published another article on Xsanity, for scripting various notifications and monitors for Xsan and packaged up into a nice package installer. You can find it here
http://www.xsanity.com/article.php/20090407150134377.

Tags: , , , , , , , , , , , , , , , ,
Posted in General Technology, IT Management, Scripts, Xsan | Comments Off

Mac OS X: Using tail

Saturday, August 16th, 2008

You can dynamically watch new lines come into log files in Mac OS X. In order to do this you can use the tail command with the -f switch. So if you want to watch your system.log file and run some processes you think will cause errors you can use the following command:

tail -f system.log

Tags: , , , , , , , , ,
Posted in General Technology, IT Management, Mac OS X, Mac OS X Server | Comments Off

Leopard Server: Advanced Setup with Server Admin

Friday, October 26th, 2007

So you selected Advanced Setup during the wizard while you were installing Mac OS X Server and now you’re looking at this new Server Admin screen that you’ve never seen before. You see the server name but there are no services in the list. This is because Apple has gone the extra step to make Server Admin less confusing and more user friendly than ever before. When you click on the Settings icon at the top of the Server Admin screen you will see the tab for Services. Here, you can enable or disable any service by checking its box and clicking on the Save button.

Once a service has been enabled then it will appear under the server in the Servers list (notice it no longer says Sites and Services). From here, you’ll notice that the old chicklets from the bottom screen are gone. Now they have been replaced with an icon set in the toolbar that changes as you click between the services. For example, the AFP Service shows Overview, Logs, Graphs, Connections and Settings. Clicking through these icons, you’ll notice that they provide the same experience that the chicklets at the bottom of the screen provided. However, by placing them at the top the user interface makes more sense. One thing that is a bit strange is the decision to move the Start and Stop buttons to the bottom of the screen. When you enable a service it will not start by default so if you want to begin using it look to the bottom of the list and click on the Start button for the service.

When you enable and then click on each service you will notice that many have the same options that they’ve had in the past. There are exceptions (like a more granular logging tab for the FTP service), as there are with every version. But for the most part many of the settings have stayed the same through a few versions of the OS because they just make sense in how they are laid out.

New Services added are Radius, Podcast Producer, MySQL (which actually existed in its own stand-alone application before) and iCal. Each of these has a great purpose and will hopefully be explored in detail as time goes on. You might notice that one service, Applications, is gone from the list. Tomcat has now been moved into the Web Service as a checkbox (Enable Tomcat).

So that’s the quick and dirty tour of the new Server Admin application. It’s sleeker and has a (in our opinion) much improved interface over the old Server Admin.

Tags: , , , , , , , , , , , , , , ,
Posted in General Technology, Mac OS X Server | Comments Off

Installing SonicWALL ViewPoint

Wednesday, May 23rd, 2007

Here are the steps to follow for installing Sonic ViewPoint. Note that a Windows system running 2000, XP, or 2003 is required.

1. Go to www.mysonicwall.com and add the ViewPoint license key to the registered appliance.

2. Download the ViewPoint installation software. It is a free download from www.mysonicwall.com (the client should have a login/password from when the SonicWALL was installed)

3. Extract and run the installer. Follow the prompts to add an SMTP server and admin accounts. Make sure that the Windows firewall is off or has an exception for ports 80, 443, and 514. Reboot the system.

4. Log into the SonicWALL appliance and enter the upgrade key from the mysonicwall.com site into the System > Licenses section on the navbar.

5. In the Logs > ViewPoint section on the navbar, Add the IP address of the computer running the ViewPoint software.

6. open a web browser to the IP address of the ViewPoint system. In the far left pane, right-click on MyReportsView and select Add Unit. Enter the information for the SonicWALL appliance in the window that appears and click OK.

Tags: , , , , , , , , , , , , , , , , , ,
Posted in Network Architecture, Security | Comments Off

Xsan: Sometimes You’re Going to Loose a Drive

Wednesday, April 4th, 2007

Sometimes a drive fails, or a RAID controller goes down on an array with a redundant drive and the parity on a RAID must be rebuilt. In other words, if you loose a drive in a RAID 5, RAID 1, RAID 0+1 or RAID 3 array you will be left with a degraded RAID (also referred to as a critical RAID) unless you have configured your Xserve RAID to use a hot spare. If you are using a hot spare on the channel of the failed drive the RAID will begin to rebuild itself automatically. If you are not using a hot spare, upgrading your degraded RAID back to a healthy state should happen as quickly as possible to avoid data loss. In the event of a second drive failure on the array most of the data could be lost – and Murphy’s Law is evil when it comes to RAIDs. The data should be backed up as quickly as possible if it has not already been backed up.

Once the data is backed up, you should perform a rebuild of the parity for the array. The partiy is rebuilt based on the data that is on the array. This does not fix any issues that may be present with actual data. In other words, if you were using the Xserve RAID as a local volume it would only repair issues with the array and not also perform a repair disk on the drives. In an Xsan any data corruption could force you to rebuild you volume from the LUNs. You would not need to relabel the LUNs, but you may have to rebuild your volume

In many situations you will be able to simply swap the bad drive out with an identical good drive and configure it as a hot spare. Then the Xserve RAID will automatically begin rebuilding the array, moving it from a degraded state into a healthy state.

However, there are often logical issues with drives and arrays. Also, hot spares do not always join the degraded array. In these situations you may need to manually rebuild an array. To do this:
Silence the alarm on the Xserve RAID.
Verify that you have a clean backup of your data.
Verify that you have a clean backup of your data again or better, have someone else check as well.
Open up your trusty Xserve RAID Spare Parts Kit and grab the spare drive module.
Remove the drive module that has gone down (typically the one with the amber light).
Install the new drive in your now empty slot.
Open RAID Admin from the /Applications/Server directory.
Click on the RAID containing the damagemed array.
Click on the Advanced button in the toolbar.
Enter the management password for the Xserve RAID you are rebuilding the parity for.
Click on the button for Verify or Rebuild Parity and click on Continue.
Select the array needing to be rebuilt.
Click Rebuild Array and be prepared to wait for hours during the rebuild process. It is possible to use the array during the rebuild process – although if you don’t have to use the array it is probably best not to as you will see a performance loss. During the rebuild the lights on the drive will flash between an amber and a green state.
Once the rebuild is complete, perform a Verify Array on the RAID.
Verify the data on the volumes using the array.
Order a new drive to replace the broken drive in your Xserve RAID Spare Parts Kit.

If the rebuild of the data does not go well and the array is lost then you will likely need to delete the array and readd it. This will cause you to loose the data that was stored on that array and possibly on the volume, so it can never hurt to call Apple first and see if they have any more steps you can attempt. This is one of the many good reasons for backing data up. Just because you are using a RAID does not mean you should not back your data up.

The Verify Array can also be used to help troubleshoot issues with corrupted arrays.

This process has been tested using firmware 1.5 and below for Xserve RAIDs.

Tags: , , , , , , , , , , , , , , , , , , , , , ,
Posted in IT Management, Mac OS X Server, Mass Deployments, Xsan | Comments Off

serveradmin in OS X

Monday, November 20th, 2006

Mac OS X Server is a strange beast. It has the ability to cause you to
think it’s the greatest thing in the world in that you can do all kinds of
complicated stuff quickly through a nice GUI. It can also dismay many of us
who know where Unix-specifics live in the OS and would prefer to configure
things there. So, where are all those settings that override so many of
the default Unix configuration files? Serveradmin is a command that gives
access to much of what you see in Server Admin and much of what you don’t.

Serveradmin use starts out with viewing data on a specific service. For
example, type sudo serveradmin fullstatus vpn and see a full status on the
settings and use of the vpn service. Or issue an sudo serveradmin settings
ipfilter command and see the settings applied to the firewall service. To
see all of the services you can configure and view type sudo serveradmin
list. Then look at doing a serveradmin start afp followed by a serveradmin
stop afp. Suddenly you are stopping and starting services on a server using
the command line, meaning you can actually issue these over an SSH session
rather than having to use ARD to connect. This can become invaluable when a
bad firewall rule locks you out of the Server Admin tool. Just issue a
serveradmin stop ipfilter and you’re right back in!

You can also set settings that aren’t available in the GUI. For example,
look at VPN. Let’s customize where we put our logs. First, type in sudo
serveradmin settings vpn. Now, look for the following entry:
vpn:Servers:com.apple.ppp.pptp:PPP:Logfile = “/var/log/ppp/vpnd.log”

To change this setting, let’s type in:
Serveradmin settings vpn:Servers:com.apple.ppp.pptp:PPP:Logfile =
“/var/log/ppp/pptpvpnd.log”

Now the PPTP logs will be stored in a separate location than the logs for
the rest of the VPN service. This couldn’t have been done using a
configuration file, but only using the serveradmin command. Nifty!

Now let’s look at NAT. NAT is cool, but there’s just two buttons: Start and
Stop. So how would we require a proxy for Internet traffic? How about
this:
Serveradmin settings nat:proxy_only = yes

Or we could log denied access attempts using:
nat:log_denied = no

These options aren’t available from the GUI at all. But what really happens
when we’re using these commands? Well, typically a plist file is being
updated. Any time you see a yes or no value then you are looking at a
boolean variable in a plist file. That log_denied variable is also stored
in /private/etc/nat/natd.plist in the lines:
log_denied

Fun stuff! In my book I actually go into a little more detail about
forwarding specific ports to other IP addresses using the NAT service as
well. That too happens in a plist.

Tags: , , , , , , , , , , , , , , , , , , , , ,
Posted in Editorial, General Technology, IT Management, Mac OS X, Mac OS X Server, Network Architecture, Scripts, Security | Comments Off

Using NPRE with Nagios

Wednesday, July 5th, 2006

Nagios is a computer Monitoring Software. Nagios runs on one central server, and has the ability to check resources on remote computers. In order to allow these remote computers to be monitored the Nagios Software team has NRPE. NRPE functions as a Daemon and plugin for executing plugins on remote hosts. When installed on the remote server it creates a Medium for the Nagios server to be able to execute commands to the remote agent. You have the ability to check a wide range of resources, CPU, Hard Drive Space, Computer Load, Server Services, DNS checks etc.

To Download NRPE and NRPE Plugins

http://www.nagios.org/download/

In this example we will be installing NRPE to a remote Linux Server. So to send the downloaded file to the remote server
scp Documents/NAGIOS/nagios-plugins-1.4.9.tar.gz USER@HOSTNAME:
scp Documents/NAGIOS/nrpe-2.8.1.tar.gz USER@HOSTNAME:
This command will send the downloaded file to the 318admin accounts home folder on the remote linux server.

NRPE Plugins
Once the file has been copied to the server and placed in an appropriate location, for example /usr/local/src
First we will uncompress the package
[root@HOSTNAME src]# tar -xzvf nagios-plugins-1.4.9.tar.gz
Then cd into nagios-plugins-1.4.9
Perform the following to compile and install the plugins

[root@HOSTNAME nagios-plugins-1.4.9]# ./configure
With this executed sucsesfuly you should expect an output similar to this:
config.status: creating po/Makefile
–with-apt-get-command:
–with-ping6-command:
–with-ping-command: /bin/ping -n -U -w %d -c %d %s
–with-ipv6: yes
–with-mysql: /usr/bin/mysql_config
–with-openssl: yes
–with-gnutls: no
–with-perl: /usr/bin/perl
–with-cgiurl: /nagios/cgi-bin
–with-trusted-path: /bin:/sbin:/usr/bin:/usr/sbin

Now we will make the package
[root@HOSTNAME nagios-plugins-1.4.9]# make
With this executed successfully the following
Making all in po

Now we will perform the make install command to install the Nagios Plugins on the remote server. Run this command as root
[root@HOSTNAME nagios-plugins-1.4.9]# make install

Nagios Documentation recommends changing the permissions on the files after the install is performed. The commands to perform this are as follows:
chown nagios:nagios /usr/local/nagios
chown -R nagios:nagios /usr/local/nagios/libexec

Now we will need to Install the NRPE Daemon to execute these plugins

NRPE Daemon
Next we will uncompress the package on the remote server. This document assumes that you have some type of root access to the server.
First we will uncompress the file, I generally move my source files to /usr/local/src
To complete these tasks first SSH into the remote server
Sudo mv nrpe-2.8.1.tar.gz /usr/local/src
Cd /usr/local/src
sudo tar -xzvf nrpe-2.8.1.tar.gz

We have now moved the package file to /usr/local/src and uncompressed the package to /usr/local/src/nrpe-2.8.1

Next a nagios user account needs to be added on the remote system. Execute this command as sudo or root
Useradd nagios
Passwd nagios

Next we must configure and compile the package before we can install it. I have found that compiling NRPE to work best when executed as root, perhaps this will be resolved in later versions. For the purspose of this documentation, it is assumed that the following commands are executed as root

Cd /usr/local/src/nrpe-2.8.1
[root@HOSTNAME nrpe-2.8.1]# ./configure

When ./configure executes properly you should see an output similar to this
*** Configuration summary for nrpe 2.8.1 05-10-2007 ***:

General Options:
————————-
NRPE port: 5666
NRPE user: nagios
NRPE group: nagios
Nagios user: nagios
Nagios group: nagios

Review the options above for accuracy. If they look okay,
type ‘make all’ to compile the NRPE daemon and client.

If you receive errors do not proceed, you must resolve what ever dependency errors you are receiving when attempting to compile.

Next we will perform the make command to ‘make’ the installation. This is done by the following command
[root@HOSTNAME nrpe-2.8.1]# make all

With a succesfull make an output similar to this should be seen
*** Compile finished ***

If the NRPE daemon and client compiled without any errors, you
can continue with the installation or upgrade process.

We are now ready to install NRPE on the remote system. To perform the install, execute the following commands as root:
[root@phillip2 nrpe-2.8.1]# make install-plugin
[root@phillip2 nrpe-2.8.1]# make install-daemon
[root@phillip2 nrpe-2.8.1]# make install-daemon-config
[root@phillip2 nrpe-2.8.1]# make install-xinetd

Now since we installed NRPE with Xinetd, we have to edit the following nrpe file to allow outside connections to NRPE from the Nagios Server
Perform this as root
[root@HOSTNAME nrpe-2.8.1]# nano /etc/xinetd.d/nrpe
Now under ‘only_from’ add the local IP address of the Nagios Server, for example you can have the following:
only_from = 127.0.0.1 192.168.1.23
Save the changes and exit the document edit program

Now we have to Edit the Services file to add the port for NRPE to run. Perform the Following as root:
[root@HOSTNAME nrpe-2.8.1]# nano /etc/services
At the bottom of the File I would just add the following Text
nrpe 5666/tcp #NRPE

Once we have edited the Xinetd File and Services file, perform a restart of Xinetd to apply the changes
This should look something like this if done correctly:
[root@HOSTNAME nrpe-2.8.1]# service xinetd restart
Stopping xinetd: [ OK ]
Starting xinetd: [ OK ]
To test that the NRPE Daemon is running you can perform this command
netstat -at|grep nrpe
tcp 0 0 *:nrpe *:* LISTEN

If you do not see the same output Nagios Documentation recommends to check the following:
– You added the nrpe entry to your /etc/services file
– The only_from directive in the /etc/xinetd.d/nrpe file contains an entry for “127.0.0.1″
– xinetd is installed and started
– Check the system log files for references about xinetd or nrpe and fix any problems that are reported

Also you can use the check_nrpe plugin to check your installation of NRPE by performing the following command:
[root@HOSTNAME nrpe-2.8.1]# /usr/local/nagios/libexec/check_nrpe -H localhost
NRPE v2.8.1

The service and Version number should be the expected output of this command.

Before we go to the Nagios server to execute the plugins, in most cases we just have to confirm that the correct devices are being called by the plugins in the config file. Most common is the incorrect hard drive will be checked, which makes the tool not as useful.

So for this server when I run the df command I would want Nagios to examine the partitions of / and /home

For this server I want NRPE to monior /dev/sda1 and /dev/sdb1 so to make this happen we will edit the nrpe.cfg file, /usr/local/nagios/etc/nrpe.cfg down at the bottom are the check commands connected to devices. I have added these two lines, and removed any other hard drive devices
command[check_sda2]=/usr/local/nagios/libexec/check_disk -w 20 -c 10 -p /dev/sda2
command[check_sdb1]=/usr/local/nagios/libexec/check_disk -w 20 -c 10 -p /dev/sdb1

-w 20 means to warn when the drive is 20% free space available
-c 10 mean to have a critical warning when 10% free space available

Configure Nagios Server to Monitor Remote Host

The Remote server must already be running Nagios, this documentation assumes this has already been done. Also if you will need to install the chek_nrpe plugin on the Nagios server. The is done in the same way that a remote host would have NRPE installed, which has already been covered.

After all the requirements have been met, its now time to define server commands for the new host.
First you have to define a services file. This is done on the Nagios Server nagios.cfg. Edit the file located in /usr/local/nagios/etc/nagios.cfg and include for example
cfg_file=/usr/local/nagios/etc/hosts.cfg
cfg_file=/usr/local/nagios/etc/services.cfg
cfg_file=/usr/local/nagios/etc/timeperiods.cfg

Then we will edit the Host.cfg to include the new remote host. Using a text editor add the following, this will add a Host called HOSTNAME with an a local ip of 192.168.1.226, also it will be given the linux-server template. Templates are configured in the nagios.cfg file

define host{
use linux-server
host_name HOSTNAME
alias HOSTNAME
address 192.168.1.226
}

Now we can edit the services file to tell Nagios what NRPE services to check on the new Remote Host. I generally use this template to check various default services

#HOSTNAME
define service{
use generic-service
host_name HOSTNAME
service_description / Free Space
is_volatile 0
check_period 24×7
max_check_attempts 3
normal_check_interval 3
retry_check_interval 1
contact_groups admins
notification_interval 120
notification_period 24×7
notification_options w,u,c,r
check_command check_nrpe!check_sda2
}

define service{
use generic-service
host_name HOSTNAME
service_description /home Free Space
is_volatile 0
check_period 24×7
max_check_attempts 3
normal_check_interval 3
retry_check_interval 1
contact_groups admins
notification_interval 120
notification_period 24×7
notification_options w,u,c,r
check_command check_nrpe!check_sdb1
}

define service{
use generic-service ; Name of service template to use
host_name HOSTNAME
service_description SMTP
is_volatile 0
check_period 24×7
max_check_attempts 3
normal_check_interval 3
retry_check_interval 1
contact_groups admins
notification_interval 120
notification_period 24×7
notification_options w,u,c,r
check_command check_smtp
}
define service{
use generic-service ; Name of service template to use
host_name HOSTNAME
service_description FTP
is_volatile 0
check_period 24×7
max_check_attempts 3
normal_check_interval 3
retry_check_interval 1
contact_groups admins
notification_interval 120
notification_period 24×7
notification_options w,u,c,r
check_command check_ftp
}
define service{
use generic-service ; Name of service template to use
host_name HOSTNAME
service_description HTTP
is_volatile 0
check_period 24×7
max_check_attempts 3
normal_check_interval 3
retry_check_interval 1
contact_groups admins
notification_interval 120
notification_period 24×7
notification_options w,u,c,r
check_command check_nrpe!check_http
}
define service {
use generic-service
host_name HOSTNAME
service_description CPU Load
is_volatile 0
check_period 24×7
max_check_attempts 3
normal_check_interval 3
retry_check_interval 1
contact_groups admins
notification_interval 120
notification_period 24×7
notification_options w,u,c,r
check_command check_nrpe!check_load
}
define service{
use generic-service
host_name HOSTNAME
service_description Current Users
is_volatile 0
check_period 24×7
max_check_attempts 3
normal_check_interval 3
retry_check_interval 1
contact_groups admins
notification_interval 120
notification_period 24×7
notification_options w,u,c,r
check_command check_nrpe!check_users
}
define service{
use generic-service
host_name HOSTNAME
service_description Total Processes
is_volatile 0
check_period 24×7
max_check_attempts 3
normal_check_interval 3
retry_check_interval 1
contact_groups admins
notification_interval 120
notification_period 24×7
notification_options w,u,c,r
check_command check_nrpe!check_total_procs
}
define service{
use generic-service
host_name HOSTNAME
service_description Zombie Processes
is_volatile 0
check_period 24×7
max_check_attempts 3
normal_check_interval 3
retry_check_interval 1
contact_groups admins
notification_interval 120
notification_period 24×7
notification_options w,u,c,r
check_command check_nrpe!check_zombie_procs
}

After you have edited the host and service files, you may want to run a check to see what if anything is wrong with your config. Run this file, I find it helpful to make this just a script that you can execute once your done. The command is
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

If everything runs without error, its safe to restart the nagio service and check the website for the new host.

How to Monitor Remote services with Nagios and NRPE

As long as both the Nagios server and Remote Host running NRPE have the nagios plugins installed this is very straight forward.

In this example we will add monitoring of HTTP and MySQL to a remote server named terrence2.

First on the Remote Host with NRPE, edit the nrpe.cfg file to include:
command[check_mysql]=/usr/local/nagios/libexec/check_mysql -h localhost
command[check_http]=/usr/local/nagios/libexec/check_http -h localhost

Then on the Nagios server simply edit the services.cfg file to add the following check command:
define service{
use generic-service ; Name of service template to use
host_name HOSTNAME
service_description HTTP
is_volatile 0
check_period 24×7
max_check_attempts 3
normal_check_interval 3
retry_check_interval 1
contact_groups admins
notification_interval 120
notification_period 24×7
notification_options w,u,c,r
check_command check_nrpe!check_http

and for MySQL

define service{
use generic-service ; Name of service template to use
host_name HOSTNAME
service_description MySQL
is_volatile 0
check_period 24×7
max_check_attempts 3
normal_check_interval 3
retry_check_interval 1
contact_groups admins
notification_interval 120
notification_period 24×7
notification_options w,u,c,r
check_command check_nrpe!check_mysql

Restart xinetd on the remote host and restart nagios on the server and your up and monitoring.

Tags: , , , , , , , , , , , , , , , , ,
Posted in IT Management, Linux | Comments Off

Mac Forensics in Los Angeles

Sunday, August 28th, 2005

Ever been hacked? Had information stolen? Who do you turn to? What do you do? No matter what the level, a security breech has occurred and action must be taken to ensure a repeat offense doesn’t happen. The first reaction to a security breech is to isolate it and fix it as soon as possible. However, writing to the systems in any way can cause clues to be overwritten. Therefore it is important to discover the identity of the attacker.

The more quickly that forensic analysis is performed the more likely that the attacker, vandal or thief will be apprehended. One of the best places to start in analysis is making a copy of the system that hasn’t been written to. For Windows this is done using a program like Ghost. On the Mac platform using Carbon Copy Cloner or the Disk Utility to create an image is a good move. It is best to get a copy of your system as soon after a security incident as possible.

On local systems, there are some valuable pieces of information that can be obtained about the identity of the person stealing data. This can be anything from the IP address of the attacker to the name of the drive they’re transferring data to. On many Operating Systems valuable logs or cached files are overwritten on a routine basis. If a clone is made, it is often best to create a clone, or a replica of the system in its current state, as soon as possible.

If it’s a server, then the logs of the server provide good clues as to where to look for the perpetrator. Once again it is helpful to create a clone of the system. However, this is not always possible on production servers. Copying the log files is the next best thing.

Firewalls can provide good clues as well. The logging cycles on firewalls typically store data for a shorter period of time than on workstations or servers. Creating a screen shot in PDF format of the firewalls logs or exporting the logs into a text file is a good starting point. Firewalls typically provide good information on what addresses are communicating with a network. This makes them good at specifically determining the identity of the attacker and according to logging levels, the attacks used.

No matter what the issue, time is of the essence. Contacting a professional to help is a good idea. Getting the FBI or the LA County District Attorneys office involved can take time and this can cause clues to be damaged, lost or destroyed. IT professionals can also assist in creating a chain of custody on the equipment that can later be used in court when and if the person who’s invaded your privacy is apprehended and put to trial.

Tags: , , , , ,
Posted in General Technology, Security | Comments Off