Archive for the ‘Mac OS X Server’ Category

Checking ClamAV Logs

Thursday, February 20th, 2014

As with all Antivirus packages, we should be checking logs routinely. The following steps basically lay out what to do:

1. Log into servers

2. There are 3 log files we are concerned with:

  • /var/log/clamd.log – this is the sytem log for CLAM-AV
  • /var/log/freshclam.log – this is the log for update definitions for CLAM-AV
  • /var/log/clamscan.log – this is the weekly output of the Clam AV scheduled scan log file

3. To read the log files perform the following commands with elevated privileges:

  • a. Cat /var/log/clamscan.log – each weekly scan is separated by a complete line of ‘—-‘
  • b. Cat /var/log/freshclam.log – make sure that Clam-AV is using a current virus definition database, and no errors are occurring
  • c. Cat /var/log/clamd.log – confirm there are no errors causing the service to crash

4. The Virus Scans are scheduled to run every Saturday at 3am every week.

5. ClamAV only supports virus definition updates to be installed up to 3 previous versions of ClamAV. Freshclam will show if the current ClamAV is out of date. To update ClamAV follow the instructions here:

http://wiki.clamav.net/Main/UpgradeInstructions

NTP and Mavericks

Tuesday, October 22nd, 2013

Configuring NTP on the latest version of OSX is, in typical Apple fashion, almost deceptively easy.  As in previous versions of OSX any Mac can serve as an Network Time Protocol server with the check of one button.  Once this is done you’ll be able to act as a network clock for any computer on your local area network.

 

To get ntpd going, simply do the following.

 

First open your system preferences and head over to date & time

Screen Shot 2013-10-22 at 4.54.17 PMYou’ll notice that the box to update from a network time server is unhecked.  And if you look at the following port scan you’ll see that the NTP port, port 123, is closed.

Picture 1

Now check the box

Screen Shot 2013-10-22 at 4.54.21 PM

And notice that the NTP daemon starts up and listens on the correct port.

Picture 2

What could be easier?  Now get out there and synchronize your stop watches!

 

Mavericks Is Here, And It’s Free!

Tuesday, October 22nd, 2013

At 318 we’ve been hard at work preparing for the release of OS X 10.9, Mavericks and OS X Server 3.0. We’ve spent a lot of time writing, testing and filing our findings away. And now the time is here. Mavericks is available on the App Store, with OS X Server and iOS 7.0.3. Additionally, JAMF has released Casper 9.2 and other vendors are releasing patches as quickly as they can.

With new updates to Safari, the addition of iBooks, the new Maps app, better integration of the Calendar, Finder tagging, a new multiple display manager, newer automation features for FileVault, iCloud Keychain, new Notifications and Finder tags, Mavericks is sure to inspire a lot of people to upgrade immediately. Especially now that Apple has announced it’s free! Before you have hundreds of people upgrade in your environment though, check out the new Caching Server 2 in Mavericks Server app. It’s awesome and will help make sure you can still stream some Howard Stern while everyone is doing the upgrade.

OS X Server also gets a bunch of new updates, with Profile Manager 3 adding more that you can control/deploy, Xcode server providing code sharing, new command line options, new options within existing services, etc. Definitely check it out!

And if you need any help with this stuff, we’re more than happy to help prepare for and implement upgrades, whether it’s just for a few clients or for a few thousand!

Enroll Existing 10.8 Machines In Profile Manager (or another MDM) Using Apple Remote Desktop

Thursday, October 17th, 2013

Since we can now do less and less with MCX, we need to rely on Profile Manager for user and machine-specific management inside of OD. This is very easy if you are re-imaging all of your machines (using automated enrollment with Deploy Studio), but what about environments that have upgraded to 10.8 organically? Or if you’re attempting to manage machines that are already in use? If you’ve got ARD or SSH access, you’re in luck, as you can very easily push an Enrollment Profile that will automatically enroll the machine at the next reboot. This is done by manipulating files in /private/var/db/ConfigurationProfiles/Setup/. You can build a set of tasks in ARD to perform the following tasks.

First, we want to rm /private/var/db/ConfigurationProfiles/Setup/.profileSetupDone. At startup, OS X looks for this file and if not found, it will load any profiles found in the containing Setup folder. Make sure this command is run as root.

ARD1

Secondly, we need to actually copy the Enrollment Profile (and Trust Profile if needed) into /private/var/db/ConfigurationProfiles/Setup/. The profiles are loaded in alphabetical order, so make sure the names of the files will ensure that the Trust Profile runs first, or the Enrollment Profile will fail if you have not properly signed your configuration profiles. A good test to see if you need to send the Trust Profile as well is to install the Enrollment Profile you downloaded manually on a fresh test system – if it installs without error, you’re OK to use just that. If you get an error about needing a Trust Profile, either adjust your settings in Server.app to properly sign the Enrollment Profile in server, or install the Trust Profile on the machines as well.

ARD2

Note: You don’t have to install two profiles using most 3rd party MDM solutions, such as Casper, FileWave, etc.

It is important that we get the correct Enrollment Profile to load onto our target machines – you can’t simply log into yourserver.com/mydevices and use the profile that results from the “Enroll” link – this is a per-device file.  We need to get our Enrollment Profile from the admin page (yourserver.com/profilemanager). Once logged in, click the “+” in the bottom left corner and select “Enrollment Profile”.

Image1

Configure the profile as shown below and then download.

Image2

You can also download the Trust Profile from the same screen if needed.

Image3

Once you’ve deleted the .profileSetupDone file and loaded the appropriate profiles, the machines will self-enroll at next reboot (or if you’re a heartless admin, you can force a reboot via ARD). You can now enjoy device management via Profile Manager!

Files Not Showing For Xsan Clients When Uploaded Through Ethernet

Tuesday, October 15th, 2013

There is a problem with Xsan when using AFP or SMB heads in front of volumes, where when a user uploads or adds a file to the volume, the file is not readily available/visible to all users. This issue doesn’t occur every time a file is uploaded and nor does it cause files to actually disappear, only to need the user to restart their Finder in order to be able to see the object.

We’ve been using this freeware app as a workaround until Apple comes up with a patch: https://www.macupdate.com/app/mac/24714/refresh-finder

Add OS X Network Settings Remotely (Without Breaking Stuff)

Monday, September 23rd, 2013

So you’re going to send a computer off to a colocation facility, and it’ll use a static IP and DNS when it gets there, the info for which it’ll need before it arrives. Just like colo, you access this computer remotely to prepare it for its trip, but don’t want to knock it off the network while prepping this info, so you can verify it’s good to go and shut it down.

It’s the type of thing, like setting up email accounts programmatically, that somebody should have figured out and shared with the community as some point. But even if my google-fu is weak, I guess I can deal with having tomatoes thrown at me, so here’s a rough mock-up:

 

#!/bin/bash
# purpose: add a network location with manual IP info without switching 
#   This script lets you fill in settings and apply them on en0(assuming that's active)
#   but only interrupts current connectivity long enough to apply the settings,
#   it then immediately switches back. (It also assumes a 'Static' location doesn't already exist...)
#   Use at your own risk! No warranty granted or implied! Tell us we're doing it rong on twitter!
# author: Allister Banks, 318 Inc.

# set -x

declare -xr networksetup="/usr/sbin/networksetup"

declare -xr MYIP="192.168.111.177"
declare -xr MYMASK="255.255.255.0"
declare -xr MYROUTER="192.168.111.1"
declare -xr DNSSERVERS="8.8.8.8 8.8.4.4"

declare -x PORTANDSERVICE=`$networksetup -listallhardwareports | awk '/en0/{print x};{x=$0}' | cut -d ' ' -f 3`

$networksetup -createlocation "Static" populate
$networksetup -switchtolocation "Static"
$networksetup -setmanual $PORTANDSERVICE $MYIP $MYMASK $MYROUTER
$networksetup -setdnsservers $PORTANDSERVICE $DNSSERVERS
$networksetup -switchtolocation Automatic

exit 0

Caveats: The script assumes the interface you want to be active in the future is en0, just for ease of testing before deployment. Also, that there isn’t already a network location called ‘Static’, and that you do want all interface populated upon creation(because I couldn’t think of particularly good reasons why not.)

If you find the need, give it a try and tweet at us with your questions/comments!


Increase Shared Memory for Postgres

Friday, August 16th, 2013

The default installation of Postgres in OS X Server can be pretty useful. You may find that as your databases grow that you need to increase the amount of shared memory that those databases can access. This is a kernel thing, so requires sysctl to get just right. You can do so manually just to get your heavy lifting done and oftentimes you won’t need the settings to persist across a restart. Before doing anything I like to just grab a snapshot of all my kernel MIBs:

sysctl -a > ~/Desktop/kernmibs

I like increasing these incrementally, so to bring up the maximum memory to 16 megs and increase some of the other settings equally, you might look to do something like this:

sysctl -w kern.sysv.shmmax=16777216
sysctl -w kern.sysv.shmmni=256
sysctl -w kern.sysv.shmseg=64
sysctl -w kern.sysv.shmall=393216

To change back, just restart (or use sysctl -w to load them back in). If you need more for things other than loading and converting databases or patching postgres, then you can bring them up even higher (I like to increment in multiples):

sysctl -w kern.sysv.shmmax=268435456

Change the Maximum Message Size In OS X Mail Server

Wednesday, August 14th, 2013

The default size limit of new mail messages in OS X 10.8 Server is 10 MB. That’s small for many environments. To increase to 20 MB, open Terminal. Use serveradmin to increase the number, plugging it into mail:postfix:message_size_limit as follows:

serveradmin settings mail:postfix:message_size_limit = 20971520

To change it back, you would use:

serveradmin settings mail:postfix:message_size_limit = 10485760

Inconsistent Upgrade Behavior on Software-Mirrored RAID Volumes

Thursday, August 8th, 2013

It came up again recently, so this post is to warn folks treading the same path in the near future. First a little ‘brass tacks’ background: As you probably know, as of 10.7 Lion’s Mac App Store-only distribution, you can choose the option to extract the InstallESD.dmg from the Install Mac OS X (insert big cat name here) application, and avoid duplicitous downloads and manual Apple ID logins. One could even automate the process on a network that supports NetInstall with a redundantly named NetInstall set to essentially ‘virtualize’ or serve up the installer app on the network.

We’ve found recently that more than a few environments are just getting around to upgrading after taking a ‘wait and see’ approach to Lion, and jumping straight to 10.8 Mountain Lion. Getting to the meat after all this preamble… it was also, at one time, considered best practice to use RAID to mirror the boot disk, even without a hardware card to remove the CPU overhead. (It hadn’t been considered a factor then, but even modern storage virtualization *cough*Drobo*cough* can perform… poorly. I personally recommend what I call a ‘lazy mirror’, having CCC clone the volume and putting less writes on the disk over time, and getting the redundancy of CCC reporting SMART status of the source and destination.)

When upgrading a software-mirror’d boot drives OS, you get a message about features been unavailable, namely FileVault2 and the Recovery Partition it relies upon. If it detects the machine being upgraded is running (a relic of a bygone era, a separate OS called quaintly) ‘Mac OS X Server,’ it additionally warns that the server functionality will be suspended until Server.app 2.x can be installed via… the Mac App Store. We’ve found it can do an upgrade of those paused services(at least those that are still provided by the 2.2.1 version of the Server application) and pick up where it left off without incident after being installed and launched.

If, however, you use a Mac App Store-downloaded application to perform the process, we’ve seen higher success rates of a stable upgrade. If instead you tried to save time with either the InstallESD.dmg or NetInstall set methods mentioned earlier, a failure symptom occurred that, post-update, the disk would never complete its first boot(verbose boot was not conclusive as to reasons, either.) Moving the application bundle to another machine(volume license codes have, of course, been applied to the appropriate AppleID on the machines designated for upgrades,) hasn’t been as successful, although the recommended repackaging of the Install app, as Apple has referred to in certain documentation, wasn’t attempted this particular time. In some cases even breaking the software mirror didn’t allow the disk to complete an upgrade successfully. Another symptom before we could tell it was going to fail is the drop-down sheet warning of the loss of server functionality would immediately cause the entire window to lose focus while about to initiate the update. A radar has not been filed due to the fact that a supported(albeit semi time-intensive) method exists and as been more consistently successful.

Allister’s Talks From Penn State MacAdmins

Wednesday, June 5th, 2013

BCC Mail In OS X Server

Thursday, April 4th, 2013

OS X Server has the ability to bcc mail that flows through it. This can be a good way to keep a copy of mail for the purposes of things like legal requirements. To enable this feature, once upon a time you could use the GUI in OS X Server. These days, the feature is still there but is now accessed through the command line as the always_bcc_enabled option within serveradmin’s mail settings. To enable this option, use the following command:

sudo serveradmin settings mail:postfix:always_bcc_enabled = yes

Once enabled, you will also need to supply an actual address to bcc mail to, which is done using always_bcc as follows:

sudo serveradmin settings mail:postfix:always_bcc = "backup@318.com

Next, you’ll want to

sudo serveradmin stop mail
sudo serveradmin start mail

Finally, if there are any issues, putting the postfix logging facility into debug mode can help you triangulate, done using the following command (and restarting the mail service again):

sudo serveradmin settings mail:postfix:log_level = "debug"

LOPSA-East 2013

Monday, March 18th, 2013

For the first year I’ll be speaking at the newly-rebranded League of Extraordinary Gentlemen League of Professional System Administrators conference in New Brunswick, New Jersey! It’s May 3rd and 4th, and should be a change from the Mac-heavy conferences we’ve been associated with as of late. I’ll be giving a training class, Intro to Mac and iOS Lifecycle Management, and a talk on Principled Patch Management with Munki. Registration is open now! Jersey is lovely that time of year, please consider attending!

 

LOPSA-East '13

PSU MacAdmins Conference 2013

Wednesday, February 27th, 2013

It's Secret!

For the third year, I’ll be presenting at PSU MacAdmins Conference! This year I’m lucky enough to be able to present two talks, “Backup, Front to Back” and “Enough Networking to be Dangerous”. But I’m really looking forward to what I can learn from those speaking for the first time, like Pepijn Bruienne and Graham Gilbert among others. The setting and venue is top-notch. It’s taking place May 22nd through the 24th, with a Boot Camp for more foundational topics May 21st. Hope you can join us!

QuickTip: Retrospect

Saturday, February 2nd, 2013

False-positive errors muddy our view, so just as many strive to quiet noisy syslogs, we can easily overcome a common complaint we see Retrospect make when doing differential SQL backups: ‘the master DB can’t be DIFF’d’.

Complicating matters is Retro’s SQL connector doesn’t seem to allow you to exclude that one as a source. Simply make an exclusion for ‘file or folder name matches: master’ and you’ll see it get to the master in the log and decide no files necessary to backup, keeping the result of that script error-free.

Sure, We Have a Mac Client, We Use Java!

Thursday, January 24th, 2013

We all have our favorite epithets to invoke for certain software vendors and the practices they use. Some of our peers go downright apoplectic when speaking about those companies and the lack of advances we perceive in the name of manageable platforms. Not good, life is too short.

I wouldn’t have even imagined APC would be forgiving in this respect, they are quite obviously a hardware company. You may ask yourself, though, ‘is your refrigerator running’ is the software actually listening for a safe shutdown signal from the network card installed in the UPS? Complicating matters is:
- The reason we install this Network Shutdown software from APC on our server is to receive this signal over ethernet, not USB, so it’s not detected by Energy Saver like other, directly cabled models

- The shutdown notifier client doesn’t have a windowed process/menubar icon

- The process itself identifies as “Java” in Activity Monitor (just like… CrashPlan – although we can kindof guess which one is using 400+ MBs of virtual memory idle…)

Which sucks. (Seriously, it installs in /Users/Shared/Applications! And runs at boot with a StartupItem! In 2013! OMGWTFBBQ!)

Calm, calm, not to fear! ps sprinkled with awk to the rescue:

ps avx | awk '/java/&&/Notifier/&&!/awk/{print $17,$18}'

To explain the ps flags, first it allows for all users processes, prints in long format with more criteria, and the x is for even if they have no ‘controlling console.’ Then awk looks for both Java and the ‘Notifier’ jar name, minus our awk itself, and prints the relevant fields, highlighted below(trimmed and rewrapped for readability):

:./comp/pcns.jar:./comp/Notifier.jar: 

com.apcc.m11.arch.application.Application

So at least we can tell that something is running, and appreciate the thoughtful development process APC followed, at least while we aren’t fashioning our own replacement with booster serial cables and middleware. Thanks to the googles and the overflown’ stacks for the proper flags to pass ps.

iOS Backups Continued, and Configuration Profiles

Friday, December 14th, 2012

In our previous discussion of iOS Backups, the topic of configuration profiles being the ‘closest to the surface’ on a device was hinted at. What that means is, when Apple Configurator restores a backup, that’s the last thing to be applied to the device. For folks hoping to use Web Clips as a kind of app deployment, they need to realize that trying to restore a backup that has the web clip in a particular place doesn’t work – the backup that designates where icons on the home screen line up gets laid down before the web clip gets applied by the profile. It gets bumped to whichever would be the next home screen after the apps take their positions.

This makes a great segue into the topic of configuration profiles. Here’s a ‘secret’ hiding in plain sight: Apple Configurator can make profiles that work on 10.7+ Macs. (But please, don’t use it for that – see below.) iPCU possibly could generate usable ones as well, although one should consider the lack of full screen mode in the interface as a hint: it may not see much in the way of updates on the Mac from now on. iPCU is all you have in the way of an Apple-supported tool on Windows, though. (Protip: activate the iOS device before you try to put profiles on it – credit @bruienne for this reminder.)

Also thanks to @bruienne to the recommendation of the slick p4merge tool

Also thanks to @bruienne to the recommendation of the slick p4merge tool

Now why would you avoid making, for example, a Wi-Fi configuration profile for use on a mac with Apple Configurator? Well there’s one humongous difference between iOS and Macs: individual users. Managing devices with profiles shows Apple tipping their cards: they seem to be saying you should think of only one user per device, and if it’s important enough to manage at all, it should be an always enforced setting. The Profile Manager service in Lion and Mountain Lion Server have an extra twist, though: you can push out settings for Mac users or the devices they own. If you want to manage a setting across all users of a device, you can do so at the Device Group level, which generates extra keys than those that are present in a profile generated by Apple Configurator. The end result is that a Configurator-generated profile will be user-specific, and fail with deployment methods that need to target the System. (Enlarge the above screenshot to see the differences – and yes, there’s a poorly obscured password in there. Bring it on, hax0rs!)

These are just more of the ‘potpourri’ type topics that we find time to share after being caught by peculiarities out in the field.

Introducing Splunk: Funny name, serious logging

Thursday, November 15th, 2012

So, my boss says:

“Write an article called ‘Getting Started with Splunk.’”

I reply:

“What, you think I know all this stuff? This really would be a getting started article.”

But here it is and WOW is Splunk cool!

My only experience with Splunk up to a couple days ago was seeing a T-shirt with “Log is my copilot”. I knew it had something to do with gathering log files and making them easier to read and search. In about an hour I had gone to Splunk’s website to research the product, downloaded and installed it, and started viewing logs from my own system. The Splunk folks have made getting their product into their customer’s hands easy and getting started even easier.

What is Splunk?

Simply put, Splunk can gather just about any kind of data that goes into a log (system logs, website metrics, etc.) into one place and make viewing that data easy. It’s accessed via web browser so it’s accessible on any computer or mobile device such as an iPad.

What do I need to run Splunk?

Practically any common operating system today can run Splunk: Mac OS X, Linux, Windows, FreeBSD and more.

How much does Splunk cost?

Don’t worry about that right now. Download and install the free version. It takes minutes to install and is a no-brainer. Let’s get started.

Getting Splunk

IT managers and directors may be interested in watching the introductory and business case videos with the corporate speak (“operational intelligence” anyone?) and company endorsements. Techs will be interested in getting started. Right on their home page is a big green Free Download button. Go there, click it and locate the downloader for your OS of choice. I downloaded the Mac OS X 10.7 installer to test (and installed it on OS X 10.8 without any issues).

Splunk home

This does require a sign-up to create an account. It takes less than a minute to complete. After submitting the information the 100 MB download begins right away.

While waiting for the download…

When the download is on its way the Splunk folks kindly redirect to a page with some short videos to watch while waiting. Watch this first one called Getting data into Splunk. It’s only a few minutes and this is the first thing to do after getting into Splunk.

Installing and starting Splunk

The download arrives as a double-clickable Apple Installer package. Double-click and install it. Toward the end it opens a simple TextEdit window with instructions for how to start, stop and access the newly installed Splunk site.

Install done

Files are installed in /Applications/splunk and resemble a UNIX file system.

Splunk application folder

Open the Terminal application found in /Applications/Utilities and run the command /Applications/splunk/bin/splunk start. If this is the first time running Splunk it prompts to accept its license agreement. Tap the spacebar to scroll through and read the agreement or type “q” to quit and agree to the license.

EULA

Accepting the agreement continues to start Splunk where it displays some brief setup messages.

Starting Splunk

The setup then provides the local HTTP address for the newly installed Splunk site. Open this in a web browser to get to the login screen. The first login requires that the administrator account password be reset.

Splunk login

Following along with the Getting data into Splunk video, Splunk will need some information. Mac OS X stores its own log files. Let’s point to those.

Click the Add Data link to begin.

New Splunk home

Since Mac OS X’s log files are local to the machine, click A file or directory of files.

Add files

Click Next to specify local files.

Add local logs

This opens a window that exposes not only Mac OS X’s visible folders but its invisible folders as well. Browse to /var/log/system.log and click the Select button.

Browse logs folder

For now, opt to skip previewing the log file and click Continue.

Path to system.log

Now, let’s opt to monitor not only the system.log file but the entire /var/log folder containing dozens of other log files as well. Note that Splunk can watch rotated and zipped log files too. Click Save to finish adding logs.

Add /var/log folder

Let’s start searching!

Succes, start searching

The Search window initially displays a list of all logs Splunk is monitoring. To narrow the search change the time filter drop down menu to Last 60 minutes. This will make the results a little easier to see on a system that’s only been running a short while.

Last 24 hours

Now, search for install*. Splunk will only search for the word “install” without providing the asterisk as a wildcard character. Splunk supports not only wildcard searches but booleans, parentheses, quotes, etc. It will return every instance recorded in the logs that matches the search criteria. It also creates an interactive bar chart along the top of the page to indicate the number of occurrences found for the search at particular times.

Search for install

To further refine the search, Option+click most any word in the log entries below and Splunk will automatically add the necessary syntax to remove an item. In this case the install* search returned installinstaller and installd. Option+clicking installd changed the search criteria to install* NOT installd.

Modified search

Now what?

Continue exploring the videos to understand Splunk’s possibilities and take advantage of its Splunk Tutorial, which is available online as well as in PDF format for offline viewing. They do a great job leading users through setup and creating reports.

Still asking about price? Good.

The free version remains free but doesn’t include many features that really make it sing such as monitoring and alerts, multiple user accounts and support beyond the Splunk website. Cost depends primarily on the amount of data you want to suck into Splunk and have it watch. It’s not cheap but for an enterprise needing to meet certain service level requirements it beats browsing through multiple servers trying to find the right log with the right information.

FYI, putting together this 1,000-word article probably took me 10 times longer than performing the Splunk install itself and beginning to learn it. It’s really well-done and easy to use. Splunk makes getting started simple.

(Sysadmin) Software Design Decisions

Wednesday, October 3rd, 2012

When approaching a task with an inkling to automate, sometimes you find an open source project that fits the bill. But the creator will work within constraints, and often express their opinion of what’s important to ‘solve’ as a problem and therefore prioritize on: a deployment tool is not necessarily a patch management tool is not necessarily a configuration management tool, and so on. One of the things I’ve dealt with is trying to gauge the intent of a developer and deciding if they are interested in further discussion/support/development of a given project. Knowing why one decision was made or another can be helpful in these situations. In that category of things I wish someone could have written so I could read it, here’s the design decisions behind the sonOfBackupRestoreScripts project I’ve been toying with as an add-on to DeployStudio(heretofore DS), which you can hopefully understand why I am not releasing as an official, supportable tool in it’s current bash form after reading the following.
I’ve adapted some of the things Google used in their outline for Simian as a model, to give this some structure.

Project Objective:

To move user home folders and local authentication/cached credentials between workstations in a customizable and optimized manner, preserving the integrity of the data/user records as much as possible

Overview:

For speed and data integrity, rsync is used to move selections of the users home folder(minus caches, trash, and common exclusions made by Time Machine). To increase portability and preserve mac-specific attributes, a disk image is generated to enclose the data. The user account information is copied separately and helpful information is displayed at the critical points as it moves from one stage to another and during the backup itself.

Requirements: DeployStudio Server / NetBoot

DS, as a service, enables an infrastructure to run the script in, and automounts a repository to interact with over the network. Meant to work optimally with or without a NetBoot environment, an architecture assumption being made during development/testing is wired ethernet, with the use of USB/Thunderbolt adapters if clients are MacBook Airs. Even old minis can function fine as the server, assuming the repo is located on a volume with enough space available to accept the uncompressed backups.

Implementation Details: Major Components / Underlying Programs

- source/destination variables

Parameters can be passed to the script to change the source/destination of backups/restores with the -s(for source) and -d(…) switches and then a path that is reachable by the NetBooted system.

- hdiutil

A simple sparsediskimage is created which can expand up to 100GBs with the built-in binary hdiutil. The file system format of that container is JHFS+, and a bunch of other best practices, cobbled together from Bombich’s Carbon Copy Cloner(heretofore CCC) and InstaDMG, are employed.

- cp

The cp binary is used to just copy the user records from the directory service the data resides on to the root of the sparseimage, and the admin group’s record is copied into a ‘group’ folder. If hashes exist in /var/db/shadow/hash, which is how passwords were stored previous to 10.7, those are moved to a ‘hashes’ folder.

- rsync

A custom, even more current build of rsync could be generated if the instructions listed here are followed. Ideally, a battle-tested version like the one bundled with CCC’s ccc_helper.app (/Applications/Carbon\ Copy\ Cloner.app/Contents/MacOS/ccc_helper.app/Contents/MacOS/rsync, which is actually a heavily customized rsync version 3.0.6) could be used, but it’s output isn’t easy to adapt and see an overview of the progress during a CLI transfer. Regardless, the recommended switches are employed in hopes to get a passing grade on the backupBouncer test. The 3.0.7 version bundled with DS itself (/Applications/Utilities/DeployStudio\ Admin.app/Contents/Frameworks/DSCore.framework/Versions/A/Resources/Tools/rsync, which for whatever reason is excluded when the assistant creates NetBoot sets) was used during development/testing.

-Exclusions

The Users folder on the workstation that’s being backed up is what’s targeted directly, so any users that have been deleted or subfolders can be removed with the exclusions file fed to the rsync command, and without catch-all, asterisk(*) ‘file globbing’, you’d need to be specific about certain types of files you want to exclude if they’re in certain directories. For example, to not backup any mp3 files, no matter where they are in the user folders being backed up, you’d add - *.mp3 Additional catch-all excludes could be used, as detailed in the script, which specifically excludes ipsw’s(iOS firmware/OS installers) like this: --exclude='*.ipsw'

-Restore

Pretty much everything done via both rsync and cp are done in reverse, utilizing the source/destination options, so a backup taken from one machine can easily be chosen to restore to another.

Security Considerations:

Very little security is applied during storage. Files are transferred over password-protected AFP, so a separate server and repo could be used to minimize potential access by whoever can access the main DS service. Nothing encrypts the files inside the sparseimages, and if present, the older password format is a hash that could potentially be cracked over a great length of time. The home folder ACL’s and ownership/perms are preserved, so in that respect it’s secure according to whoever has access to the local file systems on the server and client.

Excluded/Missing Features:
(Don’t You Wish Every Project Said That?)

Hopefully this won’t sound like a soul-bearing confession, but here goes:
No checks are in place if there isn’t enough space on destinations, nor if a folder to backup is larger than the currently hard-coded 100GB sparseimage cap (after exclusions.) Minimal redirection of logs is performed, so the main DS log can quickly hit a 2MB cap and stop updating the DS NetBoot log window/GUI if there’s a boatload of progress echo’d to stdout. The process to restore a users admin group membership(or any other group on the original source) is not performed, although the group’s admin.plist can be queried after the fact. Nor is there even reporting on Deleted Users orphaned home folders if they do actually need to be preserved, by default they’re just part of the things rsync excludes. All restrictions are performed in the Excludes.txt file fed to rsync, so it cannot be passed as a parameter to the script.
And the biggest possible unpleasantness is also the #1 reason I’m not considering continuing development in bash: UID collisions. If you restore a 501 user to an image with a pre-existing 501 user that was the only admin… bad things will happen. (We’ve changed our default admin user’s UID as a result.) If you get lucky, you can change one user’s UID or the other and chown to fix things as admin before all heck breaks lose… If this isn’t a clean image, there’s no checking for duplicate users with newer data, there’s no filevault1 or 2 handling, no prioritization so if it can only fit a few home folders it’ll do so and warn about the one(s) that wouldn’t fit, no version checking on the binaries in case different NetBoot sets are used, no fixing of ByHostPrefs(although DS’s finalize script should handle that), no checks with die function are performed if the restore destination doesn’t have enough space, since common case is restoring to same HD or a newer, presumably larger computer. Phew!

Wrapup:

The moral of the story is that the data structures available in most of the other scripting languages are more suited for these checks and to perform evasive action, as necessary. Bash does really ungainly approximations of tuples/dictionaries/hash tables, and forced the previous version of this project to perform all necessary checks and actions during a single loop per-user to keep things functional without growing exponentially longer and more complex.

Let’s look forward to the distant future when this makes it’s way into Python for the next installment in this project. Of course I’ve already got the name of the successor to SonOfBackupRestoreScripts: BrideOfBackupRestoreScripts!

DeployStudio Scripting Tips

Tuesday, September 25th, 2012

I’ve given a presentation on my affinity for DeployStudio, yet with it being closed source, it’s sometimes like an enigma (wrapped in a mystery (wrapped in bacon)). However, a workflow exists to enable scripting within it, although the only option besides automatically running it when dropped into a workflow is non-interactively passing arguments to a script. Even with little in the way of documented information, projects have popped up to take advantage of the framework provided.

Most noticeably, Rusty Myers’ BackupRestore scripts enabled quite an interesting workflow: first, you could run one workflow to tar(or ditto) the user folders to a new Backup directory in the repo, with a few customizable exceptions. And then, when either specified or dropped into a workflow that had a restore action precede it, you could put the users and their associated passwords back into place. This is obviously pretty darn useful for clean(er) migrations and/or OS upgrades, or simply refreshing existing workstations with a new base set of software. Many folks in the MacEnterprise community contributed features, including FileVault(version 1) support, and updates were made for the curveball Lion introduced in respect to how passwords are stored(nested inside the user record plist itself.)

I’m in the process of creating a successor to this project, so I thought I’d share some of the experiences I’ve had and pointers I’ve come across as a sysadmin, not a software developer, attempting to get a repeatable task accomplished inside this framework. Tip number zero is the same advice given to all students of programming in scripting languages: don’t write a lot before running and getting feedback. So, I booted a laptop with a few small user folders to my DeployStudio-generated netboot set, authenticated to the interface, and opened terminal. That netboot set includes the optional Python framework(Ruby is another option if you’d like access to that scripting language), which I’ll be using in the future. Along with selecting “Display Runtime log window by default”, I extended the “Quit runtime automatically after__” number to 90 seconds, so when testing inside of a workflow I wouldn’t be kicked out as I iterated and repeated the process.

To get started, I made an “admin” folder inside the repo, put an updated version of the rsync binary in it(since the one in /usr/bin that ships with OS X is pretty ancient), and started writing a script in that directory which I could therefore run from terminal on the netbooted laptop over VNC/ARD. For starters, here’s tip #1:
DeployStudio mounts the repo in /tmp/DSNetworkRepository. While /tmp isn’t read only, you will get out-of-space errors and general unhappiness if you use it for anything except mountpoints.
Tip #2:
No, you can’t use symlinks in those subfolders to point somewhere else on the DeployStudio server, since it assumes the directory the symlink would point to is relative to the root of the netboot set instead. (No, really, it wouldn’t even work when using ln -s on the machine the repo’s hosted from instead of Finder’s aliases, which definitely don’t work in scripts.)
tip #3:
For persnickety folks like myself that MUST use a theme in terminal and can’t deal to not have option set as the meta key, you’re probably bummed that the Preferences menu item is greyed out and command-comma doesn’t seem to work. There is a way, though: from the Shell menu choose Show Inspector. Then from the settings tab, double-click the theme you prefer. The full settings window will appear, and you can have your modern conveniences again.
tip #4:
How does DeployStudio decide what is the first mounted volume, you may wonder? I invite(dare?) you to ‘bikeshed‘(find a more optimized way to accomplish a relatively trivial task) this particular piece of code:
system_profiler SPSerialATADataType | awk -F': ' '/Mount Point/ { print $2}' | head -n1
In the case of the restore script, hard-coding the DS_LAST_RESTORED_VOLUME variable (on line 44 in Rusty’s current Lion version), or changing the argument in the workflow to pass that path with the -v flag will remove the dependency on restoring an image before putting user folders back in place.

Two more tips before I go, which are both specific to the task I’m trying to accomplish. Ownership on files will not be preserved when moved to the repo with rsync, so you can create a sparse image or sparsebundle as a destination, and it will even retain ACL’s (certain patched rsync binaries complain about smb_acl’s, like the one I used, which is bundled in DeployStudio’s Tools folder.) As mentioned about /tmp in the NetBoot environment earlier, sparseimages should be created in a subfolder of the repo, or you could easily run into ‘out of space’ errors.

When logging, you can use tee or simply redirect output, but more customized feedback in the actual log window in the Deploy Studio netboot runtime is helpful. There’s a “custom_logger” function used in some of the bundled scripts… which literally does nothing but echo $1 – pretty rudimentary. For output that doesn’t display when run as part of a script, you can redirect that output to /dev/stdout and have it shown instead of fooling around with echo or printf.
e.g. rsync -avz /source /destination >/dev/stdout
There may be a lag if verbose output happens in a short amount of time, as the main log file in the repo is being written to simultaneously to what is printed onscreen.

Oh, and the silliest thing I’ve noticed: your script needs to be owned by root:wheel and 777 in the scripts folder of the repo in order to show up in the workflow interface for selection… It’s got it’s quirks, but it’s just about the best out there!

MacSysAdmin 2012 Slides and Videos are Live!

Thursday, September 20th, 2012

318 Inc. CTO Charles Edge and Solutions Architect alumni Zack Smith were back at the MacSysAdmin Conference in Sweden again this year, and the slides and videos are now available! All the 2012 presentations can be found here, and past years are at the bottom of this page.

Configuration Profiles: The Future is… Soon

Tuesday, September 4th, 2012

Let’s say, hypothetically, you’ve been in the Mac IT business for a couple revisions of the ol’ OS, and are familiar with centralized management from a directory service. While we’re having hypothetical conversations about our identity, let’s also suppose we’re not too keen on going the way of the dinosaur via planned obsolescence, so we embrace the fact that Configuration Profiles are the future. Well I’m here to say that knowing how the former mechanism, Managed Preferences (née Managed Client for OS X, or MCX) from a directory service interacted with your system is still important.

 
Does the technique to nest a faux directory service on each computer locally(ergo LocalMCX), and utilize it to apply management to the entire system still work in 10.8 Mountain Lion? Yes. Do applied Profiles show settings in the same directory Managed Preferences did? Yes… which can possibly cause conflicts. So while practically, living in the age of Profiles is great when 802.1x used to be so hard to manage, there are pragmatic concerns as well. Not everyone upgraded to Lion the moment it was released, just over a year ago, so we’re wise to continue using MCX first wherever the population is significantly mixed.

 
And then there’s a show-stopper through which Apple opened up a Third-Party Opportunity (trademark Arek Dreyer): Profiles, when coming straight out of Mac OS X Server’s 10.7 or 10.8 Profile Manager service, can only apply management with the Always frequency. Just like the former IBM Thinkpads, you could have any color you wanted as long as it’s black. No friendly defaults set by your administrator that you can change later, Profiles settings, once applied, was basically frozen in carbonite.

 
So what party stepped in to address this plight? ##osx-server discussions were struck up by the always-generous never-timid @gregneagle, about the fact that Profiles can actually, although undocumented, contain settings that enable Once or Often frequency. Certain preferences can ONLY by managed at the Often level, because they aren’t made to be manageable system-wide, like certain application (e.g. Office 2011) and Screen Saver preferences (since those live in the user’s ~/Library/ByHost folder.)

 
The end result was Tim Sutton’s mcxToProfile script, hosted on Github, which works like a champ for both of the examples just listed. Note: this script utilizes a Profiles Custom Settings section only, so for the things already supported by Profiles (like loginwindow) it’s certainly best to get onboard with what the $20 Server.app can already provide. But another big plus of the script is… you can use it script without having ProfileManager set up anywhere on your network.

 
So there you go, consider using mcxToProfile to update your management, and give feedback to Tim on the twitters or the GitHubs!

Mountain Lion Now Available On The App Store

Wednesday, July 25th, 2012

Mountain Lion and Mountain Lion Server are now available on the OS X App Store. With a host of new features, notably including the ability to mirror desktops to Apple TV, Notification Center, Messages and other features, Mountain Lion is sure to be a boon to many an organization. But some features are also now missing, such as DHCP and Podcast Producer in OS X Server and of all things, negative mode.

Mountain Lion is one of the more incremental OS updates that Apple has released and so migrating from Lion to Mountain Lion is sure to be a breeze for many environments. But it’s still a major OS update and so should be taken carefully, as new features such as Gatekeeper are likely to give some environments fits. Before jumping into the App Store, be aware of all the various aspects of deployment. 318 can work with customers to plan and implement such migrations. Call your Professional Services Manager or contact sales@318.com for more information.

Evaluating the Tokens, or Order of Expansion in Bash

Monday, July 23rd, 2012

Previously in our series on commonly overlooked things in bash, we spoke about being specific with the binaries our script will call, and mentioned conventions to use after deciding what to variable-ize. The rigidity and lack of convenience afforded by bash starts to poke through when we’re trying to abstract re-usable inputs through making them into variables, and folks are commonly tripped up when trying to have everything come out as intended on the other side, when that line runs. You may already know to put quotes around just about every variable to catch the possibility of spaces messing things up, and we’re not even touching on complex ‘sanitization’ of things like non-roman alphabets and/or UTF-8 encoding. Knowing the ‘order of operation expansion’ that the interpreter will use when running our scripts is important. It’s not all drudgery, though, as we’ll uncover features available to bash that you may not have realized exist.

For instance, you may know curly braces can be used did you know there’s syntax to, for example, expand to multiple extensions for the same filenames by putting them in curly braces, comma-separated? An interactive example(with set -x)
cp veryimportantconfigfile{,-backup}
+ cp veryimportantconfigfile veryimportantconfigfile-backup

That’s referred to as filename (or just) brace expansion, and is the first in order of the (roughly) six types of expansion the bash interpreter goes through when evaluating lines and ‘token-ized’ variables in a script.

Since you’re CLI-curious and go command line (trademark @thespider) all the time, you’re probably familiar with not only that you can use tilde(~) for a shortcut to the current logged-in users home directory, but also that just cd alone will assume you meant you wanted to go to that home directory? A users home gets a lot of traffic, and while the builtin $HOME variable is probably more reliable if you must include interaction with home directories in your script, tilde expansion (including any subdirectories tagged to the end) is the next in our expansion order.

Now things get (however underwhelmingly) more interesting. Third in the hit parade, each with semi-equal weighting, are
a. the standard “variable=foo, echo $variable” style ‘variable expressions’ we all know and love,
b. backtick-extracted results of commands, which can also be achieved with $(command) (and, worse came to worse, you could force another expansion of a variable with the eval command)
c. arithmetic expressions (like -gt for greater than, equal, less than, etc.) as we commonly use for comparison tests,
and an interesting set of features that are actually convenient (and mimic some uses of regular expressions), called (misleadingly)
$. dollar sign substitution. All of the different shorthand included under this category has been written about elsewhere in detail, but one in particular is an ad-hoc twist on a catchall that you could use via the ‘shell options’, or shopt command(originally created to expand on ‘set‘, which we mentioned in our earlier article when adding a debug option with ‘set -x‘). All of the options available with shopt are also a bit too numerous to cover now, but one that you’ll see particularly strict folks use is ‘nounset‘, to ensure that variables have always been defined if they’re going to be evaluated as the script runs. It’s only slightly confusing that a variable can have an empty string for a value, which would pass this check. Often, it’s the other way around, and we’ll have variables that are defined without being used; the thing we’d really like to look out for is when a variable is supposed to have a ‘real’ value, and the script could cause ill affects by running without one – so the question becomes how do we check for those important variables as they’re expanded?
A symbol used in bash that will come up later when we cover getopt is the colon, which refers to the existence of an argument, or the variables value (text or otherwise) that you’d be expecting to have set. Dollar sign substitution mimics this concept when allowing you to ad hoc check for empty (or ‘null’) variables by following a standard ‘$variable‘ with ‘:?’ (finished product: ${variable:?})- in other words, it’s a test if the $variable expanded into a ‘real’ value, and it will exit the script at that point with an error if unset, like an ejector seat.

Moving on to the less heavy expansions, the next is… command lookups in the run environments PATH, which are evaluated like regular(western) sentences, from left to right.
As it traipses along down a line running a command, it follows that commands rules regarding if it’s supposed to expect certain switches and arguments, and assumes those are split by some sort of separation (whitespace by default), referred to as the Internal Field Separator. The order of expansion continues with this ‘word splitting’.

And finally, there’s regular old pathname pattern matching – if you’re processing a file or folder in a directory, it find the first instance that matches to evaluate that – pretty straightforward. You may notice we’re often linking to the Bash Guide for Beginners site, as hosted by The Linux Documentation Project. Beyond that resource, there’s also videos from 2011(iTunesU link) and 2012(youtube) Penn State Mac Admins conference on this topic if you need a refresher before we forge ahead for a few more posts.

Back to Basics with Bash

Tuesday, July 17th, 2012

The default shell for Macs has been bash for as long as some of us can remember(as long as we forget it was tcsh through 10.2.8… and before that… there was no shell, it was OS 9!) Bash as a scripting language doesn’t get the best reputation as it is certainly suboptimal and generally unoptimized for modern workflows. To get common things done you need to care about procedural tasks and things can become very ‘heavy’ very quickly. With more modern programming languages that have niceties like API’s and libraries, the catchphrase you’ll hear is you get loads of functionality ‘for free,’ but it’s good to know how far we can get, and why those object-oriented folks keep telling us we’re missing out. And, although most of us are using bash every time we open a shell(zsh users probably know all this stuff anyway) there are things a lot of us aren’t doing in scripts that could be better. Bash is not going away, and is plenty serviceable for ‘lighter’, one-off tasks, so over the course of a few posts we’ll touch on bash-related topics.

Some things even a long-time scripter may easily overlook is how we might set variables more smartly and _often_, making good decisions and being specific about what we choose to variable-ize. If the purpose of a script is to customize things in a way that’s reusable, making a variable out of that customization (say, for example, a hostname or notification email address) allows us to easily re-set that variable in the future. And in our line of work, if you do something once, it is highly probable you’ll do it again.

Something else you may have seen in certain scripts is the PATH variable being explicitly set or overridden, under the assumption that may not be set in the environment the script runs in, or the droids binaries we’re looking for will definitely be found once we set the path directories specifically. This is well-intentioned, but imprecise to put it one way, clunky to put it another. Setting a custom path, or having binaries customized that could end up interacting with our script may cause unintended issues, so some paranoia should be exhibited. As scientists and troubleshooters, being as specific as possible always pays returns, so a guiding principle we should consider adopting is to, instead of setting the path and assuming, make a variable for each binary called as part of a script.

Now would probably be a good time to mention a few things that assist us when setting variables for binaries. Oh, and as conventions go, it helps to leave variable names that are set for binaries as lowercase, and all caps for the customizations we’re shoving in, which helps us visually see only our customized info in all caps as we debug/inspect the script and when we go in to update those variables for a new environment. /usr/bin/which tells us what is the path to the binary which is currently the first discovered in our path, for example ‘which which’ tells us we first found a version of ‘which’ in /usr/bin. Similarly, you may realize from its name what /usr/bin/whereis does. Man pages as a mini-topic is also discussed here. However, a more useful way to tell if you’re using the most efficient version of a binary is to check it with /usr/bin/type. If it’s a shell builtin, like echo, it may be faster than alternatives found at other paths, and you may not even find it necessary to make a variable for it, since there is little chance someone has decided to replace bash’s builtin ‘cd’…

The last practice we’ll try to spread the adoption of is using declare when setting variables. Again, while a lazy sysadmin is a good sysadmin, a precise one doesn’t have to worry about as many failures. A lack of portability across shells helped folks overlook it, but this is useful even if it is bash-specific. When you use declare and -r for read-only, you’re ensuring your variable doesn’t accidentally get overwritten later in the script. Just like the tool ‘set’ for shell settings, which is used to debug scripts when using the xtrace option for tracing how variables are expanded and executed, you can remove the type designation from variables with a +, e.g. set +x. Integers can be ensured by using -i (which frees us from using ‘let’ when we are just simply setting a number), arrays with -a, and when you need a variable to stick around for longer than the individual script it’s set in or the current environment, you can export the variable with -x. Alternately, if you must use the same exact variable name with a different value inside a nested script, you can set the variable as local so you don’t ‘cross the streams’. We hope this starts a conversation on proper bash-ing, look forward to more ‘back to basics’ posts like this one.

Pass the Time With Easily-Accessible Conference Videos

Thursday, June 7th, 2012

It was the pleasure of two 318′ers to attend and present at PSU Mac Admins Conference last month, and lickety-split, the videos have been made available! (Slides should appear shortly.) Not only can you pass the time away with on-demand streaming from YouTube, you can also download them for offline access (like the plane) at the iTunesU channel. Enjoy the current most popular video!

Setting Up Time Machine Server in Lion Server

Sunday, May 20th, 2012

Setting Up The Podcast Service In Lion Server Video

Friday, May 18th, 2012

Video On Setting Up Lion Server As A Software Update Server

Monday, May 14th, 2012

Video On Setting Up File Sharing In Lion Server

Friday, May 11th, 2012

Creating Users and Groups in Lion Server

Thursday, May 10th, 2012