Posts Tagged ‘Mac App Store’

Inconsistent Upgrade Behavior on Software-Mirrored RAID Volumes

Thursday, August 8th, 2013

It came up again recently, so this post is to warn folks treading the same path in the near future. First a little ‘brass tacks’ background: As you probably know, as of 10.7 Lion’s Mac App Store-only distribution, you can choose the option to extract the InstallESD.dmg from the Install Mac OS X (insert big cat name here) application, and avoid duplicitous downloads and manual Apple ID logins. One could even automate the process on a network that supports NetInstall with a redundantly named NetInstall set to essentially ‘virtualize’ or serve up the installer app on the network.

We’ve found recently that more than a few environments are just getting around to upgrading after taking a ‘wait and see’ approach to Lion, and jumping straight to 10.8 Mountain Lion. Getting to the meat after all this preamble… it was also, at one time, considered best practice to use RAID to mirror the boot disk, even without a hardware card to remove the CPU overhead. (It hadn’t been considered a factor then, but even modern storage virtualization *cough*Drobo*cough* can perform… poorly. I personally recommend what I call a ‘lazy mirror’, having CCC clone the volume and putting less writes on the disk over time, and getting the redundancy of CCC reporting SMART status of the source and destination.)

When upgrading a software-mirror’d boot drives OS, you get a message about features been unavailable, namely FileVault2 and the Recovery Partition it relies upon. If it detects the machine being upgraded is running (a relic of a bygone era, a separate OS called quaintly) ‘Mac OS X Server,’ it additionally warns that the server functionality will be suspended until Server.app 2.x can be installed via… the Mac App Store. We’ve found it can do an upgrade of those paused services(at least those that are still provided by the 2.2.1 version of the Server application) and pick up where it left off without incident after being installed and launched.

If, however, you use a Mac App Store-downloaded application to perform the process, we’ve seen higher success rates of a stable upgrade. If instead you tried to save time with either the InstallESD.dmg or NetInstall set methods mentioned earlier, a failure symptom occurred that, post-update, the disk would never complete its first boot(verbose boot was not conclusive as to reasons, either.) Moving the application bundle to another machine(volume license codes have, of course, been applied to the appropriate AppleID on the machines designated for upgrades,) hasn’t been as successful, although the recommended repackaging of the Install app, as Apple has referred to in certain documentation, wasn’t attempted this particular time. In some cases even breaking the software mirror didn’t allow the disk to complete an upgrade successfully. Another symptom before we could tell it was going to fail is the drop-down sheet warning of the loss of server functionality would immediately cause the entire window to lose focus while about to initiate the update. A radar has not been filed due to the fact that a supported(albeit semi time-intensive) method exists and as been more consistently successful.

Use Casper to collect Mac App Store IDs

Sunday, November 11th, 2012

An administrator may need to allow his users access to the Mac App Store but might prefer they download software only under sanctioned Apple IDs. Using an extension attribute in Casper, he can compile a list of all Apple IDs used on every Mac.

When a user enters an Apple ID to access the Mac App Store, it gets stored in his Home folder in:

~/Library/Preferences/com.apple.storeagent.plist

So long as he doesn’t sign out of the Mac App Store (he’ll probably just quit when he’s done) the ID remains in the file. Multiple users on a machine may use multiple Apple IDs because the credentials are stored for each user rather than once for the computer.

The following script gathers a list of unique Apple IDs from all user accounts and then returns the list to the Casper JSS as an extension attribute.

#!/bin/sh # Get a list of existing .plist files USERFOLDERS=$( find /Users/*/Library/Preferences \ -name com.apple.storeagent.plist ) # Make a list of Apple IDs found in the .plists for APLIST in $USERFOLDERS do IDLIST=$( echo "$IDLIST\n$( defaults read $APLIST AppleID )" ) done # Remove blank and duplicate lines IDLIST=$( echo "$IDLIST" | sed '/^$/d' | uniq ) # Return the result echo "<result>$IDLIST</result>"

To add the following script as an extension attribute in the JSS:

  1. Navigate to Settings tab –> Inventory Options –> Inventory Collection Preferences –> Extension Attributes tab –> Add Extension Attribute.
  2. Name this Extension Attribute “Mac App Store Apple IDs”.
  3. Set the Data Type to String.
  4. Set the Input Type to Script and paste in the script.
  5. Click the OK button and then Save the Extension Attribute.

Building a Mac and iOS App Store Software Update Service

Wednesday, November 9th, 2011

Let’s say you run a network with a large number of Mac OS X or iOS (or, more likely, both) devices. Software Update and the two App Stores (Mac App Store and iOS App Store) make keeping all those devices up-to-date a pretty straightforward process. They are a huge improvement compared with the rather old-fashioned practice of looking through applications, visiting the web site for each one and manually downloading updated versions. When updating two or more similar machines, of course, one only needed to download the updated version once, then copy it to each other machine. Better, but a process that when performed across a lot of machines requires a lot of work.

However, even though the App Store and Software Update Server in Mac OS X Server make things easier, there’s no simple way to download things once and distribute the downloaded files to multiple machines for items purchased on the App Store. When large updates come out (such as a new version of iOS), you’re essentially downloading huge amounts of data to each and every machine, and if machines are set to automatically download updates, you could even have a large number of them downloading simultaneously.

Of course you can run your own Software Update service in Mac OS X Server, but this requires that every client machine be configured to use the local server. This works well for machines under your control, but for all those people who bring in their own laptops this doesn’t help.

What’s worse is that there’s currently no way whatsoever to run a Software Update-like service for App Store purchases. Imagine if you have a lab of dozens or hundreds of Macs with Final Cut X or iPads (or iPhones, iPod Touches, whatever comes out next with iMovie or ). Any time there’s an update you’re potentially downloading over a gigabyte per machine in the case of Final Cut X or 70 megabytes or so in the case of iMovie. That can easily add up to a tremendous amount of traffic and the congestion, complaints and headaches which go with it..

What’s needed is an easy way to cache App Store downloads. While we’re at it, it would also be nice to transparently have machines use our own Software Update server. Let’s be even a little more ambitious and do this without needing Mac OS X Server. Aw, heck – let’s make it work on any reasonably Unix-like OS.

So how do we do this? The App Stores and Software Update services use http for fetching files. So what we need to do is to capture those http requests and either redirect them to a local store of Software Update files or locally cached App Store files.

Just as an aside, it’d be tremendously difficult to create a local store of App Store files if for no other reason than the fact that there are currently more than half a million applications. Add to this the rate at which updates become available and your machine would probably never be finished attempting to download all of the applications! Considering this, we’re looking at running Apache and squid on our Unix-like machine and doing a little redirection magic on whatever device does NAT or routes for us.

Note: There’s no reason that the same machine can’t do both NAT/routing and Apache/squid, although in most environments we are assuming that the machine would simply be a proxy for Mac or iOS-based devices. To make this example end-to-end though, we’ll run the router on the host.

Our example uses a Mac OS X (non-Server) machine running Leopard which is doing both NAT and running our Apache and squid software. We’re simply using the Internet Sharing service, the public network interface is en0 (which we don’t use anywhere) and the interface which will serve our iOS and Apple clients is en1 and has the address 10.0.2.1.

Everyone has their own favorite way of installing software on Unix-like OSes and a discussion about which is best and why would certainly be outside the scope of this article. In these examples we’re using NetBSD’s pkgsrc for no other reason than the fact that it will compile packages from source with a base directory which is easily configurable (feel free to use ports or some other automated tool according to what platform you are using). Get pkgsrc (usually via cvs; we’ll assume it’s put into /usr which can be as simple as:

cd /usr ; setenv CVSROOT :pserver:anoncvs@anoncvs.netbsd.org:/cvsroot ; cvs checkout -P pkgsrc

And then run /usr/pkgsrc/bootstrap/bootstrap like so:

cd /usr/pkgsrc/bootstrap/
./bootstrap --prefix /usr/local --pkgdbdir /usr/local/var/db/pkg --sysconfdir /usr/local/etc --varbase /usr/local/var --ignore-case-check

This puts all files into /usr/local including logs and configuration files, so keeping your system clean is simple and keeping track of the differences between built-in and pkgsrc software is easy. Next, install pkgsrc’s www/squid and www/apache (and net/wget if your Unix doesn’t already have it):

cd /usr/pkgsrc/www/squid
bmake update
cd /usr/pkgsrc/www/apache22
bmake update
cd /usr/pkgsrc/net/wget
bmake update

Note that on systems like Mac OS X which come with GNU make by default, that pkgsrc uses bmake; if you have BSD make already, just use make. Another note is that /usr/local/sbin is not in Mac OS X’s path by default, so add /usr/local/sbin to /etc/paths if you’re going to use it.

Now that the software is installed in consistent locations we can configure it. The squid.conf file only needs one line to be changed; everything else is added. Find the line which says:

http_port 3128

And change it to:

http_port 3128 intercept

Then add the following lines:

maximum_object_size_in_memory 4096 KB
cache_replacement_policy heap LFUDA
cache_dir ufs /usr/local/var/squid/cache 16384 16 256
maximum_object_size 2097152 KB
refresh_pattern -i .ipa$ 360 90% 10800 override-expire ignore-no-cache ignore-no-store ignore-private ignore-reload ignore-must-revalidate
refresh_pattern -i .pkg$ 360 90% 10080 override-expire ignore-no-cache ignore-no-store ignore-private ignore-reload ignore-must-revalidate
acl no_cache_local dstdomain 10.0.2.1
cache deny no_cache_local
redirect_program /usr/local/bin/rewrite.pl

These settings are chosen to cache large files up to 2 gigabytes in size in a 16 gig cache on disk and to ignore cache directives with regards to .pkg and .ipa files. Adjust to your own liking. Of course, replace 10.0.2.1 with the private IP of your machine. The cache deny with that address is used to make sure that redirected Software Update files are not cached in squid which would just take up room which better used for App Store files.

The URL rewriting script (create /usr/local/bin/rewrite.pl) just changes Apple Software Update URLs to point to our server:

#!/usr/bin/env perl
$|=1;
while (<>) {
s@http://swscan.apple.com@http://10.0.2.1/swscan.apple.com@;
s@http://swcdn.apple.com@http://10.0.2.1/swcdn.apple.com@;
s@http://swquery.apple.com@http://10.0.2.1/swquery.apple.com@;
print;
}

Next we configure Apache. The location you choose for the Software Update files can be anywhere (in our example, they’re on a FireWire attached drive mounted at /Volumes/sw_updates/) which needs to be allowed in the Apache configuration.

Add to /usr/local/etc/httpd/httpd.conf:

<Directory “/Volumes/sw_updates/”>
Options Indexes FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
</Directory>
<VirtualHost *:80>
ServerAdmin hostmaster@318.com
DocumentRoot “/Volumes/sw_updates”
ErrorLog “/usr/local/var/log/httpd/swupdate_error_log”
CustomLog “/usr/local/var/log/httpd/swupdate_access_log” common
</VirtualHost>

The log lines are purely optional. If you don’t add them, logs will still be written at /usr/local/var/log/httpd/access_log and error_log.

Next, we configure ipfw (in the case of Mac OS X or FreeBSD) to redirect all port 80 traffic transparently to our squid instance. If you’re using a different device for NAT/routing or different firewalling software such as ipfilter, see the examples listed below.

ipfw add 333 fwd 10.0.2.1,3128 tcp from any to any 80 recv en1

Note that on Snow Leopard and Lion you’ll need to make this change, too:

sysctl -w net.inet.ip.scopedroute=0

ipfilter would look like this for the same ipfw task from above (if you’re using Linux):

rdr en1 0.0.0.0/0 port 80 -> 10.0.2.1 port 3128 tcp

Again, the local private IP is 10.0.2.1 and the local private interface is en1; substitute your IP and interface.

Finally, we need to mirror all Apple Software Updates. A simple shell script can do this. Save this file somewhere (named mirror_swupdate.sh, for instance) and run it from cron now and then, perhaps once a night:

#!/bin/sh

 

location=$1 # This is the root of our Software Update tree
mkdir -p $1
cd $1

for index in index-leopard-snowleopard.merged-1.sucatalog index-leopard.merged-1.sucatalog index-lion-snowleopard-leopard.merged-1.sucatalog
do
wget --mirror http://swscan.apple.com/content/catalogs/others/$index

 

for swfile in `cat swscan.apple.com/content/catalogs/others/$index | grep "http://" | awk -F">" '{ print $2 }' | awk -F"<" '{ print $1 }'`
do
echo $swfile
wget --mirror "$swfile"
done
done

Invoke this with the top of the tree of your Software Update files as you’ve used in the Apache config, like so:

./mirror_swupdate.sh /Volumes/sw_updates

Expect this to run for a long time the first time you run this because you’ll be downloading around 60 gigabytes of updates. Every time it runs afterwards, though, files won’t be downloaded again unless they change (which they won’t; new updates will show up as new files).

Start squid and Apache, then tail your Apache log and run Software Update to test:

/usr/local/share/examples/rc.d/apache start
/usr/local/share/examples/rc.d/squid start
tail -f /usr/local/var/log/httpd/swupdate_access_log

At this point, you can redirect your software updates to the host. Updates for both the Mac App Store and iOS are also now cached. In the next article we’ll look at using some squid extensions to enable you to block applications from the App Stores or block updates in the event that an update is problematic.

Using Squidman as a Web Proxy for OS X

Thursday, October 27th, 2011

Squid is an open source package available at http://www.squid-cache.org that caches web files to a local server, increasing throughput for users and decreasing the amount of traffic on WAN connections. A Mac OS X software package named SquidMan, which includes Squid is available at http://web.me.com/adg/squidman/index.html. SquidMan makes installing and using Squid much easier, giving nice buttons to use for management rather than managing Squid using configuration files.

Once SquidMan is downloaded, copy the SquidMan application bundle to the /Applications directory. Then open it. At the Helper Tool Installation screen click on the Yes button.

At the Squid Missing screen click on the OK button to install squid itself.

The Preferences screen then opens. Click on the Clients tab and, if you would like to restrict access to only a set of IP addresses, define them (or use the net mask to define a range).

Click on the General tab. Here, provide the following information:

  • HTTP Port: The port number that the proxy will run on.
  • Visible hostname: The hostname of the server (e.g. proxy.318.com).
  • Cache size: The total amount of space used for the proxies cache.
  • Maximum object size: The maximum size of single cached files.
  • Rotate logs: The frequency with which log files are rotated (I usually use Manually here).
  • Start Squid on launch: Automatically start squid when SquidMan is launched, and delay start by x number of seconds.
  • Quid Squid on logout: Define whether logging out of the server also stops squid.
  • Show errors produced by Squid: Displays squid’s errors in SquidMan.

Click on the Parent and define a proxy server that this one will use (if there is one, otherwise it just uses the web to directly access files). This feature is only used if you are daisy chaining multiple squid servers.

Click on the Direct tab and enter any sites that should not be proxied. Internal staging environments are a great example of sites that should bypass proxy servers.

At the Template tab, enter any custom variables.

Squid is usually used to cache and speed up web access, so the default configuration file is optimized for small files. In order to cache larger files effectively, change the configuration to allow for larger files (up to 64 megabytes) and allow for more total disk storage of cached files (up to 8 gigabytes in our tests for a few specific projects, but much larger is fine). This usually depends on the total available disk space on the machine which will run squid.

These are some of the options which we updated for a specific project we’re working on in the squid.conf (Template):

http_port 3128 transparent (add transparent if using NAT to redirect http requests):
maximum_object_size_in_memory 65536 KB
cache_dir ufs /usr/local/var/squid/cache 8192 16 256
maximum_object_size 65536 KB

These days, we prefer to use squid running in NetBSD’s pkgsrc, although any method of installation (such as the squidman approach) should be acceptable.

Next, click on the SquidMan application which should have been running the whole time and click Start Squid.

The squid daemon then starts. Looking at the processes running on the host reveals that it is run as follows:

/usr/local/squid/sbin/squid -f /Users/admin/Library/Preferences/squid.conf

Client systems can then be configured to use the squid proxy, or PAC (Proxy auto-config) file can be configured to configure clients. Another option being transparent parodying:

rdr de0 0.0.0.0/0 port 80 -> (local Squid server) port 3128 tcp