Introduction

In emergency or critical situations it is sometime necessary to quickly power down an entire network of servers. This could be due to a natural disaster, planned maintenance, or a variety of unforeseen circumstances. This document attempts to scope a quick yet safe method of powering down all servers and network devices, then walking through powering everything back up.

Part I – Shutting Down

Methodology and Theory

This section will explain the general principles of an emergency shutdown. The general procedure is as follows:

1) Notify everyone of the shutdown. Have everyone (ideally) turn off his or her computer.
2) Have all server logins and passwords handy.
3) The shutdown sequence is: Servers, then RAID arrays and attached storage,
then network devices.
4) Make sure every device is completely powered off, and not in stand-by mode.

The most essential part of this is making sure you turn off the servers BEFORE you turn off any storage devices. Most operating systems store often-used files and settings in memory, so shutting servers down first allows the OS to write/update any files beforehand. Once the servers are shutdown and off, generally speaking, it is fine to power down storage.

Shutting Down a Windows-Based Server

1) If you are on a locked screen, and at the desktop, press CTRL-ALT-DEL
at the same time.
2) Enter the administrator login and password. This should put you on a desktop screen.
3) Click on the Start button, then click on “Shut Down”
4) Make sure the option for “Shut down” is selected. On some Windows servers,
you may have to enter a reason before you’re able to press ok.
Type “emergency shutdown” if there is a box asking for a reason. Press OK.
5) Depending on your sever setup, there might be several applications that need
to be shut down. Once Windows tells every application that the machine is turning off,
the applications may ask for permission to quit. Just click “Yes” or “Ok” if any
boxes appear.
6) WATCH AND MAKE SURE THE SERVER SHUTS DOWN. Servers can take
a long time to shutdown/start up due to all the programs and services they need to
keep track of, so this may take up to 3-5 minutes.
7) The server should power down completely. Look at the server and make sure there are
no lights on (which would indicate that the server is in stand-by mode). The surest way
to verify power-off is to check the fan in the back of the box. If the fans are off,
the server is off.
8) If the system has been shut down, but there are still lights on it, you can force
the machine to turn off by pressing the power button, and holding it down
for 10 seconds. This will force the system to lose power. Every PC created after 1997
has this ability.
9) Repeat this procedure for every Windows Server you have.

Shutting down a Mac OS X Server

1) If you are seeing a locked screen, or a screen saver, press a key on the keyboard.
This will wake the system.
2) In the login box, enter the administrator login and password. It should take you to
the desktop.
3) Click on the Apple symbol in the top left corner of the screen.
4) Click on Shut Down.
5) The server will ask if you are sure you want to do this. Click “yes”
6) Depending on your sever setup, there might be several applications that need to
be shut down. Once OSX tells every application that the machine is turning off,
the applications may ask for permission to quit. Just click “Yes” or “Ok” if
any boxes appear.
7) WATCH AND MAKE SURE THE SERVER SHUTS DOWN. Servers can take
a long time to shutdown/start up due to all the programs and services they need
to keep track of, so this may take up to 3-5 minutes.
8) If the system has been shut down, but there are still lights on it, you can force
the machine to turn off by pressing the power button, and holding it down for 10 seconds.
This will force the system to lose power. Every computer created after 1997 has
this ability.
9) Repeat this procedure for every Apple OSX Server you have.

Shutting Down a Linux Server

Note: In order to do this, you will need both the administrator login/password, AND an additional password known as the “root” password. The root password is the King-Of-All-Kings password. It allows you to do ANYTHING to the server, without restriction. Be VERY careful what you type when you are using “root” mode.

1) If you are not at a command line prompt, you are probably looking at a “login:” prompt.
Enter the admin login and password.
2) You will be looking at command line prompt now. Type “su” and press enter.
This means “I want to be a SuperUser”
3) It will ask you for the root password. Enter it here. NOTE: When you type this
password, it will not display it on the screen. This is for security reasons (so nobody
can peek over your shoulder)
4) If you typed it correctly, the prompt should change to a # at the end of it.
5) Type “shutdown now” and press enter.
6) The screen should automatically fill with information about the server shutting down,
and the various services that are being turned off.
7) The machine should shut down completely.

Shutting Down Arrays and Storage

Once the servers are down, most of the hard work is done. Look at the arrays, or network storage. There should be no disk activity on them (ie: no lights should be blinking) If there are posted procedures for a specific method of shutdown, please follow them. If not, once the servers are shut down, as a general rule, it is ok to find the power switch and turn them off.

Shutting Down Routers and Network Devices

Most routers, firewalls, and network devices have no moving parts, and are designed to handle a power outage gracefully. In an emergency, it would be ok to just quickly unplug them after the servers, storage, and workstations are shut down. This is not the best policy, nor is it inclusive for every network device out there. But it is acceptable in a critical situation, where time is of the essence.

Part II – Starting up the network.

Methodology and Principles.

The only concern when starting everything up is the sequence. When shutting down, the order was workstations, servers, storage, then network devices. When starting everything back up, the process is REVERSED.

Plug all routers, firewalls and switches back in. Wait at last 2 minutes for everything to come back online, and for all the lights to start blinking again.

Now power the RAID arrays and storage on. Let the drives spin up and let the systems prepare themselves for use. Wait at least 3-5 minutes for this, even if it looks like everything is done.

One the storage is up and running, it time to power up each server…ONE BY ONE. It is critical to not only turn each server on, but to WATCH each one boot up and get to the login screen. Each machine may take as long as 5 minutes to do this, due to the programs and services being loaded. If any messages appear, write them down and call Three18 (310-581-9500) immediately.

After everything in the server room is up and running, have the users turn on their workstations and attempt to use their usual programs (ie: email, web, filemaker, etc) Make note of any errors or unusual messages. Be sure to write them all down, and call Three18. For the most part, essential services should be good to go (email, internet)

And, as always, call Three18 if you have any questions or concerns. We are available 24 hours a day for emergency situations. 310-581-9500.