MOON

	Lightning Shutdown & Startup / Jul 8, 2021 / M Franz, P Gabor, D Brown (with thanks to Ralf Jansen, Taras Golota et al.)	✔✘
General rule 0	Call Gary Gray, Chris Johnson or other members of the A-team ahead of time if it is not clear how to proceed!
General rule 1	Always make sure you will know how to reconnect any cables you disconnect!

Case A	Observatory in “warm” shutdown.
Case A	Assumes telescope is stowed , TCS glue, and all other software needed to run the telescope is stopped , amplifiers `OFF` , Neslab `OFF` , and hydrostatic bearing `OFF` . GO IMMEDIATELY TO STEP 1.
Case B	The telescope is on the sky observing, and a big boom arrives outta' nowhere → STEP 0.
STEP 0	Just follow this checklist. Do not worry about the warm shutdown checklist. No need to exit applications.
Control room	TCSglue: Stow the telescope
Control room	TCSglue: Disable drives
Control room	TCSglue: Close mirror cover
Control room	TCSglue: Close dome slit
	→ Go to STEP 1

STEP 1	The following action items take place on the building's Level 2
Control room	Out of the three new X-terms, pick the one where you can reach its own “surfer” desktop most readily (typically by closing various remote sessions). On the X-term's own desktop, open a terminal. In the terminal type either `haltvatt` or `./haltvatt` and hit `ENTER`. The shutdown script will run, and it will power down every computer, including all four xterms (the three new ones and the older one), the only exception is `vatttel`. (It is no longer necessary to use the power button on `vattdata` to shut it down fully).
Control room	Note. The `haltvatt` script is not resident, and so cannot be used, on vattxt3 (the older xterm with the smaller monitor).
TCS room	Turn `vatttel` off (big black toggle button)
TCS room	Power down the M2 PI controller
TCS room	Check that the amplifiers are off
TCS room	Turn off the GPS unit (flip the small on/off switch).
TCS room	No need to power down the NTP server (small box on top of the GPS unit).
Server (Instrument) room	Open the computer rack. Check that the servers are off by inspecting their power indicators (they ought to be amber).
Server (Instrument) room	Press and hold the power button on each of the HP UPSs (bottom of rack, under the `vattdata` server), until you hear the power go off (~5 sec).
Server (Instrument) room	Close the computer rack. (The doors are electrically bonded to the frames, and a closed rack provides more protection for people and for the computers.)

STEP 2	The following action items take place on the building's Level 1
Lvl 1, loading dock	Turn off the UPS unit in the loading dock; turn the key to the `OFF` position
Lvl 1, Telecom room	Go to panel A, and switch off the three bound-together 100 A circuit breakers marked `Panel C` in red.
Lvl 1, loading dock	Unplug the UPS unit from the wall socket
Lvl 1, loading dock	Unplug building supply lines from the back of the UPS unit
Lvl 1, loading dock	Hang the plugs up on hooks on the wall next to the UPS, using the lanyards provided on each plug.
Lvl 1, loading dock	For long shutdowns, disconnect the DC supplies with the DC breakers internal to the UPS units (Do not attempt unless instructed.)

STEP 3	The following action items take place back on the building's Level 2
TCS room	Unplug AC power from west wall raceway quad: there are four sockets marked in red `Left Raceway Quad / UPS power`, and four sockets marked in white `Right Raceway Quad / Non-UPS'd Conditioned`. Disconnect all five power cords plugged into these sockets.
TCS room	Disconnect the GPS antenna from the back of the black GPS receiver
Server (Instrument) room	At the rear of the rack, unplug the two twistlock plugs feeding the rack power chords where they are plugged into the 220V wall outlets. They have red labels mentioning circuit breakers `7 and 9`, and `11 and 13`, resp.

STEP 4	The following action items take place in various locations
Case A	Telescope already in warm shutdown
Dome	Double check that the dome short is in
Dome	Close dome vents
	Done

Case B	Telescope was on the sky
Dome	FIRST! UNPLUG SLIT POWER and INSERT DOME SHORT
Dome	Insert the stow pins
Dome	⇓ Shut off everything on the telescope necessary to enter warm shutdown: ⇓
	Finder 'scopes: power down the older one and close covers on both
	Guider camera: power down (inline switch)
	Close dome vents
Basement	Stop the hydrostatic bearing
Silo, lvl 2	Turn Neslab `OFF`
	Done

STARTUP	Reverse the steps above
STARTUP	Use the vatt_lightning_startup_checklist
Note 1	Once you have reconnected power cords, and engaged the circuit breakers, etc., it is a good idea to start by booting up `vattdata`. Let it do its thing for 5-10 mins, then look at its terminal: On the KVM (keyboard-video-mouse) console (Tips on how to use the unit.), highlight the entry labeled `vattdata` using the up/down arrow keys, and press `Enter`. When the RISC drives are mounted and the system is fully booted, the text stops scrolling on the screen and an ASCII list of 12 options appears. This indicates that all is well and no further action is required.
Note 2	Do not attempt to boot any of the other servers until `vattdata` is running (wait time ~10 mins)
Note 3	Booting up the servers in the computer racks in the Instrument Room. Slide out the KVM (keyboard-video-mouse) console. (Tips on how to use the unit.) Proceed to the other servers and power them all up. You may monitor the progress of this operation using the KVM. If you are still connected to `vattdata`, hit `prn sc` and then `Alt+D` to disconnect from the `vattdata` text interface. Green or red status indicators appear in the KVM's list of computers. Now you may also connect the KVM to the servers listed (scroll and `Enter`).
Note 4	The GPS unit can take up to 10 minutes to startup fully (two blue lights: `Ready` and `Power`). Turn it on as immediately after having plugged in the raceway cables and closed all circuit breakers. Do not attempt to boot `VATTtel` unless the GPS unit is ready.
Note 5	Boot `VATTtel` (follow these instructions: (Re)Booting VATTtel) only when `vattdata` is running. `VATTtel` uses its ancient network card to ftp a *.cmd file from `vattdata`, and it does not respond well to time pressure.
Note 6	Coming out of lightning shutdown the guide box is likely to run into communication problems. The remedy is to run the `INDImon` script (its launcher on `VATTcontrol`'s desktop is labeled `Restart indidriver`), selecting option `(3) Filter/Guidebox driver`. See What's New at VATT for more details.

MOON

User Tools

Site Tools

Page Tools