Lightning Shutdown & Startup / Jul 8, 2021 / M Franz, P Gabor, D Brown (with thanks to Ralf Jansen, Taras Golota et al.) ✔✘
General rule 0=) Call Gary Gray, Chris Johnson or other members of the A-team ahead of time if it is not clear how to proceed!
General rule 1=) Always make sure you will know how to reconnect any cables you disconnect!
Case A Observatory in “warm” shutdown.
Assumes telescope is stowed :?:, TCS glue, and all other software needed to run the telescope is stopped :?:, amplifiers OFF :?:, Neslab OFF :?:, and hydrostatic bearing OFF :?:. GO IMMEDIATELY TO STEP 1.
Case B The telescope is on the sky observing, and a big boom arrives outta' nowhere → STEP 0.
STEP 0Just follow this checklist. Do not worry about the warm shutdown checklist. No need to exit applications.
Control room TCSglue: Stow the telescope
Control room TCSglue: Disable drives
Control room TCSglue: Close mirror cover
Control room TCSglue: Close dome slit
→ Go to STEP 1
STEP 1The following action items take place on the building's Level 2
Control room Out of the three new X-terms, pick the one where you can reach its own “surfer” desktop most readily (typically by closing various remote sessions). On the X-term's own desktop, open a terminal. In the terminal type either haltvatt or ./haltvatt and hit ENTER. The shutdown script will run, and it will power down every computer, including all four xterms (the three new ones and the older one), the only exception is vatttel. (It is no longer necessary to use the power button on vattdata to shut it down fully).
Note. The haltvatt script is not resident, and so cannot be used, on vattxt3 (the older xterm with the smaller monitor).
TCS room Turn vatttel off (big black toggle button)
TCS room Power down the M2 PI controller
TCS room Check :?: that the amplifiers are off
TCS room Turn off the GPS unit (flip the small on/off switch).
No need to power down the NTP server (small box on top of the GPS unit).
Server (Instrument) roomOpen the computer rack. Check that the servers are off by inspecting their power indicators (they ought to be amber).
Server (Instrument) roomPress and hold the power button on each of the HP UPSs (bottom of rack, under the vattdata server), until you hear the power go off (~5 sec).
Server (Instrument) roomClose the computer rack. (The doors are electrically bonded to the frames, and a closed rack provides more protection for people and for the computers.)
STEP 2The following action items take place on the building's Level 1
Lvl 1, loading dockTurn off the UPS unit in the loading dock; turn the key to the OFF position
Lvl 1, Telecom roomGo to panel A, and switch off the three bound-together 100 A circuit breakers marked Panel C in red.
Lvl 1, loading dockUnplug the UPS unit from the wall socket
Lvl 1, loading dockUnplug building supply lines from the back of the UPS unit
Lvl 1, loading dockHang the plugs up on hooks on the wall next to the UPS, using the lanyards provided on each plug.
Lvl 1, loading dockFor long shutdowns, disconnect the DC supplies with the DC breakers internal to the UPS units (Do not attempt unless instructed.)
STEP 3The following action items take place back on the building's Level 2
TCS roomUnplug AC power from west wall raceway quad: there are four sockets marked in red Left Raceway Quad / UPS power, and four sockets marked in white Right Raceway Quad / Non-UPS'd Conditioned. Disconnect all five power cords plugged into these sockets.
TCS room Disconnect the GPS antenna from the back of the black GPS receiver
Server (Instrument) roomAt the rear of the rack, unplug the two twistlock plugs feeding the rack power chords where they are plugged into the 220V wall outlets. They have red labels mentioning circuit breakers 7 and 9, and 11 and 13, resp.
STEP 4The following action items take place in various locations
Case ATelescope already in warm shutdown
DomeDouble check that the dome short is in
DomeClose dome vents
Done
Case BTelescope was on the sky
DomeFIRST! UNPLUG SLIT POWER and INSERT DOME SHORT
DomeInsert the stow pins
Dome⇓ Shut off everything on the telescope necessary to enter warm shutdown: ⇓
Finder 'scopes: power down the older one and close covers on both
Guider camera: power down (inline switch)
Close dome vents
BasementStop the hydrostatic bearing
Silo, lvl 2Turn Neslab OFF
Done
STARTUP Reverse the steps above
Use the vatt_lightning_startup_checklist
Note 1Once you have reconnected power cords, and engaged the circuit breakers, etc., it is a good idea to start by booting up vattdata. Let it do its thing for 5-10 mins, then look at its terminal: On the KVM (keyboard-video-mouse) console (Tips on how to use the unit.), highlight the entry labeled vattdata using the up/down arrow keys, and press Enter. When the RISC drives are mounted and the system is fully booted, the text stops scrolling on the screen and an ASCII list of 12 options appears. This indicates that all is well and no further action is required.
Note 2Do not attempt to boot any of the other servers until vattdata is running (wait time ~10 mins)
Note 3Booting up the servers in the computer racks in the Instrument Room. Slide out the KVM (keyboard-video-mouse) console. (Tips on how to use the unit.) Proceed to the other servers and power them all up. You may monitor the progress of this operation using the KVM. If you are still connected to vattdata, hit prn sc and then Alt+D to disconnect from the vattdata text interface. Green or red status indicators appear in the KVM's list of computers. Now you may also connect the KVM to the servers listed (scroll and Enter).
Note 4=) The GPS unit can take up to 10 minutes to startup fully (two blue lights: Ready and Power). 8-) Turn it on as immediately after having plugged in the raceway cables and closed all circuit breakers. Do not attempt to boot VATTtel unless the GPS unit is ready.
Note 5Boot VATTtel (follow these instructions: (Re)Booting VATTtel) only when vattdata is running. VATTtel uses its ancient network card to ftp a *.cmd file from vattdata, and it does not respond well to time pressure.
Note 6Coming out of lightning shutdown the guide box is likely to run into communication problems. The remedy is to run the INDImon script (its launcher on VATTcontrol's desktop is labeled Restart indidriver), selecting option (3) Filter/Guidebox driver. See What's New at VATT for more details.