Troubleshooting: RCC

The RCC and its subsystems

Fig. 1: RCC control panel top middle section with manual control switch key

Fig. 2: RCC control panel bottom-left part

Fig. 3: Emergency cable switch at the west side of the north pillar of the RCC telescope

Solution 1: quit ccdsh and run crio-reset from the terminal on m6. Wait for the script to finish and try to rotate the dome again. Do not this without notifying the support astronomer.

Solution 2: Consult the rcc status command in ccdsh. If CRIO FPGA status is on normal, then check the frequency switcher in the dome. It is in a gray box with a high voltage sign on it, right next to the manual dome rotating wheel. Several standard cabinet keys can be found in the dome, check the control desk or the top of telescope control cabinet next to the door. If E8 can be seen on the frequency switcher, the malfunction is probably caused by power failure. Exit from ccdsh and press the 0 button on the frequency switcher to reset it. If it does not work, turn off and on the dome control several times. The switches are in a built-in cabinet right behind the control deck. Note that the switch labeled "Csúszógyűrű" has to be turned off first, and turned on second. If the "E8" message does not turn into "Stop" after a few tries, notify the head of technical department. Also note that this problem can be caused by a partial power failure, which cannot be noticed otherwise (i.e. in very rare cases, it might happen during observations, while the telescope continues to operate normally). After a long power failure, it may be necessary to start the CRIO (just start the ccdsh) before checking the frequency switcher, as it can be turned off completely.

Solution 1: Try to switch to manual dome positioning by rcc dome manual and rotate the dome to its base position: set dome azimuth=156.3.

Solution 2: if it does not work, try crio-reset, which turns on and off the main power. This is equivalent to restarting the CRIO by manually turning off and on its main power (two red switches on the gray electric cabinet in the dome). Do not do this without the support astronomer!

Solution: Check the CRIO error code with criorcc -l. One of the emergency switches could be pushed in the dome, check the red buttons. If this does not help, restart the crio with crio-reset. This also powers down the OCELOT EMCCD and m16, so do a proper shutdown beforehand (check that the camera is warmed up, stop the ixon-server, shutdown m16, see Usage of OCELOT on the RCC.) Do this only, if you are absolutely sure what you are doing!

Solution: you need to move it out of the forbidden zone manually, as follows:
- 1. Turn the key switch on the control panel to 'manual / kézi üzem' position (Fig. 1).
- 2. The controls on the bottom left section of the control panel (Fig. 2) should now be lit.
  - 2a. If not, check the emergency switches.
  - 2b. The cable limitation switches at the bottom of the north pillar should be set to 'I' (Fig. 3).
  - 2c. If it is in '0' position, check collision of the telescope to the pillar (very unlikely event). In case of collision, do not touch anything, call help immediately.
  - 2d. If no collision, switch the blue lever back to 'I' position. Now the bottom-left controls of the RCC control panel should really be lit.
- 3. The blue control lamp in the top-left of the central key panel ('Vezérlés be - ki') is lit, indicating the error.
- 4. First acknowledge the error by pushing the bottom-left button ('Hiba nyugta').
- 5. Now pressing the 'Vezérlés be - ki' button together with the alpha/delta +/- buttons, the telescope can be moved slowly.
- 6. Take into account that if the telescope is in west position (telescope towards the control panel, counter weight on the other side), the meaning of the +/- buttons is reversed.
- 7. When finished, do not forget to turn the key switch (Fig. 1, 'Kézi üzem') back to remote control position ('0'). Otherwise you are going to have another round to the dome in the night.

Problem: The focuser reached its end point at 85.5. It cannot be moved back with set focus, and CRIO restart cannot help either. Solution: One has to move the focuser manually, inside the telescope. It's not trivial, and should not be attempted alone. It's best to leave it to the technical staff. Documentation of an issue on 2020.01.16. can be found here (in Hungarian): Media:rcc_fokusz_javitas.pdf

Problem: The RCC webcam image is blank, the kiosk shows a big yellow "!", and rcc-test shows an error as well. Solution: One has to unplug the POE cable with "Kupolatér kamera" sign inside the RCC dome from the gray box to restart the camera. After a minute the camera image should be back.

CRIO Troubleshooting: for the meaning of the error codes given by criorcc -l, consult this table: CRIORCC.

Problem: rcc-test returns several error messages, e.g. after a power failure. Solution: Check if all virtual interfaces on m5 showing up: /sbin/ifconfig. If not do ifconfig eth0:1 10.10.174.135 netmask 255.255.0.0 and ifconfig eth0:2 172.31.171.135 netmask 255.255.255.0 (taken from /etc/rc.local on m5).

Problem: rcc-test returns: [ Err! ] Checking MAXNet configuration... failed. , and : acectrl: error: connection to 172.31.171.104:23 timed out (retries: 3). Solution: Restart the ACE spectrograph main controller box. It is on the side of the telescope, a big grey box. Just unplug it, and plug it back a few seconds later. A CRIO and ccdsh restart maybe needed after this.

Problem: rcc-autoguide.qsi returns: 0x00007531: Cannot Start Exposure then hangs. Solution: the camera is overheated (it's probably summer), and the CCD should be cooled less. Try raising the temperature setpoint by rcc-autoguide.qsi -T {something-not-so-cold}.