Dec 2022: 1ms1 has been replaced with lesedis2 (username & password unchanged) and lesedi-nuc has been replaced with ioserver1. The autoguider functionality is now intrinsically linked to Mookodi, so the startup procedure involves multiple PCs.
'''Anyone observing in the dome will use lesedi-nuc to operate the telescope and instrument via the web GUIs. SiTech and the autoguider backend are running on ioserver1, so to safeguard them from interference, observers should only use ioserver1 to view the weather page.'''
== SiTech ==
The SiTech software runs Lesedi the telescope (except mirror covers) on lesedis2 (="lesedi server 2", i.e. Lesedi's TCS PC). There is a main GUI for the telescope (SiTechExe.exe), a communications GUI (ServoSCommunicator.exe), a GUI for each rotator (SiTechRotatorTCP.exe and "SiTechRotatorTCP.exe left"), one for the secondary and tertiary mirrors (SiTechFocuserTCP.exe), one for the dome (DomeControlTCP.exe) and a GUI that communicates with each subsystem's software (ObservatoryControlSAAO.exe). All of these GUIs must be running for the telescope to function fully, and only one instance of each can be run at a time, else problems will arise.
'''Leave the SiTech software running at all times during normal operations. It is vital to ensure that no parameter is changed in any of these GUIs.'''
To prevent accidental interference, and to provide a streamlined user experience, a [[Lesedi#TCS Overview|browser-based TCS]] has been developed for the user. It is intended that users will interact with the browser and not with SiTech. This page details engineering tasks and should not be available to general users.
'''Note that the SiTech software cannot control the mirror covers since we put them under PLC control. These can only be operated from the browser TCS.''' In case of software failure, they can be closed by pressing the button on each cover motor, being careful to take the weight of the cover (which will otherwise fall under gravity).
== Starting SiTech software ==
sudo systemctl restart lesedi
sudo systemctl restart lesedi-web
exit ''(to leave the ssh session)''
exit ''(to close terminal #2)''
8. If you ever need to restart either ServoSCommunicator or SiTechExe, you will first need to close all the other subsystem GUIs, and reopen them in the order above (or just run the start-up script). Leave terminal #1 open: unless closed for engineering, SiTech should be running at all times.
9. If working in the dome, bundle all the GUIs onto the left-hand monitor (it makes it easier to find them if someone needs to VNC from a smaller screen) then minimise them - they won't need to be accessed during the night unless something goes wrong. If working in Cape Town, you can close the VNC session with ioserver1 by clicking the right-hand "Disconnect" button on the Remmina tool bar (or by pressing the X in the Teamviewer menu).
== Starting the LCU ==
The Local Control Unit (LCU) should be running during normal operations. In terminal #2 (lesedis2) check its status and start it if necessary:
sudo systemctl status lesedi-lcu
sudo systemctl start lesedi-lcu
If you need to stop or restart the LCU for any reason, the commands are:
sudo systemctl stop lesedi-lcu
The LCU logs to /var/log/lesedi-lcu.log - if you need to check the logs, in a terminal use the command:
less /var/log/lesedi-lcu.log
== Starting Autoguider software ==
'''These steps must be followed if SiTech has been restarted,''' hence the numbering continues from the section above. There are now autoguider-related programmes on both lesedis2 and the Mookodi PC, so the process involves both PCs.
10. From ioserver1, open another in the terminal tab, log in connected to the Mookodi PC and lesedis2 (terminal #2), run following commands:
ssh mookodiobserver@mookodilesedis2.suth.saao.ac.za
systemctl stop lodestar_source_extr.service
systemctl start lodestar_source_extr.service
11. Open another terminal tab (terminal #3) on ioserver1 and log in to lesedis2. If the autoguider backend software was open when SiTech shutdown then it has probably hung and won't respond to the red EXIT button. We need to kill it:
ssh -Y observer@lesedis2.suth.saao.ac.za
ps aux | grep readPLC
kill -9 xxxxx ''(where xxxxx is the process ID reported from the ps aux task - kill all such processes)''
12. Then start the autoguider backend software(still in terminal #3):
sudo xauth merge ~/.Xauthority
sudo ./main_prog -depth 24
13. Open another terminal tab (terminal #4) on ioserver1 and start the guider web server:
ssh observer@lesedis2.suth.saao.ac.za
gunicorn -k gevent 'app:create_app()' -b 0.0.0.0:5001
14. Go back to the terminal tab If this step fails (symptom "Socket connected to is False"), close the Mookodi PC autoguider GUI and start the autoguider comms/catalogue services:repeat steps 12 and 13.
14. Open a final terminal tab (terminal #5) ssh to mookodi and start the autoguider comms/catalogue services:
ssh mookodi@mookodi.suth.saao.ac.za
systemctl stop gaia_list.service
systemctl start gaia_list.service
sudo systemctl stop Mookodi_source_extract
sudo systemctl start Mookodi_source_extract
15. Test the browsers:
* Point a browser to http://lesedis2.suth.saao.ac.za:5000 to load the TCS. If it loads ok, '''close the browser''' so the observer can log in from elsewhere. If it fails, repeat step 7. If it still fails, start again from step 4.
* Point a browser to 10.2.2.32:5001 to load the autoguider. If it loads ok, reset and initialise the XY slides, then '''close the browser''' so the observer can log in from elsewhere - '''there must never be more than one instance of the autoguider open'''. If it fails, start again from step 10.
16. Leave terminal tabs #1 - #5 open to make the next restart quicker.
== Stopping SiTech Software ==
10. Close the Script, PointXP6 and Astrometry windows, the SiTechCamera GUI and CTRL-C in the terminal to stop running the astrometry programme. If going back to normal operations, stop all SiTech software and reopen from ioserver1, then do a "sudo systemctl restart lesedi" and "sudo systemctl restart lesedi-web" from lesedis2 and restart the autoguider software.
== "SiTech Ready" config: illustrated guide ==
This section illustrates how the SiTech GUIs will look if there are no issues, and points out potential problems to look out for.
<br clear=all>
[[File:SiTechExe.png|800px|left]]
<br clear=all>
[[File:rightrotator.png|800px|left]]
<br clear=all>
[[File:rightrotatorconfig.png|800px|left]]
<br clear=all>
[[File:focuser.png|800px|left]]
<br clear=all>
[[File:dome.png|800px|left]]
<br clear=all>
== Troubleshooting ==
The pink warning box will disappear from the relevant GUI if the problem is resolved. If the problem persists, see point 3 below.
3. If clicking on a "Bad comms" warning button does not fix the problem, the SiTech software will need to be stopped and restarted. It is vital that all the SiTech GUIs are closed, and that they are reopened in the correct order, so follow [[#Starting SiTech software| this procedure.]] then restart all the other Lesedi/Mookodi services.
4. The most common issue is '''loss of dome commsDome and mirror covers close unexpectedly, telescope and rotators park.'''. The symptom will A full shutdown like this can only be that triggered by the dome won't move to a new target, or that it has closed without warning (the telescope will not park, mirror covers will not close: those actions suggest an LCU shutdown). A red warning button will appear labelled "Bad Dome Comms" in Perhaps the Dome GUI - clicking it should solve the problem. If it does, go weather is bad (or mistakenly thought to the browser TCS Advanced tabbe bad), and in or the central panel click "TAKE CONTROL", then turn dome "Tracking" ON LCU has crashed and OPEN if the dome had closed without good reason everything to keep it safe in the middle of observingits absence. If comms are not restored this way, then press Shutdown on the browser TCS and follow the full instructions on [[#Starting SiTech softwarethe LCU| starting SiTech softwareRestart the LCU]] to restart. Then reload the browser TCS and run the a telescope startup procedure.
5. '''Focus Dome closes unexpectedly and no other subsystem is unresponsiveaffected'''(mirror covers stay open, telescope keeps tracking, etc.). This is typically a comms loss to Check the SiTech focuser dome GUI and doesn't respond to the instructions for a "watchdog timeout" in point 2the small font list of statuses. Follow If this is listed, power cycle the full instructions on dome muscle box, then close the autoguider software, close the SiTech software, and [[#Starting SiTech software| starting restart SiTech software]] to restartand all subsequent services.
6. '''Loss of dome comms'''. The symptom will be that the dome won't move to a new target, or that it has closed without warning (without parking the telescope or closing the mirror covers: those actions suggest an LCU shutdown). A red warning button will appear labelled "Bad Dome Comms" in the Dome GUI - clicking it should solve the problem. If it does, go to the browser TCS Advanced tab, and in the central panel click "TAKE CONTROL". If the dome had closed without good reason in the middle of observing, CLOSE the mirror covers, then turn dome "Tracking" ON, OPEN the dome then the mirror covers. If comms are not restored this way, then press Shutdown on the browser TCS and follow the full instructions on [[#Starting SiTech software| starting SiTech software]] to restart. Then reload the browser TCS and [[#Starting SiTech software| run the telescope startup procedure]]. 7. '''Focus is unresponsive'''. This is typically a comms loss to the SiTech focuser GUI and doesn't respond to the instructions in point 2. Follow the full instructions on [[#Starting SiTech software| starting SiTech software]] to restart. 8. '''No rotator comms'''. First try shutting down and restarting all SiTech. If that doesn't work, log into lesedis2 and do the following:
ssh observer@lesedis2.suth.saao.ac.za
ls -l /dev/ttyU*
The serial number of the rotator controller is A603G14W, so this confirms that the rotators are communicating on port ttyUSB1. If the above command gives you a different serial number, edit the command to try each port from ttyUSB0 to ttyUSB5 until you find A603G14W. Now go to the ServoSCommunicator GUI and select the Rotators tab. In the dropdown menu, select the appropriate port - in this case ttyUSB1. Go to each of the Left Rotator GUI and the Right Rotator GUI and click on the pink warning complaining of no comms. If the warning doesn't clear, shutdown and restart all SiTech. ServoSCommunicator should wake up with the ttyUSB1 port selected, and communications to the rotator GUIs should be restored.
79. '''The telescope won't slew''' - in robotic mode this is likely discovered as '''telescope isn't in the park position in the morning'''. First check that the telescope is "unparked " in the software and -- if trying to point -- that the target is within the visibility window(the TCS web GUI has a visibility calculator). If those things are fine, the telescope may have triggered the hardware azimuth hardware limit switch. This sometimes happens if the telescope is at azimuth ~165163-175 degrees. Go to the SiTechExe GUI (the long, thin one) and at the top of the "Scope " tab, check if any of the arrows are red or orange. If they are, click the "STOP" button toward the bottom of the GUI, then click the opposite arrow a few times to move nudge the telescope out of the limit. If the motors are in "blinky", you may need to click the button at the bottom to switch the motors to automatic. 10. '''Telescope won't slew: motors are in "blinky" mode and won't switch to auto.''' There is likely an error on the alt and/or az controller board which will need to be reset. Find the SiTechExe GUI (the long, thin one) and try pressing the button at the very bottom to switch the motors to Auto. If it does nothing, select the "Features" tab on the same GUI. Press the "Controller Stuff" button, which will open the "Controller Information Window". There are two columns in this GUI, one for alt, one for az. There is a text box about two-thirds of the way down each column, which will either say "No Errors", or list some errors. If there is an error in the box, click the "Reset Errors" button directly underneath that text box. '''Be very careful not to click any other button.''' If that has fixed the problem, the motors will now be in Auto mode - the button in the top left corner will say "To Manual (Blinky)", and the long, thin SiTechExe button will say "Motors Automatic" at the bottom. Close the Controller Information Window and resume operations. This almost always solves the problem, but if it persists, go to 10a).
8. '''Telescope won10a) If point 10 didn't slew: motors are in "blinky" mode and won't switch to auto.''' There is likely an error on the alt and/or az controller board which will need to be reset. Find the SiTechExe GUI (the long, thin one) and try pressing the button at the very bottom to switch get the telescope motors back to Auto. If it does nothingmode, select the "Features" tab on the same GUI. Press the "Controller Stuff" buttonclose all SiTech software, which will open power cycle the "Controller Information Window". There are two columns raspberry Pi in this GUI, one for alt, one for az. There is a text the altaz controller box about two-thirds of under the way down each columnNorth pier, which will either say "No Errors", or list some errorsthen restart SiTech. If there is an error in the box, click the "Reset Errors" button directly underneath that text box. motors aren't in Auto and can''Be very careful not t be switched to click any other button.''' If that has fixed the problem, the motors will now be in Auto mode - the button in the top left corner will say "To Manual (Blinky)"on startup, and the long, thin SiTechExe button will say "Motors Automatic" at the bottomcheck for errors as per point 10 above. Still got trouble? Close the Controller Information Window and resume operations. This almost always solves the problem, but if it persists, go Go to 8a)10b.
8a10b) If point 8 didn10 & 10a) don't get solve the telescope motors back to Auto modeproblem, close all SiTech software, power cycle the raspberry Pi in and reset the SiTech altaz controller box under the North pier, then restart the SiTechsoftware. If the motors aren't in Auto and can't be switched to Auto on startup, check for errors as per point 8 above. Still got trouble? Go to 8b.
8b) If 8 & 8a) don11. 't solve ''North-South on the problem, close all SiTech software and reset image is not aligned with columns on the instrument detector.''' On the SiTech altaz controller under rotator GUI, on the North pier"Config Page 1" tab, then restart there are two boxes that need to be ticked to keep North aligned with either "up" or "down" on the SiTech softwaredetector. If the motors aren't in Auto and can't be switched to Auto Tick these two boxes on startupthe SiTech Right Rotator Control GUI for Mookodi, check or the Left Rotator for errors as per point 8 above.Sibonise:* "Track PA (Parallactic Angle) (Keep North Up)"* "If PA is out of range, Track PA+-180"