Difference between revisions of "SiTech"

Revision as of 10:26, 12 September 2022

The SiTech software runs Lesedi on 1ms1 (="1-metre server 1", i.e. Lesedi's TCS PC). 1ms2 is Lesedi's backup machine. There is a main GUI for the telescope (SiTechExe.exe), a communications GUI (ServoSCommunicator.exe), a GUI for each rotator (SiTechRotatorTCP.exe and "SiTechRotatorTCP.exe left"), one for the secondary and tertiary mirrors (SiTechFocuserTCP.exe), one for the dome (DomeControlTCP.exe) and a GUI that communicates with each subsystem's software (ObservatoryControlSAAO.exe). All of these GUIs must be running for the telescope to function fully, and only one instance of each can be run at a time, else problems will arise.

Leave the SiTech software running at all times during normal operations. It is vital to ensure that no parameter is changed in any of these GUIs.

To prevent accidental interference, and to provide a streamlined user experience, a browser-based TCS has been developed for the user. It is intended that users will interact with the browser and not with SiTech. This page details engineering tasks and should not be available to general users.

Note that the SiTech software cannot control the mirror covers since we put them under PLC control. These must be operated from the browser TCS.

Starting SiTech software

The software should be opened on lesedi-nuc, by connecting by ssh to 1ms1 and running the software using the mono libraries. If in Sutherland, you can do this from the lesedi-nuc PC in the warm room, else if working remotely from Cape Town, VNC to lesedi-nuc:

1. Open Remmina Remote Desktop Client, select "VNC" from the dropdown menu and double click on Lesedi_Nuc from the list of servers. Enter the password and the lesedi-nuc desktop will be displayed on the screen.

2. Look for an existing terminal on lesedi-nuc, else open a new one, and log in to 1ms1:

         ssh -Y observer@1ms1.suth.saao.ac.za  (you shouldn't need the full path from inside the network, just observer@1ms1) 

3. Check for existing instances of the software already running:

         ps aux | grep SiTech
         ps aux | grep Servo
         ps aux | grep Dome
         ps aux | grep Obs

4. If any existing jobs are listed -- and you are sure they are not in use by someone working elsewhere -- kill them all. Note that there will be two jobs each to kill for SiTechExe and ServoSCommunicator:

         sudo kill -9 xxxxx    (where xxxxx is the relevant process ID)

5.(a) All the SiTech software can be opened using a startup script in the home directory, then leave the terminal running:

         cd
         sudo ./startup.sh

5.(b) or opened individually in the following order, putting each task in the background (the first two must use "sudo", and don't respond well to "&"):

         cd bin 
         sudo mono SiTechExe.exe  (always open this one first, and wait until the GUI has loaded and the message window has closed)
         CTRL-Z
         bg
         sudo mono ServoSCommunicator.exe   (always open this one second, and wait for the GUI to fully populate and values start updating)
         CTRL-Z
         bg
         mono DomeControlTCP.exe &
         mono SiTechRotatorTCP.exe left &
         mono SiTechRotatorTCP.exe &
         mono SiTechFocuserTCP.exe &
         mono ObservatoryControlSAAO.exe &  (always open this one last)

6. Important: every time the SiTech software is restarted, you also need to do the following:

• Navigate to the Dome tab of ServoSCommunicator and tick the box called "Read PLC Comms" (this logs dome comms to help us track down an ongoing & evasive bug).
• In a new terminal, restart Lesedi's web server and backend:

         ssh observer@1ms1.suth.saao.ac.za
         sudo systemctl restart lesedi
         sudo systemctl restart lesedi-web
         exit (to leave the ssh session)
         exit (to close the terminal)

• Restart the autoguider backend software. If it's open it has probably hung and won't respond to the red EXIT button, so kill it and then restart it.

        ssh observer@1ms1.suth.saao.ac.za
        ps aux | grep readPLC
        kill -9 xxxxx (where xxxxx is the process ID reported from the ps aux task - kill all such processes)
        cd /home/observer/src/xyslides2ports
        sudo ./main_prog -depth 24

7. If you ever need to restart either ServoSCommunicator or SiTechExe, you will first need to close all the other subsystem GUIs, and reopen them in the order above.

8. If working in the dome, bundle all the GUIs onto the left-hand monitor (it makes it easier to find them if someone needs to VNC from a smaller screen) then minimise them - they won't need to be accessed during the night unless something goes wrong. If working in Cape Town, you can close the VNC session with lesedi-nuc by clicking the right-hand "Disconnect" button on the Remmina tool bar.

Stopping SiTech Software

If one of the SiTech GUIs hangs or reports a comms error, the easiest fix is to close all GUIs in this order, using the x in the top left-hand corner of each:

1. ObservatoryControlSAAO.exe;
2. DomeControlTCP.exe, both left and right SiTechRotatorTCP.exe and SiTechFocuserTCP.exe in any order;
3. ServoSCommunicator.exe;
4. SiTechExe.exe.

Then follow the instructions on starting SiTech software to restart.

Making a pointing model using SHOC

1. Open Remmina and VNC into lesedi-nuc.

2. Make sure all the usual SiTech software is running and ready the telescope for observing, then in /home/observer/bin open the programme that connects to SHOC and run astrometry:

    sudo mono SiTechCamera.exe
    sudo ./RunAstrometry

3. The SHOC browser(s) must be closed, but there's no need to turn the camera off. In the SiTechCamera GUI, select the Latest Image tab, enter an exposure time and click "TakeImage". Check that the image loads (the image scaling can't be adjusted, so if you can't see any stars, make sure you're pointing at stars and tracking, the telescope is focused and that the exposure time is sensible for a 1m). Watch the output in the terminal you used to start SiTechCamera for status info.

4. Make sure that SHOC's rotator is tracking, and on "Config Page 1" of the rotator GUI the boxes are ticked beside "Track PA" and "If PA is out of range, Track PA+-180". On the "Tertiary mirror" tab of the SiTech TCP Focuser GUI, ensure that the tertiary mirror is at the correct angle ("Current Position" = "Final Position (Right Fork)". Make sure the dome is set to "follow scope".

5. In SiTechExe GUI:
On Scope tab, click SkyView. In the window that opens, turn off most things in the Objects dropdown menu, but turn on calibration points and future calibration points. On Scope tab, right-click on the large, central "PointXP...Calpoints" button, and select "Yes" in the blue pop-up window that asks "Do you want to clear all PXP calibration points?". On Feature tab, click Run Script. Two windows open: Astrometry and Script.

6. In the Script window:
Select the Plate Solve Stuff tab, and enter the number of calibration points you want to include in the model (use 150 for a thorough model, or just 40 to get started. N.B it is possible to adjust the size of the zone of avoidance around the zenith (currently set to 5°) in SiTechExe/Config/Change Config!/Mount Params - "Cal Points Pole Distance (degs)), the exposure time (depends on sky conditions, but typically ≤8s), SHOC binning (1) and the delay between the telescope settling and the image being taken (7s is reasonable - note that the dome needs to settle in this time) and make sure "Wait for rotator" is ticked to stop the image being taken while the rotator is getting into position. Click "Make PointXP Run". The model will start gathering points with the telescope pointing South (if starting from the park position in the East), so you can either rotate the dome to the South before running the script, or press "Play", then "Pause" while the dome catches up with the telescope - you can check the dome position in the "Dome" tab of the Dome Control GUI (make sure it's set to track the telescope). This is the only time during the script that you would need to pause on account of the dome. Note that the way to resume the script is to click "Unpause", not "Play".

The script then moves the telescope from calibration point to calibration point in numerical order, taking an image with SHOC, using Astrometry.net to match the image to a catalogue, determining the position of the telescope, then moving on the telescope to the next point. Things to watch (lots!):

• The Script window, Plate Solve Stuff tab gives a running commentary of events, showing how far it has progressed through the script displayed on the right-hand side, and what is happening (slewing, waiting, counting down the preset delay, saving image, etc).
  
• The terminal window from which you launched SiTechCamera - it gives a running commentary of what the camera is doing (exposing, transferring data, getting status, etc).
  
• The Astrometry window has a dialogue box showing whether it is currently idle, or busy trying to match coordinates to an image. It then gives the results for each successful calibration (plate scale, rotation angle, number of extracted and matched stars, and how long it took to crunch). If the rotation angle is more that a few arcminutes, check that parallactic angle tracking is ticked on the rotator tab.
  
• On Skyview: for each successful calibration, a yellow mark will appear inside each red (numbered) calibration point. Click "Refresh" periodically to keep the two aligned as time passes.
• On SiTechExe: the central "PointXP" button keeps a count of the number of successful calibration points, and the RMS and Peak of the model as it builds.
• The SiTechCamera GUI displays each image. If points are failing to calibrate, check whether there are stars in the images.

7. If an image fails to calibrate, you can return to that point when the script has finished running. Left click on the failed calibration point in the Skyview window. It'll create a pop up saying e.g. "This is future cal point #86", click on that, then in the blue pop up window that follows, click "GoTo". Click "Do a single photo-init" in the Script window. It will take an image with the same camera parameters as were run in the script, unless you first e.g. adjust the exposure time, then click "Save PointXP Script Settings". If the image is solved successfully, a pop up window with a few options will appear - click on "Load cal star" then "OK".

8. To examine and edit the parameters of the resulting pointing model, left-click on the SiTechExe GUI "PointXP" button to open the PointXP window. Note that this does not update while the script is running, so you'll need to close and reopen the window to register new points. A typical pointing model for Lesedi has RMS≤2", Peak≤7". Points can be deleted by clicking on them in the display (deleted points turn red) or toggling "true" and "false" on the "CalPoints" tab, where you can also sort the list of points by e.g. error to match the points on the plot. Phi corresponds to azimuth, theta to altitude. The sensitivity parameter should be low, certainly <300.

9. Save the model by clicking "Save Text Calibration File" on the Main PXP tab of the PointXP6 window. The naming convention is YYYYMMDD_SHOC_Right_NorthUp_150points.PXP for a 150 point model made with SHOC on the right-hand rotator tracking the parallactic angle. If you intend to use the model, also save it as AutoLoadRight.PXP (overwrite existing file) so that it is loaded automatically when selecting the SHOC port. The model file selected for each port is set on the Tertiary Mirror tab of the SiTech TCP Focuser GUI (currently /usr/share/SiTech/SiTechExe/AutoLoadRight.PXP - if you change this, remember to click "SAVE Settings" on the Configuration tab).

10. Close the Script, PointXP6 and Astrometry windows, the SiTechCamera GUI and CTRL-C in the terminal to stop running the astrometry programme. If going back to normal operations, stop all SiTech software and reopen from lesedi-nuc, then do a "sudo systemctl restart lesedi" from 1ms1 and restart the autoguider software.

Troubleshooting

1. If the web GUI is unresponsive, showing a status that you don't believe, or reporting errors, the first thing to try is to reload the browser. If that doesn't solve the problem, restart the web server and backend using the following commands in a terminal:

      ssh observer@1ms1.suth.saao.ac.za
      sudo systemctl restart lesedi
      sudo systemctl restart lesedi-web

Then reload the browser window.

2. If there is a comms loss between a subsystem (rotator, secondary/tertiary mirror, dome) and ServoSCommunicator, a red(ish) warning button will appear in that subsystem's GUI complaining of Bad Comms. The first fix to try is to click on the red warning button. If the problem is solved, the button should disappear. If it doesn't, navigate to the appropriate subsystem's tab in the ServoSCommunicator GUI and reselect the port from the dropdown list. The ports are as follows:

• Rotators: /dev/ttyUSBRot2
• Secondary & tertiary mirrors: /dev/ttyUSBFoc3
• Dome: /dev/ttyUSBDom4

The pink warning box will disappear from the relevant GUI if the problem is resolved. If the problem persists, see point 3 below.

3. If clicking on a "Bad comms" warning button does not fix the problem, the SiTech software will need to be stopped and restarted. It is vital that all the SiTech GUIs are closed, and that they are reopened in the correct order, so follow this procedure.

4. The most common issue is loss of dome comms. The symptom will be that the dome won't move to a new target, or that it has closed without warning (the telescope will not park, mirror covers will not close: those actions suggest an LCU shutdown). A red warning button will appear labelled "Bad Dome Comms" in the Dome GUI - clicking it should solve the problem. If it does, go to the browser TCS Advanced tab, and in the central panel click "TAKE CONTROL", then turn dome "Tracking" ON and OPEN if the dome had closed without good reason in the middle of observing. If comms are not restored this way, then press Shutdown on the browser TCS and follow the full instructions on starting SiTech software to restart. Then reload the browser TCS and run the telescope startup procedure.

5. Focus is unresponsive. This is typically a comms loss to the SiTech focuser GUI and doesn't respond to the instructions in point 2. Follow the full instructions on starting SiTech software to restart.

6. No rotator comms. First try shutting down and restarting all SiTech. If that doesn't work, log into 1ms1 and do the following:

      ssh observer@1ms1.suth.saao.ac.za
      ls -l /dev/ttyU*

The output will be something like this:

      crw-rw---- 1 root     dialout  188, 1 Mar  3 20:07 /dev/ttyUSB1
      crwxr-xr-x 1 observer observer 188, 2 Mar  3 20:07 /dev/ttyUSB2
      crw-rw---- 1 root     dialout  188, 3 Mar  3  2022 /dev/ttyUSB3
      crw-rw---- 1 root     dialout  188, 5 Mar  3  2022 /dev/ttyUSB5
      lrwxrwxrwx 1 observer observer      7 Dec 15 01:34 /dev/ttyUSBDom4 -> ttyUSB2
      lrwxrwxrwx 1 root     root          7 Feb 28 10:35 /dev/ttyUSBFoc3 -> ttyUSB3
      lrwxrwxrwx 1 root     root          7 Feb 28 10:35 /dev/ttyUSBRot2 -> ttyUSB1
      lrwxrwxrwx 1 root     root          7 Feb 28 10:35 /dev/ttyUSBScp1 -> ttyUSB5

Look for the USB port associated with ttyUSBRot2 - in this case it is ttyUSB1. Check the serial number of the device on port ttyUSB1:

   observer@sa1:~$ udevadm info -a -n /dev/ttyUSB1 | grep '{serial}' 
   ATTRS{serial}=="A603G14W"
   ATTRS{serial}=="0000:00:14.0"

The serial number of the rotator controller is A603G14W, so this confirms that the rotators are communicating on port ttyUSB1. If the above command gives you a different serial number, edit the command to try each port from ttyUSB0 to ttyUSB5 until you find A603G14W. Now go to the ServoSCommunicator GUI and select the Rotators tab. In the dropdown menu, select the appropriate port - in this case ttyUSB1. Go to each of the Left Rotator GUI and the Right Rotator GUI and click on the pink warning complaining of no comms. If the warning doesn't clear, shutdown and restart all SiTech. ServoSCommunicator should wake up with the ttyUSB1 port selected, and communications to the rotator GUIs should be restored.

7. The telescope won't slew. First check that the telescope is unparked and that the target is within the visibility window. If those things are fine, the telescope may have triggered the hardware azimuth limit switch. This sometimes happens if the telescope is at azimuth ~165. Go to the SiTechExe GUI (the long, thin one) and at the top of the Scope tab, check if any of the arrows are red or orange. If they are, click the opposite arrow a few times to move the telescope out of the limit. If the motors are in "blinky", you may need to click the button at the bottom to switch the motors to automatic.

8. Telescope motors are in "blinky" mode and won't switch to auto. There is likely an error on the alt and/or az controller board which will need to be reset. Find the SiTechExe GUI (the long, thin one) and try pressing the button at the very bottom to switch the motors to Auto. If it does nothing, select the "Features" tab on the same GUI. Press the "Controller Stuff" button, which will open the "Controller Information Window". There are two columns in this GUI, one for alt, one for az. There is a text box about two-thirds of the way down each column, which will either say "No Errors", or list some errors. If there is an error in the box, click the "Reset Errors" button directly underneath that text box. Be very careful not to click any other button. If that has fixed the problem, the motors will now be in Auto mode - the button in the top left corner will say "To Manual (Blinky)", and the long, thin SiTechExe button will say "Motors Automatic" at the bottom. Close the Controller Information Window and resume operations. This almost always solves the problem, but if it persists, go to 8a).

8a) If point 8 didn't get the telescope motors back to Auto mode, close all SiTech software, power cycle the raspberry Pi in the altaz controller box under the North pier, then restart SiTech. If the motors aren't in Auto and can't be switched to Auto on startup, check for errors as per point 8 above. Still got trouble? Go to 8b.

8b) If 8 & 8a) don't solve the problem, close all SiTech software and reset the SiTech altaz controller under the North pier, then restart the SiTech software. If the motors aren't in Auto and can't be switched to Auto on startup, check for errors as per point 8 above.