View source for Quick Look Tool

SpUpNIC Quick Look Reduction Tool


== Introduction ==

The Quick Look tool is designed to allow users to quickly assess the quality of their data. It automatically produces simple 1D extractions of science data with a reasonable guess for the target of interest (selectable as either the highest peak or the most central peak in the spatial direction); attempts a sky subtraction using two sidebands (which will also subtract the bias from the raw frame), and produces a 1D extraction.

For arcs, automatic wavelength calibration is attempted. The accuracy of this is appropriate for the spirit of a quick look tool and is typically ~2A. This allows the observer to check that the wavelength range of interest is being covered and to identify spectral features in science spectra. This is not intended to be a final science-quality level reduction and there are a number of caveats on the extraction and calibration process listed below. However, for some users, this may be entirely sufficient for their science needs. Even if it is not precise enough, it can be used as a starting point for a more careful reduction and notes on this are given here https://topswiki.saao.ac.za/index.php/Using_QL_outputs.

<gallery>FullSci.png|Overview of GUI</gallery>

== Change Log ==

'''* 21 December 2016:''' Creation of this page. Initial notes on using the QL GUI, now that the software is stable. Additional details, including videos showing common tasks will be added in the near future. For questions about anything on this page, or use of the software, please email David Gilbank "gilbank AT saao dot ac dot za"

== Quick Start ==

* At the start of a new night of observing, press the "'''Refresh from SpUpNIC Control'''" button. This updates the data directory where the QL looks for new data. 

* As a new image is written to disk from the observing GUI (SpUpNIC Control), it will appear in the QL window. Each image will appear as the 2D raw image (upper panel) and a 1D extraction (lower panel). More details of how this is done are given below.

* If an '''arc''' is found, the QL tool will attempt an automatic wavelength calibration and plot the calculated wavelength scale on the image. The accuracy of this solution can be easily seen by comparing with the library reference arc (green spectrum) plotted over the top.

* If a '''science''' frame is found, the wavelength scale shown will be taken from the previous arc. NOTE: there is no checking that the previous arc is for the same target/configuration as the science frame. It is simply assumed that the observer will always take an arc prior to a science exposure.

* As new data rolls in, it is added to the '''Observing Log''' (accessible under the tab of the same name). This will provide a summary of the most useful header keywords for each image. This may be written to a file at the end of the night to produce a record of all the data produced. It is also possible to click anywhere in the row on any of the image names listed in the log and reload the corresponding image in the QL tool.
**'''Useful trick''': if you want to apply a wavelength solution of an arbitrary arc to a science frame, simply load the arc followed by the science frame. Frames may also be loaded by typing the name in the "Current Frame" box and clicking update. 

* For each frame, a '''1D extraction''' in both ascii and FITS format is automatically written to a subdirectory, redux/, of the input data directory.

== Layout ==
=== Input data directory ===

[[File:DataDir.png|thumbnail]]

The input data dir is the directory being monitored for the arrival of new files. Make sure this points to the place where your data are being written by the SpUpNIC control GUI. The easiest way to do this is by clicking the "Refresh from SpUpNIC Control" button at the start of the night. As an alternative, another directory may be selected and frames loaded manually to examine/process existing data.

=== Info pane ===
<gallery>InfoPane.png|Info pane</gallery>
Below the input data directory is a list of information about the current frame, generated from its header. The '''"Current Frame"''' box may be used to load another frame, either using the file browser ('''"Browse"''' button) or by simply typing a new file name within the box and pressing enter/'''"Update"'''.

The '''"Arc Used"''' box keeps a record of the arc used to wavelength calibrate a given science frame. 
NOTE: updating this will not apply a solution from a different arc. If you would like to do this, load the desired arc in the "Current Frame" box, wait for the wavelength solution to appear, and then load the science frame to which you would like to attach this calibration.

=== 2D spectrum view ===

[[File:Im2Dpane.png|thumbnail]]

The uppermost plot window in the QL GUI shows the raw 2D Spectrum as it comes off the telescope. The GUI attempts to scale this in a sensible way (using IRAF/ds9's zscale algorithm, by default). Alternate scaling may be selected by choosing from the drop down menu on the right of the panel and pressing "Update". Manual values of z0 and z1 may be chosen by entering them in the appropriate boxes and choosing "Manual" as the scaling option. Alternative colour maps may be selected.

[[File:Sci2D.png|thumbnail]]

The 2D window shows two horizontal cyan lines which should coincide with the upper and lower bounds of the spectrum. These have been manually set for each grating and show the bounds over which analysis will take place. For example, side bands for sky subtraction are truncated at these edges. If these are not in the appropriate place, they may be tuned by editing a configuration file as described here https://topswiki.saao.ac.za/index.php/Quick_Look_Tool#Edges_of_2D_spectrum . These bounds are also used to define where an arc is extracted (see end of section below).

Above the plot are check boxes for displaying the extraction and side band regions used in the sky subtraction and spectral extraction. The display of these may be turned on/off with these check boxes. NOTE: this does not affect their use, only their display in the 2D view.

=== Profile Pane ===

[[File:ProfPlotSci.png|thumbnail]]

The profile pane shows a cross section in the spatial direction through the spectrum. This controls the regions used for extraction and sky subtraction of the science data. The default is to search for the '''highest peak''' in the spectrum automatically and to fit a region +/-2 sigma around a Gaussian fit to this peak. An alternative is to choose the '''most central peak''', assuming observers try to centre the object within this slit. This setting is useful when a brighter object than the intended target also falls within the slit. If neither of these settings is suitable, the user may '''manually''' identify the extraction region using the extraction window sliders below the plot window and clicking '''"Update with Manual Settings"''' to apply the changes. To return back to auto extraction, simply select the '''Auto''' radio button. Note: there may be a small delay after adjusting any of these, since the software is recalculating the extracted spectrum and updating all the plot windows in real-time.

Similarly, the side bands used to identify the sky may be tweaked interactively. The regions are not allowed to cross the extraction region, nor to fall outside of the cyan slit boundary. If one of the regions is contaminated by another object, it may be deselected using the check box below the band slider. 

Finally there is an option to choose the '''extraction type'''. The default is to '''sum''' all of the rows within the extraction window. Note: no correction is made for spectral tilt, which may be important for users concerned with the highest possible spectral resolution. The automatic extracted spectra always use this option regardless of the setting chosen for display. The alternative choice is to only show the '''peak row'''. This can be useful when checking that the spectra are not saturating.

Finally, for an arc, most of these options are disabled. The '''arc extraction''' simply uses the '''single row''' at the mid point between the two cyan extraction edge lines. This single row extraction: minimises the potential impact of cosmic rays; the broadening of lines by summing over the tilted lines in the full window; and should provide sufficient S/N with even short exposures.
<gallery>
ProfPlotArc.png|Profile pane for an arc
</gallery>

=== 1D spectrum view ===

[[File:Im1Dpane.png|thumbnail]]

The plot of the 1D spectrum is probably the most important/useful of all. It shows the extracted, sky-subtracted (if applicable) spectrum obtained using the settings given in the Spatial Profile pane, and is updated in real-time as settings are changed. 

If the spectrum is labelled as an '''arc''' in the header, the QL tool will attempt to perform automated wavelength calibration. This is generally robust and no changes of settings should be needed. For the less frequently used gratings or for unusual grating angles, the default parameters may not work so well, and so notes for advanced users to change settings to the fitting routine are provided here https://topswiki.saao.ac.za/index.php/Quick_Look_Tool#WLC_advanced_tab .
After wavelength calibration of an arc, a '''library reference spectrum''' will be overplotted in green allowing the user to quickly check the quality of the calibration. As mentioned in the introduction, this is intended to be a quick look calibration at the ~2A level and solutions must be refined by the user if greater precision is needed to meet the science requirements. As a further aid, the wavelength of the midpoint of the spectrum is printed to the bottom of the screen, allowing the observer to monitor any possible drifts in the wavelength coverage.

Should the user, for some reason, not want to use the wavelength calibration, or just view the data in pixel units, the '''"ignore WLC"''' button may be used. In this case, the output extracted spectrum written will also only contain the pixel solution. To retrieve the wavelength solution, reload the arc by pressing "Update" next to the Current Frame box.

[[File:Sci1D.png|thumbnail]]

For a '''science spectrum''', the plot will show both the extracted objected and the subtracted background (which is the sky + bias). The '''wavelength calibration''' of the preceding arc will be used, under the assumption that the observer will always be taking an arc prior to a science exposure. If this is incorrect and the wrong WLC becomes attached, use "ignore WLC".

In addition, buttons for '''smoothing''' the spectra are provided, as well as an option to '''print''' the 1D view. This will produce a PDF file in /tmp/tmp.pdf and send it to the printer in the warm room.

=== GUI log ===

<gallery>
LogPane.png|Log Pane
</gallery>
The lower right corner of the GUI contains a log of actions. In most cases this is just a record of the frames found and loaded. For some operations, such as a Hartmann focus run, this provides other useful information.

=== Observing Log Tab ===
<gallery>
ObsLog.png | Observing Log tab
</gallery>

The observing log tab contains a record of observations as they roll off the telescope. This is automatically generated from the headers as images appear on disk. This log may be saved to the '''redux/''' subdirectory of the input data directory. 

A useful feature of the log is the ability to reload previous images. Simply click anywhere on the row containing the desired image and click '''"Load Selected Image"'''. This can be particularly useful for applying an arbitrary arc's WLC to a science frame. Simply load the arc followed by the science frame and the WLC will be transferred and the extracted spectrum (with this solution) automatically written to disk.

== Examples ==
<gallery>
fullArc.png | Example view of the whole GUI for an arc
fullSci.png | Example view of the whole GUI for a science frame
</gallery>

== Output Files ==

The QL tool puts output files in a subdirectory of the root data directory named redux/
e.g. if your data are being saved to '''/home/ccd/data/20161212/''', output files will appear in '''/home/ccd/data/20161212/redux/'''

The default files to produce (if the appropriate check boxes have not been unchecked in the QL GUI) are a '''1D ascii file''' and a '''1D extracted FITS spectrum''', a0161390'''_1D.txt''' and a0161390'''.ms.fits''' respectively, for an input image named a0161390.fits. Both of these contain the same data, just in a different format depending on the user's preference. Additionally, a log recording the extraction settings used is stored in a0161390'''.json'''. This last file is probably not of interest to most users.

=== ascii output ===
The ascii spectrum contains the following columns, as labelled in the header:
'''wavelen objCounts skyAndBiasCounts'''

* If the file has been wavelength calibrated, the first column contains the wavelength in A, one line per pixel number. Otherwise the first column will be pixel number (i.e. just running number from 1 to Npixels). 

* The second column is the counts in the extraction aperture after sky subtraction (if selected). Note: this is always '''summed counts within the aperture''', even if another extraction option (such as peak counts) is selected.

* The third column contains the '''counts per pixel''' from the sky aperture(s). If sky subtraction has been unselected, these will be 0.

=== 1D FITS file ===

The .ms.fits file contains the 1D ascii data converted to look like an IRAF multispec format file. This makes it easy to read into IRAF. This conversion includes: placing extracted spectra and sky bands in appropriate IRAF multispec "bands"; applying the WCS following IRAF's WAT format; including aperture details in the way IRAF understands to state where the spectrum was extracted from the 2D image.




'''Other notes:''' Again, the extraction is only carried out on the individual image (named in the filename). The data has been flipped in x such that the output, extracted spectra run blue to red in increasing x. If comparing with the raw data, keep this in mind. 

For using these as the starting point for your '''science-quality reduction''': https://topswiki.saao.ac.za/index.php/Using_QL_outputs

=== .json file ===

The .json file contains a log of the settings of parameters used in the extraction. It may be of interest to users wishing to write their own code for taking these extractions as a starting point, or to users as a double check of which arc was used to wavelength calibrate their data ("currArc").

example contents of .json file:

<small>{
"useSB1": true, 
"useSB2": true, 
"extractWin_x1": 69.0, 
"extrType": "1", 
"SBwidth": 15, 
"SB1_x1": 48.0, 
"dataDir": "/home/visitor/spupdata/20160404/", 
"SB1_x0": 33.0, 
"currFluxCal": "none", 
"obsType": "ARC", 
"SB2_x1": 48.0, 
"LockDisplays": false, 
"autoExtraction": 1, 
"extractWin_x0": 53.0, 
"currFlat": "none", 
"SBoffset": 5, 
"SkySub": false, 
"currArc": "a0161389", 
"scaling": "ZScale", 
"smoothPix": 0, 
"outDataDir": "/home/visitor/spupdata/20160404/redux/", 
"z0": 369.965, 
"z1": 1010.042, 
"sx": 2148, 
"currFrame": "a0161389", 
"pkFindType": 1, 
"showSideBands": true, 
"SB2_x0": 33.0, 
"currBias": "none"
}</small>

== Advanced config ==

=== Edges of 2D spectrum ===

'''Please don't modify software without discussing and clearing the modification with the software developers or instrument PI.''' To edit the '''bounds of the 2D spectrum''' (cyan lines in 2D plot window) edit as root the following file:
'''/usr/local/etc/qlgui/pars.json'''
The upper and lower bounds of the slit (which can be read off the 2D plot window using the cursor) are given by '''sl0''' and '''sl1''' respectively for the given grating name. e.g. for grating 5, the upper and lower bands are gr5_sl1 and gr5_sl0. Edit the numbers appropriately and resave the file. The changes should take effect the next time a spectrum is loaded.

=== WLC advanced tab ===
In rare cases, the automatic wavelength calibration may fail. The "WLC advanced" tab provides settings to tweak the method used in the fitting. The 2x2 grid of plots on the left show some useful diagnostics of the fitting.

Suggestions for parameter adjustments are given next to the various options on the lower right of the screen. After adjusting any of these, please make sure to reload the current frame on the main tab (not the WLC advanced tab) for the changes to take effect. Note: if the GUI is restarted, these settings will revert to their default values. If you find a set of values which work better than the defaults for an unusual arc/grating angle (or need help adjusting settings), please email "carel AT @saao dot ac dot za" for these to be incorporated into the QL software. The default values should be robust for the most commonly used setups.