Difference between revisions of "Data Transfer"

Latest revision as of 09:13, 20 October 2017

Transfer

A script runs daily on astro.suth, and pulls recent data from all the instruments of which it is aware. At the moment, this includes shocnawe, shocndisbelief and SpUpNIC, and STE on the 40inch but we will be adding more instruments as we have the opportunity. We are also configuring a version of this script to run on astro.cape, after the astro.suth script has completed; this will synchronize data between Sutherland and Cape Town. There should then be two back-ups of the data on each instrument.

The script is run daily at 08:00 AM from a cron job, but can also be run manually. It is possible to specify a specific date to synchronize (-d option), as well as a specific instrument (-i option). One can also specify the number of past days (counting back from today) for which the data is synchronized (the -n option).

By default the last 30 days are synchronized; this should ensure that if data is not copied on some particular day, it will be copied on a later day. If the instrument is unmounted before the scheduled sync time, or data is acquired after the sync but before unmounting, then this might lead to data not being immediately copied for that day. If the machine is off for more than 30 days (as currently configured), then a manual sync should be done after the instrument is remounted.

On astro.suth the data is stored under /data/telescopedata. The path to the data then takes the form <telescope>/<inst>/<YYYY>/<MMDD>/. Here the telescope can be one of 74in, 40in or 1m, and the inst is the abbreviated instrument name (one od sha, shd or spupnic). So for example, data acquired by shocnawe on the 74-inch telescope on the night of the 8th/9th of September, 2016, would be in/data/telescopedata/74in/sha/2016/0908. Note: the data is archived on the astro servers using this standard SAAO/SALT data path; some instruments may not yet use this exact directory structure for local (on-instrument) data storage.

Verification

There's a script that runs daily at 9:00 to confirm that all data have been copied from the instruments onto astro.suth and astro.cape. The script can be viewed here while logged into Bitbucket.

How it works

This requires Paramiko, is currently configured inside of my home directory on astro2015.cape.saao.ac.za at /home/briehan/check_data.py and works by checking for the existence of fits files inside of the directories for this month and last month.

The crontab for my user:

MAILTO=briehan@saao.ac.za,amanda@saao.ac.za,carel@saao.ac.za
0 9 * * * /home/briehan/venv/bin/python /home/briehan/checkdata.py

Example output

An example of the output generated by the script is included below:

/data/74in/sha/
--------------------------------------------------------------------------------
FILES ON sha.suth.saao.ac.za THAT ARE NOT ON astro.suth.saao.ac.za
--------------------------------------------------------------------------------

2016/0907/SHA_20160907.0122.fits
2016/0907/SHA_20160907.0123.fits
2016/0907/SHA_20160907.0124.fits
2016/0907/SHA_20160907.0125.fits
2016/0907/SHA_20160907.0126.fits
2016/0907/SHA_20160907.0127.fits
2016/0907/SHA_20160907.0128.fits
2016/0907/SHA_20160907.0129.fits
2016/0907/SHA_20160907.0130.fits
2016/0907/SHA_20160907.0131.fits
2016/0907/SHA_20160907.0132.fits
2016/0907/SHA_20160907.0133.fits
2016/0907/SHA_20160907.0134.fits
2016/0907/SHA_20160907.0135.fits
2016/0907/SHA_20160907.0136.fits
2016/0907/SHA_20160907.0137.fits
2016/0907/SHA_20160907.0138.fits
2016/0907/SHA_20160907.0139.fits
2016/0907/SHA_20160907.0140.fits
2016/0907/SHA_20160907.0141.fits
2016/0907/SHA_20160907.0142.fits
2016/0907/SHA_20160907.0143.fits
2016/0907/SHA_20160907.0144.fits
2016/0907/SHA_20160907.0145.fits
2016/0907/SHA_20160907.0146.fits
2016/0907/SHA_20160907.0147.fits
2016/0907/SHA_20160907.0148.fits
2016/0907/SHA_20160907.0149.fits
2016/0907/SHA_20160907.0150.fits
2016/0907/SHA_20160907.0151.fits
2016/0907/SHA_20160907.0152.fits
2016/0907/SHA_20160907.0153.fits
2016/0907/SHA_20160907.0154.fits
2016/0907/SHA_20160907.0155.fits
2016/0907/SHA_20160907.0156.fits
2016/0907/SHA_20160907.0157.fits
2016/0907/SHA_20160907.0158.fits
2016/0907/SHA_20160907.0159.fits
2016/0907/SHA_20160907.0160.fits
2016/0907/SHA_20160907.0161.fits
2016/0907/SHA_20160907.0162.fits
2016/0907/SHA_20160907.0163.fits


/data/telescopedata/74in/sha/
--------------------------------------------------------------------------------
FILES ON astro.suth.saao.ac.za THAT ARE NOT ON astro.cape.saao.ac.za
--------------------------------------------------------------------------------

None