Data Transfer

Transfer

A script runs daily on astro.suth, and pulls recent data from all the instruments of which it is aware. At the moment, this includes shocnawe, shocndisbelief and SpUpNIC, but we will be adding more instruments as we have the opportunity. We are also configuring a version of this script to run on astro.cape, after the astro.suth script has completed; this will synchronize data between Sutherland and Cape Town. There should then be two back-ups of the data on each instrument.

By default, the script runs from a cron job, but can also be run manually. The script synchronizes data from the last N days, where N is specified by the user. The default value is 30; this should ensure that if data is not copied on some particular day, it will be copied on a later day. If the instrument is unmounted before the scheduled sync time, or data is acquired after the sync but before unmounting, then this might lead to data not being immediately copied for that day. If the machine is off for more than 30 days (a currently configured), then a manual sync should be done.

It is possible to specify a specific date to synchronize (-d option), as well as a specific instrument (-i option).

Verification

There's a script that runs daily at 9:00 to confirm that all data have been copied from the instruments onto astro.suth and astro.cape. The script can be viewed here while logged into Bitbucket.

How it works

This requires Paramiko, is currently configured inside of my home directory on astro2015.cape.saao.ac.za at /home/briehan/check_data.py and works by checking for the existence of fits files inside of the directories for this month and last month.

The crontab for my user:

MAILTO=briehan@saao.ac.za,amanda@saao.ac.za,carel@saao.ac.za
0 9 * * * /home/briehan/venv/bin/python /home/briehan/checkdata.py

Example output

An example of the output generated by the script is included below:

/data/74in/sha/
--------------------------------------------------------------------------------
FILES ON sha.suth.saao.ac.za THAT ARE NOT ON astro.suth.saao.ac.za
--------------------------------------------------------------------------------

2016/0907/SHA_20160907.0122.fits
2016/0907/SHA_20160907.0123.fits
2016/0907/SHA_20160907.0124.fits
2016/0907/SHA_20160907.0125.fits
2016/0907/SHA_20160907.0126.fits
2016/0907/SHA_20160907.0127.fits
2016/0907/SHA_20160907.0128.fits
2016/0907/SHA_20160907.0129.fits
2016/0907/SHA_20160907.0130.fits
2016/0907/SHA_20160907.0131.fits
2016/0907/SHA_20160907.0132.fits
2016/0907/SHA_20160907.0133.fits
2016/0907/SHA_20160907.0134.fits
2016/0907/SHA_20160907.0135.fits
2016/0907/SHA_20160907.0136.fits
2016/0907/SHA_20160907.0137.fits
2016/0907/SHA_20160907.0138.fits
2016/0907/SHA_20160907.0139.fits
2016/0907/SHA_20160907.0140.fits
2016/0907/SHA_20160907.0141.fits
2016/0907/SHA_20160907.0142.fits
2016/0907/SHA_20160907.0143.fits
2016/0907/SHA_20160907.0144.fits
2016/0907/SHA_20160907.0145.fits
2016/0907/SHA_20160907.0146.fits
2016/0907/SHA_20160907.0147.fits
2016/0907/SHA_20160907.0148.fits
2016/0907/SHA_20160907.0149.fits
2016/0907/SHA_20160907.0150.fits
2016/0907/SHA_20160907.0151.fits
2016/0907/SHA_20160907.0152.fits
2016/0907/SHA_20160907.0153.fits
2016/0907/SHA_20160907.0154.fits
2016/0907/SHA_20160907.0155.fits
2016/0907/SHA_20160907.0156.fits
2016/0907/SHA_20160907.0157.fits
2016/0907/SHA_20160907.0158.fits
2016/0907/SHA_20160907.0159.fits
2016/0907/SHA_20160907.0160.fits
2016/0907/SHA_20160907.0161.fits
2016/0907/SHA_20160907.0162.fits
2016/0907/SHA_20160907.0163.fits


/data/telescopedata/74in/sha/
--------------------------------------------------------------------------------
FILES ON astro.suth.saao.ac.za THAT ARE NOT ON astro.cape.saao.ac.za
--------------------------------------------------------------------------------

None