Farm Data Transfer

Instructions to transfer data

Use screen command after log in

• After you log into the farm cluster, you can type screen command in your terminal, afterwards press Return to start. Now you can type you favorite wget/rsync command to download/transfer the data. When you wget/rsync command starts, type control+A+Z it would put your screen working in the background. If you want to resume your screen, type screen -list it would list all your screen working in the background and you will see the list of screen as below.

  There is a screen on: 19385.pts-0.c8-25 (12/22/2016 12:31:13 PM) (Detached)

• Now if you want to resume to one of the screen working in the background, type:

screen -r 19385.pts-0.c8-25

it would go back your previous wget/rsync command. If you want to exit the screen, then type exit.

Using rsync from your own computer

• Another way to transfer files is to run rsync from your local machine. To do this, type:

  $ rsync -avz -e 'ssh' [path file comes from] [path file goes to]

Instructions to download and upload data from iplant

• Now many people store the raw data on the iplant. If you want to download the data from iplant, be careful sometimes the connection would break before you finish uploading or downloading the files, and you can use below to avoid the problem (at least working for me):

  $ iget -K -P -T --retries 3 --lfrestart checkpoint-file path_to_your_file

• If you want to upload all the files in the directory to iplant, you can use below command:

  $ iput -K -P -b -f -r -T --retries 3 -X checkpoint_file upload_path_directory location_iplant_upload

• If you want to download many things, you can put multiple iput and iget command in a bash script. You can use command below to document the process of upload or download. In the file record_download_upload_progress.txt you can see how many percentage of your files have been upload or download to the iplant.

  $ sh bash_script_upload_download_iplant.sh | tee -a record_download_upload_progress.txt

Farm is a great cluster as long as you treat it nicely, and in return farm would do great things for your research, enjoy the farm cluster!