The cycle of download and move

Background

My cluster has two nodes:

​ node_1: Public IP, small disk space;

​ node_2: Private IP, large disk space;

So, I need to download files on node_1 and move all of them to node_2 at intervals.

Method

  1. After enough large files are downloaded, I just move them to node_2;

  2. Copy files’ attributes to node_1

    cp -r --attributes-only <node_1_directory> <ndoe_2_directory>
    
  3. Download again by -nc option on node_1:

    wget --no-check-certificate -nc -e robots=off -r -l inf -np -R .html,.tmp -nH --cut-dirs=3 https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/501249136/ --header "Authorization: Bearer <APP_KEY>" -P ./
    
  4. Delete empty fiels:

    find . -type f -empty -delete
    
  5. Repeat 1-4……


Say something

Thank you

Your comment has been submitted and will be published once it has been approved.

Click here to see the pull request you generated.

OOPS!

Your comment has not been submitted. Please go back and try again. Thank You!

If this error persists, please open an issue by clicking here.