Background
My cluster has two nodes:
node_1: Public IP, small disk space;
node_2: Private IP, large disk space;
So, I need to download files on node_1 and move all of them to node_2 at intervals.
Method
After enough large files are downloaded, I just move them to node_2;
Copy files’ attributes to node_1
cp -r --attributes-only <node_1_directory> <ndoe_2_directory>
Download again by
-nc
option on node_1:wget --no-check-certificate -nc -e robots=off -r -l inf -np -R .html,.tmp -nH --cut-dirs=3 https://ladsweb.modaps.eosdis.nasa.gov/archive/orders/501249136/ --header "Authorization: Bearer <APP_KEY>" -P ./
Delete empty fiels:
find . -type f -empty -delete
Repeat 1-4……
Say something
Thank you
Your comment has been submitted and will be published once it has been approved.
OOPS!
Your comment has not been submitted. Please go back and try again. Thank You!
If this error persists, please open an issue by clicking here.