|From:||Michael A Nesmith|
|Subject:||Question on copying files to AWS|
|Date:||Fri, 16 Nov 2018 19:13:43 +0000|
I am new to both AWS and GNU Parallel software.
Please pardon a beginner's questions.
I need to copy/move files from my AWS Linux EC2 instance (Redhat 6.10) to an S3 bucket that was previously created.
Normally the command to copy a file (for example, messages) would look like this:
aws s3 cp messages s3://hhs-cms-epl-epor-stg-predev-v3
To verify the file is in the S3 bucket, I use the command:
aws s3 ls hhs-cms-epl-epor-stg-predev-v3 | grep messages
The problem is that I need to upload a lot of files (100,000 or so) and some of them are quite big (1 GB to 5 GB), and it takes a long time to upload so many files or very big ones.
I need to speed this process up.
The questions are:
1) Would GNU parallel software speed up this copy process significantly?
2) What is the syntax of the command when using S3 buckets?
If I understand the man page and the beginner youtube page, to use the parallel option one should use something similar to:
parallel aws s3 cp messages s3://hhs-cms-epl-epor-stg-predev-v3
However the output I get is:
# parallel aws s3 cp messages s3://hhs-cms-epl-epor-stg-predev-v3
parallel: Warning: Input is read from the terminal. You either know what you
parallel: Warning: are doing (in which case: YOU ARE AWESOME!) or you forgot
parallel: Warning: ::: or :::: or to pipe data into parallel. If so
parallel: Warning: consider going through the tutorial: man parallel_tutorial
parallel: Warning: Press CTRL-D to exit.
The file does not copy.
Any advice on how to do this properly?
Thank you in advance to any who have time to help.
|[Prev in Thread]||Current Thread||[Next in Thread]|