[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Question on copying files to AWS

From: Rob Sargent
Subject: Re: Question on copying files to AWS
Date: Sun, 18 Nov 2018 14:31:46 -0700

You might pipe ls -R into that command but you’ll want access to multiple machines to run a slew of parallel cp jobs. GNU parallel can use a hosts file to start remote jobs. Each of those jobs needs a supply of files to copy

On Nov 16, 2018, at 12:13 PM, Michael A Nesmith <address@hidden> wrote:

Hi, everyone,

I am new to both AWS and GNU Parallel software.

Please pardon a beginner's questions.

I need to copy/move files from my AWS Linux EC2 instance (Redhat 6.10) to an S3 bucket that was previously created.

Normally the command to copy a file (for example, messages) would look like this:

     aws s3 cp messages s3://hhs-cms-epl-epor-stg-predev-v3 

To verify the file is in the S3 bucket, I use the command:

     aws s3 ls hhs-cms-epl-epor-stg-predev-v3 | grep messages

The problem is that I need to upload a lot of files (100,000 or so) and some of them are quite big (1 GB to 5 GB), and it takes a long time to upload so many files or very big ones.

I need to speed this process up.

The questions are:

1) Would GNU parallel software speed up this copy process significantly?

2) What is the syntax of the command when using S3 buckets?

If I understand the man page and the beginner youtube page, to use the parallel option one should use something similar to:

     parallel aws s3 cp messages s3://hhs-cms-epl-epor-stg-predev-v3

However the output I get is:

# parallel aws s3 cp messages s3://hhs-cms-epl-epor-stg-predev-v3
parallel: Warning: Input is read from the terminal. You either know what you
parallel: Warning: are doing (in which case: YOU ARE AWESOME!) or you forgot
parallel: Warning: ::: or :::: or to pipe data into parallel. If so
parallel: Warning: consider going through the tutorial: man parallel_tutorial
parallel: Warning: Press CTRL-D to exit.
address@hidden tmp2]#

The file does not copy.

Any advice on how to do this properly?

Thank you in advance to any who have time to help.




Michael A. NeSmith  AWS UNIX Cloud Administrator

C-HIT |Enterprise Portal Contract

301.799.7864 | address@hidden

10480 Little Patuxent Pkwy, Ste 800, Columbia MD 21044

CMMI ML Dev 3 & Srv 3

Splunk Certified


“This electronic mail and any files transmitted within are confidential and is intended solely for the individual or entity to whom it is addressed. This message contains information that is privileged and is otherwise protected from disclosure to anyone other than the intended recipient. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the named addressee you should not disseminate, distribute or copy this e-mail and you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.”

reply via email to

[Prev in Thread] Current Thread [Next in Thread]