parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Using parallel to move files from EC2 to AWS S3 bucket


From: Michael A Nesmith
Subject: Using parallel to move files from EC2 to AWS S3 bucket
Date: Wed, 28 Nov 2018 14:08:15 +0000

Hi,


I am trying to move files from an EC2 Linux instance to a S3 staging bucket.


I have the parallel option working as follows:


--------------------------------------------------------------------------------------------------------


=====================================================================================================================================

address@hidden predev]# cat mytest3.sh
TRANSACTIONID=ec9ba83b
VERTICALID=rfx
VERSION=predev
FILEPATH=/nfsshare/predev/rfx/inbound
FILENAME=NFS.ICPRIV.R.RF3.AHC.EBB21.1636227-EB_B21_QQ_4.txt
file="/nfsshare/predev/rfx/inbound/NFS.ICPRIV.R.RF3.AHC.EBB21.1636227-EB_B21_QQ_4.txt"


ls "$file" | parallel --will-cite aws s3 mv {} s3://hhs-cms-epl-epor-stg-"${VERSION}"-v3/"$VERTICALID""_""$FILENAME" \
                              --content-disposition "attachment" \
                                         && \
              echo "file moved to s3 staging bucket"




======================================================================================================================================


address@hidden predev]# ./mytest3.sh
move: ../../../../nfsshare/predev/rfx/inbound/NFS.ICPRIV.R.RF3.AHC.EBB21.1636227-EB_B21_QQ_4.txt to s3://hhs-cms-epl-epor-stg-predev-v3/rfx_NFS.ICPRIV.R.RF3.AHC.EBB21.1636227-EB_B21_QQ_4.txt
file moved to s3 staging bucket

address@hidden predev]# aws s3 ls hhs-cms-epl-epor-stg-predev-v3 | grep NFS.ICPRIV.R.RF3.AHC.EBB21.1636227-EB_B21_QQ_4.txt | wc -l
1


----------------------------------------------------------------------------------------------------------


The problem now seems to be the use of the --metadata option as below:

============================================================================================================================

address@hidden predev]# cat mytest2.sh
TRANSACTIONID=ec9ba83b
VERTICALID=rfx
VERSION=predev
FILEPATH=/nfsshare/predev/rfx/inbound
FILENAME=NFS.ICPRIV.R.RF3.AHC.EBB21.1636227-EB_B21_QQ_4.txt
file="/nfsshare/predev/rfx/inbound/NFS.ICPRIV.R.RF3.AHC.EBB21.1636227-EB_B21_QQ_4.txt"


ls "$file" | parallel --will-cite aws s3 mv {} s3://hhs-cms-epl-epor-stg-"${VERSION}"-v3/"$VERTICALID""_""$FILENAME" \
                              --content-disposition "attachment" \
                              --metadata \
                                        "{\"source\":\"nfs\", \
                                        \"originalname\":\"$FILENAME\", \
                                        \"uid\":\"null\", \
                                        \"verticalid\":\"$VERTICALID\", \
                                        \"read\":\"icprivusrgrp\", \
                                        \"write\":\"icprivusrgrp\", \
                                        \"owner\":\"icprivusrgrp\", \
                                        \"transactionID\":\"$TRANSACTIONID\", \
                                        \"callbackurl\":\"null\"}" && \
              echo "file moved to s3 staging bucket"




=============================================================================================================================

With the --metadata option, I get the following error:


address@hidden predev]# ./mytest2.sh

Error parsing parameter '--metadata': Invalid JSON: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
JSON received: {source:nfs,
address@hidden predev]#



Any ideas from anyone on how to overcome this?  I can now move data to the S3 bucket with the parallel option, but really need to have the metadata set as well.


Thoughts?





Thanks,

Mike





Michael A. NeSmith  AWS UNIX Cloud Administrator

C-HIT |Enterprise Portal Contract

301.799.7864 | address@hidden

10480 Little Patuxent Pkwy, Ste 800, Columbia MD 21044

CMMI ML Dev 3 & Srv 3

Splunk Certified

 

“This electronic mail and any files transmitted within are confidential and is intended solely for the individual or entity to whom it is addressed. This message contains information that is privileged and is otherwise protected from disclosure to anyone other than the intended recipient. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the named addressee you should not disseminate, distribute or copy this e-mail and you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.”



reply via email to

[Prev in Thread] Current Thread [Next in Thread]