Re: EC2 integration for ssh hosts?

[Ole Tange]

On Thu, Mar 3, 2011 at 10:52 AM, Matt Oates (Home) <mattoates@gmail.com> wrote:
>
> To start out thanks for the great work on parallel it lets me get my
> work done! Stuff that took a day now takes an hour :)

Glad you like it.

> I'm a bioinformatician and I need to deal with a tonne of data in
> parallel. GNU parallel has been great for this on our 24 CPU machine
> so far! Looking at the archives for this list, I'm looking forward to
> --pipe with bated breath. FASTA files are all I deal with.

Wow? You have no fastq files? Lucky you.

--pipe was included in the FOSDEM release (20110205).

> However, we're reaching the limits of most hardware laying around, my
> research group are moving to the Amazon Elastic Cloud. I was wondering
> if anyone can gauge how much heartache there would be in hacking on
> GNU parallel using the Net::Amazon::EC2 module to bring up a bunch of
> cloud instances, and then pass this off as the list of ssh remote
> hosts? I could just wrap something around parallel in the shell but it
> feels like a nice optional feature for others..?

While I am a bioinformatician, too, and while we too are looking at
EC2, I do not see a Net::Amazon::EC2 interface becoming a part GNU
Parallel: One of the primary goals is to keep the additional
requirements to install GNU Parallel very low. Requiring
Net::Amazon::EC2 installed to run will go very hard against that goal.

I am not sure what the best way to interface GNU Parallel with
Net::Amazon::EC2, but my suggestion would be for you to wrap something
around GNU Parallel and post your wrapping here. If you write your
wrapper in Perl, it might be easier to see if we can find a good way
to make a general solution.


/Ole