Re: Octave Interpreter

[Stefan Seefeld]

On 2014-10-01 21:16, Bipin Mathew wrote:
> Hello Manu,
>
>     I was actually thinking of working on something along the same
> lines. Googling around, I see a few efforts along these lines:
>
> http://octave.sourceforge.net/parallel/
> http://octave.sourceforge.net/mpi/index.html
>
> While I have not used these personally, looking at the examples, it
> appears one needs to be quite aware of MPI / parallel programming to
> make effective use of these package. It would be nice if all these
> things could be made more transparent to the end user.
>
> Many years ago, I actually worked with a team at MIT Lincoln
> Laboratory, designing this sort of  thing for Matlab:
>
> http://www.ll.mit.edu/mission/cybersec/softwaretools/pmatlab/pmatlab.html
>
> The idea was to hide as much of the parallelism as possible. This was
> done by using a special "dmatrix" class to represent your matrix,
> which was, in reality, spread across several servers. I don't know if
> this is still an active project there ( Any of my old Lincoln friends
> can comment? :-) ).

Interesting that you bring this project up, as I was considering
proposing something similar. While I haven't worked at the MIT/LL, I
have collaborated with them.

> Of course technology has moved on since then and there is probably
> alot of new technology we can leverage to make something similar that
> cleaner/faster/smarter ( i.e. more awsome ). For example, metadata for
> where the data was was stored in a static file back in days. I reckon
> we can just query a distributed file system to get this same
> information these days, along with all the benefits of replication /
> performance / resilience.

I believe Jeremy Kepner's original implementation was deliberately using
very simple means to implement the transport protocol (all file-based),
to make it easy for users to get off the ground.

In collaboration with the MIT/LL et al. I have been working on a C++ API
specification (http://portals.omg.org/hpec/content/specifications) as
well as library implementation (http://openvsip.org/) incorporating much
of the concepts of pMatlab's distributed arrays (i.e., using "maps" to
define the distribution of blocks of data across many computing nodes).

I was considering working on scripting frontends for that implementation
in Python and Octave, and I have been watching with interest the recent
work on classdef support in Octave, as I was considering using that to
define the high-level API for Maps, Blocks, and Views.
(In fact, I presented the idea to add Python and MATLAB "frontends" to
VSIPL++ backends for a seamless integration into a high-performance
computing platform at an IEEE conference two years ago:
http://ieee-hpec.org/2012/index_htm_files/Seefeld.pdf, and the ideas
were received quite enthusiastically.)

>  All this being said, perhaps someone more familiar with the source
> code could comment on how easy / impossible such a thing would be
> given the current state of Octave? In terms of your direct question of
> how you can build a compiler for Octave, I believe a good place to
> start is to look at the lex/yacc grammar files:
>
> octave/libinterp/parse-tree/lex.ll
> octave/libinterp/parse-tree/oct-parse.yy

I'm honestly not entirely sure I understand the question: What
additional support in terms of language support is needed to provide a
"distributed Octave" ? What I was expecting (though I haven't
investigated much of it yet) is infrastructure work akin to
http://ipython.scipy.org/talks/0903_siamcse09_ipython_dist_bgranger.pdf,
to allow an Octave "frontend" to dispatch computation to multiple "engines".

Would any of this require changes or additions to the language /
interpreter, rather than just additional functions ?

I think a good path towards support for pMatlab in Octave would be

* infrastructure work that allows multiple "octave engines" to be
controlled from one "octave controller" via some scheduler, akin to
ipython's implementation)
* an Octave package that adds an MPI-like API on top of the above
* A new API defining distributed arrays in terms of "maps"

I'd be happy to collaborate on this, in particular as there are
discussions in the OMG HPEC Working Group (http://portals.omg.org/hpec/)
to add new language bindings for Python and MATLAB/Octave to the VSIP
specifications.

    Stefan

-- 

      ...ich hab' noch einen Koffer in Berlin...