Re: Parallelization with live stream of data

[Olaf Till]

On Fri, Sep 14, 2012 at 05:26:06PM +0300, Jose wrote:
> Hello all.
> 
> I have a live stream of measurements that I need to process with
> octave in real time. The processing basically consists on the
> execution of a script when a new measurement arrives, with the
> measurement as argument. The problem is that the time between
> measurements might be shorter than the time required to execute the
> script, and I would loose data, as the measurements come at constant
> rate.
> 
> One solution that I can think of is parallelization: as I have
> several cores in my machine, I could assign the processing of
> incoming measurements to the free cores (octave instances). Once
> that a measurement has been processed the core is ready to process a
> new measurement.
> 
> To me this sounds like a classical problem that many others might
> have faced before.
> 
> In this sense, I think I cannot use parcellfun, as I need to assign
> cores to measurements dynamically. I have also seen the packages
> multicore and parallel, and the MPITB toolbox. Before diving
> fiercely into code, documentation and so on, I'd appreciate some
> comments/suggestions from experienced users, as I am totally new to
> this world of parallelization.
> 
> BR
> Jose

Although I have not as yet dived in into how parcellfun works, very
probably you can use it. The helptext says the (maximum) recommended
number of processes is the number of cores (or one less, but I would
not take one less), but surely the author has provided scheduling so
each process will take a new job from the pool of unchanged jobs as
soon as the previous is over. Be sure to give it well more jobs in one
call than the number of cores, since there will of corse be no full
usage of all cores when there are less unfinished jobs left then there
are cores.

If I remember right the multicore package uses temporary files for
data exchange and so I'd say don't use it. The parallel package is for
a cluster of computers, so probably nothing for your case.

Olaf

-- 
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc
Description: Digital signature