Re: Shared memory between parallel threads

[Olaf Till]

On Sun, Apr 27, 2014 at 10:21:32PM +0200, c. wrote:
> 
> On 27 Apr 2014, at 20:28, Olaf Till <address@hidden> wrote:
> 
> > On Tue, Apr 22, 2014 at 02:58:15PM +0000, Anton Flugge wrote:
> >> Hi,
> >> 
> >> I am trying to run calls to the kmeans function in parallel. In
> >> principal that works well with parcellfun in the parallel toolbox,
> >> however, the matrix that I want to use kmeans on is quite large (a
> >> couple of GB), and parcellfun creates separate copies of the
> >> workspace/variables for each parallel thread, which means I run out
> >> of memory quicker than I run out of processor cores. Now kmeans only
> >> needs read access, and no write access, to the input data, which
> >> means in principal it shouldn't be a problem to share the same
> >> matrix/memory between all threads, however, I could not find a
> >> solution anywhere on how to do that in Octave... is it possible to
> >> have global variables accessed by multiple parallel threads?
> >> 
> >> Thanks,
> >> Anton
> > 
> > AFAIK Linux processes use copy-on-write, so that if Octave really only
> > reads data it does not generate copies of that data in the child
> > processes. In fact, on my system:
> > 
> > octave:4> a = ones (500000000, 1);
> > 
> > 'a' occupies now half of my systems RAM and 1/6 of total systems
> > memory including swap space. Nevertheless
> > 
> > octave:5> r = parcellfun (10, @ (x) sum (a), cell (10, 1))
> > parcellfun: 10/10 jobs done
> > r =
> > 
> >   500000000
> >   500000000
> >   500000000
> >   500000000
> >   500000000
> >   500000000
> >   500000000
> >   500000000
> >   500000000
> >   500000000
> > 
> > with 10 child processes works, although
> > 
> > octave:6> repmat (a, 1, 10);
> > error: out of memory or dimension too large for Octave's index type
> > 
> > indicates that 10 copies of 'a' would be too much, as expected.
> > 
> > Olaf
> > 
> > -- 
> 
> Olaf, 
> 
> I think you might be misinterpreting the results of your example,
> look at this:
> 
> >> a = ones (500000000, 1);
> >> for ii = 1:10
> eval (sprintf ("b_%d = a; b_%d(end) = 2;", ii, ii))
> endfor
> 
> >> clear all
> >> a = ones (500000000, 1);
> >> b = repmat (a, 1, 10);
> error: out of memory or dimension too large for Octave's index type
> error: called from:
> error:   /opt/local/share/octave/3.8.0/m/general/repmat.m at line 105, column 
> 9
> >> 
> 
> the reason for the error is not that the memory has been filled
> but that you are trying to create an array that is larger than what
> can be indexed by and index of type "int".

You are right, the example with repmat was bad. But it was only meant
as a demonstration of what is actually easily calculated, that my
systems total memory would not be enough for 10 copies (seemingly your
system has more total memory).

Olaf

-- 
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc
Description: Digital signature