Re: Aw: Re: memory leak in regexprep ? (windows/3.7.2+/VS)

[Sergei Steshenko]

----- Original Message -----
> From: Tatsuro MATSUOKA <address@hidden>
> To: Christoph Ellenberger <address@hidden>
> Cc: address@hidden
> Sent: Thursday, March 28, 2013 4:07 AM
> Subject: Re: Aw: Re: memory leak in regexprep ? (windows/3.7.2+/VS)
> 
> --- On Wed, 2013/3/27, Christoph Ellenberger wrote:
>>  > --- On Wed, 2013/3/27, Tatsuro MATSUOKA wrote:
>>  > > --- On Wed, 2013/3/27, Tatsuro MATSUOKA wrote:
>>  > > > --- On Tue, 2013/3/26, Christoph Ellenberger  wrote:
>>  > > > > I am experiencing some strange behavior and wanted to 
> check if it is  a bug or me....
>>  > > > > It happens when I try to read in and manipulating some 
> ascii data files. It only happens with the VS version of octave. (tested with 
> several gcc versions on windows and didn't showed up).
>>  > > > > Ok here is some example code to reproduce:
>>  > > > > First generating an ascii file:
>>  > > > > s="0123456789";
>>  > > > > for i=1:21, s=cstrcat(s,s);,endfor
>>  > > > > save TestData.dat s
>>  > > > >  
>>  > > > > Then readin and manipulate:
>>  > > > > fid = fopen("TestData.dat","r");
>>  > > > > s = fscanf(fid,'%c');
>>  > > > > fclose(fid);
>>  > > > > s = regexprep(s,'0','2');
>>  > > > >  
>>  > > > > Clear all and repeat:
>>  > > > > clear all
>>  > > > > 
>>  > > > > fid = fopen("TestData.dat","r");
>>  > > > > s = fscanf(fid,'%c');
>>  > > > > fclose(fid);
>>  > > > > s = regexprep(s,'0','2');
>>  > > > >  
>>  > > > > Depending on datasize and how many replacement were 
> performed it either fails on the new readin or regexprep with a: "error: 
> out of memory or dimension too large for Octave's index type"
>>  > > > > Any help is appreciated
>>  > > > > Christoph
>>  > > > 
>>  > > > I have tested on octave-3.6.2 VS, 3.6.2 MinGW, 3.7.2+ VS and 
> 3.6.2 cygwin
>>  > > > (win 7 64 bit Home premium, 4GB memory)
>>  > > > 
>>  > > > 
>>  > > > 1. octave-3.6.2 VS and 3.6.2 
> MinGW*****************************
>>  > > > >> fid = 
> fopen("TestData.dat","r");
>>  > > > >> s = fscanf(fid,'%c');
>>  > > > warning: range error for conversion to character value
>>  > > > >> fclose(fid);
>>  > > > >> s = regexprep(s,'0','2');
>>  > > > >> clear all
>>  > > > >> fid = 
> fopen("TestData.dat","r");
>>  > > > >> s = fscanf(fid,'%c');
>>  > > > error: memory exhausted or requested size too large for 
> range of Octave's index type -- trying to return to prompt
>>  > > > 
> ********************************************************************
>>  > > > 
>>  > > > 
>>  > > > 2. octave-3.7.2+ VS*****************************
>>  > > > >> fid = 
> fopen("TestData.dat","r");
>>  > > > >> s = fscanf(fid,'%c');
>>  > > > warning: range error for conversion to character value
>>  > > > >> fclose(fid);
>>  > > > >> s = regexprep(s,'0','2');
>>  > > > >> clear all
>>  > > > >> fid = 
> fopen("TestData.dat","r");
>>  > > > >> s = fscanf(fid,'%c');
>>  > > > error: out of memory or dimension too large for Octave's 
> index type
>>  > > > 
> ********************************************************************
>>  > > > 
>>  > > > 3.Octave-3.6.2 
> Cygwin**********************************************
>>  > > > >> fid = 
> fopen("TestData.dat","r");
>>  > > > >> s = fscanf(fid,'%c');
>>  > > > >> fclose(fid);
>>  > > > >> s = regexprep(s,'0','2');
>>  > > > >> clear all
>>  > > > >> fid = 
> fopen("TestData.dat","r");
>>  > > > >> s = fscanf(fid,'%c');
>>  > > > >> fclose(fid);
>>  > > > >> s = regexprep(s,'0','2');
>>  > > > 
> ********************************************************************
>>  > > > 
>>  > > > For me, both VS and MinGW version caused problem. Cygwin 
> version seemed to work correctly.
>>  > > > 
>>  > > > Regards
>>  > > > 
>>  > > > Tatsuro
>>  > > 
>>  > > 
>>  > > I have also tested on 
>>  > > octave-3.2.4 and 3.4.3 MinGW version.
>>  > > 
>>  > > 4. octave-3.2.4 and 3.4.3 MinGW *****************************
>>  > > >> fid = fopen("TestData.dat","r");
>>  > > >> s = fscanf(fid,'%c');
>>  > > warning: range error for conversion to character value
>>  > > >> fclose(fid);
>>  > > >> s = regexprep(s,'0','2');
>>  > > >> clear all
>>  > > >> fid = fopen("TestData.dat","r");
>>  > > >> s = fscanf(fid,'%c');
>>  > > warning: range error for conversion to character value
>>  > > >> fclose(fid);
>>  > > >> s = regexprep(s,'0','2');
>>  > > **************************************************
>>  > > 
>>  > > The problem seemed to be happened on octave-3.6 or later on 
> native windows.
>>  > > 
>>  > > Regards
>>  > > 
>>  > > Tatsuro 
>>  > 
>>  > Sorry for successive post.
>>  > 
>>  > I have tested the following  :
>>  > 
>>  > %*********************************
>>  > s="0123456789";
>>  > for i=1:21, s=cstrcat(s,s);,endfor
>>  > save TestData.dat s
>>  > load TestData.dat
>>  > s = regexprep(s,'0','2');
>>  > clear all
>>  > load TestData.dat
>>  > s = regexprep(s,'0','2');
>>  > %*********************************
>>  > 
>>  > The above worked correctly on octave-3.2.4 (MinGW), 3.4.3 (MinGW), 
> 3.6.2 (MinGW,VS, Cygwin) and 3.7.2+ (VS).
>>  > 
>>  > I think that the problem does not rely on regexprep but on fscanf for 
> large data.
>>  > 
>>  > Regards
>>  > 
>>  > Tatsuro
>>  > 
>>  Thank you very much for the fast confirmation of the same problem. Actually 
> it must be a strange combination of both I tested:
>> 
>>  octave-3.7.2+ VS*****************************
>>  fid = fopen("TestData.dat","r");
>>  s = fscanf(fid,'%c');
>>  warning: range error for conversion to character value
>>  fclose(fid);
>>  clear all
>>  fid = fopen("TestData.dat","r");
>>  s = fscanf(fid,'%c');
>>  warning: range error for conversion to character value
>>  fclose(fid);
>>  clear all
>>  fid = fopen("TestData.dat","r");
>>  s = fscanf(fid,'%c');
>>  warning: range error for conversion to character value
>>  fclose(fid);
>>  clear all
>>  ********************************************************************
>> 
>>  So it can't be fscanf only. On the other hand I tested as well (because 
> your last example the second regexprep does nothing, as all 0 were already 
> exchanged)
>> 
>>  octave-3.7.2+ VS*****************************
>>  fid = fopen("TestData.dat","r");
>>  s = fscanf(fid,'%c');
>>  warning: range error for conversion to character value
>>  fclose(fid);
>>  s = regexprep(s,'0','2');
>>  s = regexprep(s,'1','2');
>>  s = regexprep(s,'2','3');
>>  error: out of memory or dimension too large for Octave's index type
>>  ********************************************************************
>> 
>>  So it clearly also depends on how many replacements are performed within 
> the regexprep. (same fails as well in 3.6.0_gcc4.6.2 and 3.2.4_gcc4.4.0)
>>  As something like this always works:
>> 
>>  octave-3.7.2+ VS*****************************
>>  fid = fopen("TestData.dat","r");
>>  s = fscanf(fid,'%c');
>>  warning: range error for conversion to character value
>>  fclose(fid);
>>  s = ctsrcat(s,"aaaaaaaaa", s);
>>  s = regexprep(s,'a','2');
>>  clear all
>>  fid = fopen("TestData.dat","r");
>>  s = fscanf(fid,'%c');
>>  warning: range error for conversion to character value
>>  fclose(fid);
>>  clear all
>>  ********************************************************************
>> 
>>  Kind regards
>>  Christoph
> 
> You are right.
> #**************************************
> s="0123456789";
> for i=1:21, s=cstrcat(s,s);,endfor
> 
> s = regexprep(s,'0','2'); 
> s = regexprep(s,'1','2'); 
> s = regexprep(s,'2','3'); 
> #**************************************
> 
> The above caused 'out of memory ...' or 'memory exhausted ...' 
> on octave 3.7.2+(VS),
> 3.6.2(VS, MinGW), 3.4.3 (MinGW) and 3.2.4 (MinGW).
> 
> But for cygwin version octave(3.6.2), the above test did not cause memory 
> error.
> 
> I think that it is better that you resist this strange behavior of regexprep 
> as 
> a bug to the bug tracker.
> 
> Regards
> 
> Tatsuro
> _______________________________________________


I've tried the above code on self-built octave-3.6.4 under Linux.

Memory consumption constantly grows, I killed the session when memory 
consumption reached 2.5G.

I think it's a bug, and since I can reproduce the failure under Linux, the bug 
is not OS-specific.

Regards,
  Sergei.