Re: Octave suddenly slow

[David Bateman]

Rob Mahurin wrote:
> Adding 10^4 empty files to a directory makes octave open a couple 
> seconds slower; adding 10^5 empty files makes it take eight minutes.  I 
> have heard of bugs in filesystems that cause this problem, but that 
> doesn't seem to be the case since "ls | wc" is fast in the shell.
>
> The filesystem bug I remember (or maybe an ls bug? or both?) was an 
> inefficient sort in the code that lists the names of files in a 
> directory.  Poking through the code, I don't see anything like that.
>
> Hmmmm ... dir_entry::read() seems to guess that it will live in a small 
> directory.  For the case I've outlined above, the attached patch should 
> reduce the number of calls to Array::resize() from ~1000 to ~10.  Not 
> tested, though.


The conclusive manner to check would be with gprof. But that requires a 
 static build with

CFLAGS="-g -pg -O2"
CXXFLAGS="-g -pg -O2"
FFLAGS="-g -pg -O2"
LDFLAGS="-g -pg -O2"
./configure --enable-static --disable-dl --disable-shared
I previously noticed that file_ops::tilde_expand consumed a lot of time 
in gprof at startup.

http://www.nabble.com/forum/ViewPost.jtp?post=12531612&framed=y

So perhaps the attached change to file_ops::tilde_expand might also 
help. For me with this change I see with the following with the command

time octave --eval quit

the wall times are

                Normal   1e4 files    1.1e5 files
With Patch       0.760    1.059         8.414
Without Patch    1.116    2.801        20.786

So for me the speeds are significantly faster with my patch (note this 
is without any packages installed). I don't think your patch is quite 
right as you start with a zero sized array with len not equal to zero. I 
prefer a patch like

--- a/liboctave/dir-ops.cc
+++ b/liboctave/dir-ops.cc
@@ -87,7 +87,7 @@
            {
              if (count >= len)
                {
-                 len += grow_size;
+                 len = (len == 0 ? grow_size : len * 2);
                  dirlist.resize (len);
                }

Adding this and my previous patch gives a startup time of

                Normal   1e4 files    1.1e5 files
With Patches     0.831    1.025         3.516
Without Patches  1.116    2.801        20.786

So the two patches together give a significant improvement. John added a 
slighly different change to dir_entry::read just now so I only committed 
my change to tilde_expand..

Regards
David


--
David Bateman                                address@hidden
35 rue Gambetta                              +33 1 46 04 02 18 (Home)
92100 Boulogne-Billancourt FRANCE            +33 6 72 01 06 33 (Mob)

# HG changeset patch
# User David Bateman <address@hidden>
# Date 1226971456 -3600
# Node ID a0b8453624afeb692861a40c276ab38d75a9a3f8
# Parent  d793ed7122be8e01b4f521e147c5337ef11ab9fa
Fast return case for file_ops::tilde_expand. Expontential dirlist resize in 
dir_entry::read

diff --git a/liboctave/ChangeLog b/liboctave/ChangeLog
--- a/liboctave/ChangeLog
+++ b/liboctave/ChangeLog
@@ -1,3 +1,14 @@
+2008-11-18  Robert S. Mahurin  <address@hidden>
+
+       * dir-ops.cc (string_vector dir_entry::read (void)): Expontential
+       resize of dirlist to avoid too many memory reallocations.
+
+2008-11-18  David Bateman  <address@hidden>
+
+       * file-ops.cc (std::string file_ops::tilde_expand (const 
+       std::string&)): Check if the string contains a tilde and fast 
+       return if not.
+
 2008-11-14  David Bateman  <address@hidden>
 
        * Array2.h (Array2<T> Array2<T>::index): Correct use of
diff --git a/liboctave/dir-ops.cc b/liboctave/dir-ops.cc
--- a/liboctave/dir-ops.cc
+++ b/liboctave/dir-ops.cc
@@ -87,7 +87,7 @@
            {
              if (count >= len)
                {
-                 len += grow_size;
+                 len = (len == 0 ? grow_size : len * 2);
                  dirlist.resize (len);
                }
 
diff --git a/liboctave/file-ops.cc b/liboctave/file-ops.cc
--- a/liboctave/file-ops.cc
+++ b/liboctave/file-ops.cc
@@ -768,53 +768,58 @@
 std::string
 file_ops::tilde_expand (const std::string& name)
 {
-  std::string result;
+  if (name.find ('~') == std::string::npos)
+    return name;
+  else
+    {
+      std::string result;
 
-  size_t name_len = name.length ();
+      size_t name_len = name.length ();
 
-  // Scan through S expanding tildes as we come to them.
+      // Scan through S expanding tildes as we come to them.
 
-  size_t pos = 0;
+      size_t pos = 0;
 
-  while (1)
-    {
-      if (pos > name_len)
-       break;
+      while (1)
+       {
+         if (pos > name_len)
+           break;
 
-      size_t len;
+         size_t len;
 
-      // Make START point to the tilde which starts the expansion.
+         // Make START point to the tilde which starts the expansion.
 
-      size_t start = tilde_find_prefix (name.substr (pos), len);
+         size_t start = tilde_find_prefix (name.substr (pos), len);
 
-      result.append (name.substr (pos, start));
+         result.append (name.substr (pos, start));
 
-      // Advance STRING to the starting tilde.
+         // Advance STRING to the starting tilde.
 
-      pos += start;
+         pos += start;
 
-      // Make FINI be the index of one after the last character of the
-      // username.
+         // Make FINI be the index of one after the last character of the
+         // username.
 
-      size_t fini = tilde_find_suffix (name.substr (pos));
+         size_t fini = tilde_find_suffix (name.substr (pos));
 
-      // If both START and FINI are zero, we are all done.
+         // If both START and FINI are zero, we are all done.
 
-      if (! (start || fini))
-       break;
+         if (! (start || fini))
+           break;
 
-      // Expand the entire tilde word, and copy it into RESULT.
+         // Expand the entire tilde word, and copy it into RESULT.
 
-      std::string tilde_word = name.substr (pos, fini);
+         std::string tilde_word = name.substr (pos, fini);
 
-      pos += fini;
+         pos += fini;
 
-      std::string expansion = tilde_expand_word (tilde_word);
+         std::string expansion = tilde_expand_word (tilde_word);
 
-      result.append (expansion);
+         result.append (expansion);
+       }
+
+      return result;
     }
-
-  return result;
 }
 
 // A vector version of the above.