Re: Slowness in function 'open'

[John W. Eaton]

On 22-Jun-2007, Jordi Gutierrez Hermoso wrote:

| I can't quite understand what they're doing at first glance. But if
| the problem is simply to read a matrix ASCII file, looks like there
| might be a better way to do it.

Before comparing speed, let's make sure we are talking about the same
function and that they handle the same set of requirements.  I'm
assuming we are discussing the function that reads a 2-d matrix from a
file  

The function that reads matrices must

  1. ignore comments (beginning with '%' or '#') anywhere on the line,
     not just when those characters appear in the first column

  2. ignore commas that separate numeric entries

  3. ensure that all rows have the same number of elements and produce an
     error otherwise

  4. handle reading Inf and NaN values even when the underlying C/C++
     library does not

The last point could account for some of the slowness as well.  The
code for that is in liboctave/lo-utils.cc (start with the function
lo_read_double).  I would certainly consider patches to improve that
code as well.

| Octave's algorithm does seem a bit strange, though. Why is it doing
| two passes in the case that the file has no header?

The idea was to first determine the number of elements to avoid having
to resize the result matrix.

| One pass should suffice. You can read all the data into an
| std::list, which is what I do, and there are a few simple checks to
| make sure that the formatting of the ASCII matrix is correct.

How much extra memory is required to manage each element of a
std::list object?  Remember that back in the dark ages when some parts
of Octave were first written very few people had machines with more
than a few megabytes of memory...

| This could be problematic for N-d arrays,
| but at least for plain matrices works fine.

There is no need to handle N-d arrays since there is no way to
determine the 3rd and higher dimensions from the data as long as we
only have two-dimensional files (rows and columns of data).

| I try to always avoid macros, for instance. To
| paraphrase Stroustrup, macros are a deficiency in the code, the coder,
| or the coding language. ;-)

I guess maybe C/C++ is the wrong language for implementing Octave
then.  Either that, or you are saying that I'm deficient.  I found no
great way to avoid them in many parts of Octave short of duplicating a
lot of code by cut and paste (and doesn't using a macro seem better
than doing that?).  Of course that was before C++ had templates, so maybe
some of the macro abuse could be eliminated now, but I doubt that all
of it could go away without actually making things worse.

jwe