Re: Benchmarking Octave 322

[Jaroslav Hajek]

On Fri, Sep 11, 2009 at 1:48 PM, Francesco Potorti` <address@hidden> wrote:
>>There is the benchmark package at
>>http://octave.sourceforge.net/benchmark/index.html
>>
>>and also some codes all over the web (mostly sadly out-of-date).
>
> In the past I used to maintain a benchmark.m, which is still available
> at <http://fly.isti.cnr.it/pub/software/octave/benchmark.m>.  It still
> works on Octave 3.2.2 and claims to compute the performance relative to
> Octave 2 on a Pentium II at 350 MHz with Debian's lapack3 and refblas3
> as of November 2005.  Its main asset is that it computes a 95%
> confidence interval for the results, something that as far as I can see
> is missing from your code.
>

You're right, that could be of some use. I just added that capability.

> I even used to publish results sent from people on
> <http://fly.isti.cnr.it/pub/software/octave/bm_results>, back when the
> reference was Sun Sparc, about 100 times slower than the current Pentium
> II reference.  I stopped updating the results when compiling Octave
> became so much dependent on libraries and compiling options that
> producing a single number for a given hardware would have been
> misleading, and maintaining a significant list would have been too
> complex.

The mistake is to present it as benchmarking the hardware. It's
benchmarking particular installations. Still, comparisons can be very
useful. A typical usage is to compare a generic distro package with a
custom compilation.

> I often thought about working to merge it with your code, but never
> found the time, and I find myself adding things in front of this on my
> wishlist.  So I am writing this mail, in the hope that you, or someone
> else, possiby with my help, can work towards a better and easy-to-use
> benchmark.  It would also be nice to have a page on the Octave wiki with
> some performance results.

The package is also low priority for me; besides, it seems almost
nobody uses it. The point was to provide a package compatible with
older Octaves, even Matlab, so that users can benchmark and share
results, and write new benchmarks.

Writing a new benchmark is easy:

% function benchmark_pinv (m,n)
% description:
% Benchmarking the pseudoinverse
%
% arguments:
% m = number of rows
% n = number of columns
%
% results:
% time_rand = time for pinv of a random m*n matrix
% time_diag = time for pinv of a random diagonal m*n matrix
%

function results = benchmark_pinv (m,n)

  benchutil_default_arg ('m', 400);
  benchutil_default_arg ('n', 400);

  benchutil_initialize (mfilename)

  a = rand (m, n);
  tic; pinv (a); time_rand = toc;
  benchutil_set_result ('time_rand')

  % Do it in a compatible way.
  a = diag (rand (max (m, n), 1));
  a = a(1:m, 1:n);
  tic; pinv (a); time_diag = toc;
  benchutil_set_result ('time_diag')


now, just run it in various ways:

octave:1> benchmark_pinv
Running benchmark: benchmark_pinv
Benchmarking the pseudoinverse

number of rows (m) = 400
number of columns (n) = 400
time for pinv of a random m*n matrix: 0.561242
time for pinv of a random diagonal m*n matrix: 2.00272e-05
octave:2> benchmark_pinv (600, 300)
Running benchmark: benchmark_pinv
Benchmarking the pseudoinverse

number of rows (m) = 600
number of columns (n) = 300
time for pinv of a random m*n matrix: 0.243105
time for pinv of a random diagonal m*n matrix: 1.40667e-05
octave:3> benchutil_average("benchmark_pinv", 10, 600, 300)
Running benchmark: benchmark_pinv
Benchmarking the pseudoinverse

number of rows (m) = 600
number of columns (n) = 300
time for pinv of a random m*n matrix: 0.237562
time for pinv of a random diagonal m*n matrix: 1.40667e-05

....

time for pinv of a random m*n matrix (avg. over 10 runs): 0.240214 +/- 1.595569%
time for pinv of a random diagonal m*n matrix (avg. over 10 runs):
0.000014 +/- 3.743879%

The descriptions are auto-extracted from the help string and progress
is displayed throughout each run, as well as auto-averaging over
multiple runs via benchutil_average.


-- 
RNDr. Jaroslav Hajek
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz