[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Benchmarking Octave 322
From: |
Jaroslav Hajek |
Subject: |
Re: Benchmarking Octave 322 |
Date: |
Fri, 11 Sep 2009 14:38:28 +0200 |
On Fri, Sep 11, 2009 at 1:48 PM, Francesco Potorti` <address@hidden> wrote:
>>There is the benchmark package at
>>http://octave.sourceforge.net/benchmark/index.html
>>
>>and also some codes all over the web (mostly sadly out-of-date).
>
> In the past I used to maintain a benchmark.m, which is still available
> at <http://fly.isti.cnr.it/pub/software/octave/benchmark.m>. It still
> works on Octave 3.2.2 and claims to compute the performance relative to
> Octave 2 on a Pentium II at 350 MHz with Debian's lapack3 and refblas3
> as of November 2005. Its main asset is that it computes a 95%
> confidence interval for the results, something that as far as I can see
> is missing from your code.
>
You're right, that could be of some use. I just added that capability.
> I even used to publish results sent from people on
> <http://fly.isti.cnr.it/pub/software/octave/bm_results>, back when the
> reference was Sun Sparc, about 100 times slower than the current Pentium
> II reference. I stopped updating the results when compiling Octave
> became so much dependent on libraries and compiling options that
> producing a single number for a given hardware would have been
> misleading, and maintaining a significant list would have been too
> complex.
The mistake is to present it as benchmarking the hardware. It's
benchmarking particular installations. Still, comparisons can be very
useful. A typical usage is to compare a generic distro package with a
custom compilation.
> I often thought about working to merge it with your code, but never
> found the time, and I find myself adding things in front of this on my
> wishlist. So I am writing this mail, in the hope that you, or someone
> else, possiby with my help, can work towards a better and easy-to-use
> benchmark. It would also be nice to have a page on the Octave wiki with
> some performance results.
The package is also low priority for me; besides, it seems almost
nobody uses it. The point was to provide a package compatible with
older Octaves, even Matlab, so that users can benchmark and share
results, and write new benchmarks.
Writing a new benchmark is easy:
% function benchmark_pinv (m,n)
% description:
% Benchmarking the pseudoinverse
%
% arguments:
% m = number of rows
% n = number of columns
%
% results:
% time_rand = time for pinv of a random m*n matrix
% time_diag = time for pinv of a random diagonal m*n matrix
%
function results = benchmark_pinv (m,n)
benchutil_default_arg ('m', 400);
benchutil_default_arg ('n', 400);
benchutil_initialize (mfilename)
a = rand (m, n);
tic; pinv (a); time_rand = toc;
benchutil_set_result ('time_rand')
% Do it in a compatible way.
a = diag (rand (max (m, n), 1));
a = a(1:m, 1:n);
tic; pinv (a); time_diag = toc;
benchutil_set_result ('time_diag')
now, just run it in various ways:
octave:1> benchmark_pinv
Running benchmark: benchmark_pinv
Benchmarking the pseudoinverse
number of rows (m) = 400
number of columns (n) = 400
time for pinv of a random m*n matrix: 0.561242
time for pinv of a random diagonal m*n matrix: 2.00272e-05
octave:2> benchmark_pinv (600, 300)
Running benchmark: benchmark_pinv
Benchmarking the pseudoinverse
number of rows (m) = 600
number of columns (n) = 300
time for pinv of a random m*n matrix: 0.243105
time for pinv of a random diagonal m*n matrix: 1.40667e-05
octave:3> benchutil_average("benchmark_pinv", 10, 600, 300)
Running benchmark: benchmark_pinv
Benchmarking the pseudoinverse
number of rows (m) = 600
number of columns (n) = 300
time for pinv of a random m*n matrix: 0.237562
time for pinv of a random diagonal m*n matrix: 1.40667e-05
....
time for pinv of a random m*n matrix (avg. over 10 runs): 0.240214 +/- 1.595569%
time for pinv of a random diagonal m*n matrix (avg. over 10 runs):
0.000014 +/- 3.743879%
The descriptions are auto-extracted from the help string and progress
is displayed throughout each run, as well as auto-averaging over
multiple runs via benchutil_average.
--
RNDr. Jaroslav Hajek
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz