octave-maintainers
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

precision problems in max()


From: Joanna Cheng
Subject: precision problems in max()
Date: Thu, 9 Feb 2012 07:58:02 +1100

Hi everyone,

As per octave-help, Maynard has pointed out a difference in precision between the original vector and the output of max(). Observe:
========================
octave:1> x = rand(20,1)
x =

   0.415270
   0.732381
   0.696288
   0.118633
   0.606966
   0.215392
   0.976676
   0.322422
   0.274226
   0.442855
   0.483941
   0.950861
   0.067341
   0.403274
   0.161985
   0.853310
   0.124031
   0.959009
   0.197386
   0.507234

octave:2>
octave:2> max(x)
ans =  0.97668

octave:4> format long
octave:5> x
x =
<snip>
   0.9766756054092811
<snip>

octave:6> max(x)
ans =  0.976675605409281
octave:7> format none
octave:8> x
x =

<snip>
 0.976675605409281
<snip>

octave:9> max(x)
ans = 0.976675605409281
octave:10> format short e
octave:11> x
x =
<snip>
  9.7668e-001
<snip>

octave:12> max(x)
ans = 9.7668e-001
octave:13> format short eng
octave:14> x
x =

<snip>
   976.6756e-03
<snip>

octave:15> max(x)
ans =  976.6756e-03
octave:16> format short
octave:17> x
x =

<snip>
   0.976676
<snip>

octave:18> max(x)
ans =  0.97668
========================

So it seems just during "short" and "long" formats that the output is rounded to one less digit in the output on max(). Is this a bug?

The spec of "short" and "long" say 5 and 15 significant digits, respectively. So it's the originals that display an extra digit. The output of max() is conforming to spec.

3.4.2, on win 7, by the way. I have verified the same occurs in 3.2.4 but didn't do extensive testing.

Kind regards,
Joanna

reply via email to

[Prev in Thread] Current Thread [Next in Thread]