[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Octave-bug-tracker] [bug #56874] corr > 1

From: A.R. Burgers
Subject: [Octave-bug-tracker] [bug #56874] corr > 1
Date: Mon, 9 Sep 2019 14:11:56 -0400 (EDT)
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763

Follow-up Comment #1, bug #56874 (project octave):

corr(x,y) - 1 = 2.2204e-16

numerical precision limit is hit in these lines:

    c = cov (x);                                                 
    s = sqrt (diag (c));                                         
    retval = c ./ (s * s'); 

results could maybe be constrained to [-1,1]

    retval = min(max(c ./ (s * s'), -1), 1);

However the result could still be 1-eps. The user of corr needs to anticipate
that the result is subject to numerical errors.


Reply to this item at:


  Message sent via Savannah

reply via email to

[Prev in Thread] Current Thread [Next in Thread]