|
From: | Dan Sebald |
Subject: | [Octave-bug-tracker] [bug #53683] numerical output format for float values with magnitude greater than precision is wrong |
Date: | Tue, 17 Apr 2018 20:46:14 -0400 (EDT) |
User-agent: | Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:55.0) Gecko/20100101 Firefox/55.0 |
URL: <http://savannah.gnu.org/bugs/?53683> Summary: numerical output format for float values with magnitude greater than precision is wrong Project: GNU Octave Submitted by: sebald Submitted on: Wed 18 Apr 2018 12:46:11 AM UTC Category: Interpreter Severity: 3 - Normal Priority: 5 - Normal Item Group: Incorrect Result Status: None Assigned to: None Originator Name: Originator Email: Open/Closed: Open Discussion Lock: Any Release: dev Operating System: Any _______________________________________________________ Details: It caught my attention that the output format is displaying too much precision for numbers that have magnitude greater than the number of digits of precision, stated imprecisely. That is, notice the progression: octave:1> for i=-2:8 > 10^i / 3 > endfor ans = 0.0033333 ans = 0.033333 ans = 0.33333 ans = 3.3333 ans = 33.333 ans = 333.33 ans = 3333.3 ans = 33333.33333 ans = 333333.33333 ans = 3333333.33333 ans = 33333333.33333 that once the left digits hits 5 the right digits jumps back up to five. Here are some related bug reports, but they aren't quite the same issue: https://savannah.gnu.org/bugs/?func=detailitem&item_id=53456 https://savannah.gnu.org/bugs/?func=detailitem&item_id=41359 _______________________________________________________ Reply to this item at: <http://savannah.gnu.org/bugs/?53683> _______________________________________________ Message sent via/by Savannah http://savannah.gnu.org/
[Prev in Thread] | Current Thread | [Next in Thread] |