[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Octave-bug-tracker] [bug #53456] output_precision: Cannot set precision

From: Oliver Heimlich
Subject: [Octave-bug-tracker] [bug #53456] output_precision: Cannot set precision higher than 16
Date: Sun, 1 Apr 2018 17:45:35 -0400 (EDT)
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0

Follow-up Comment #7, bug #53456 (project octave):

I agree that a limit on output_precision is of little use if it is used for
both single and double precision numbers: If it is forbidden (or discouraged)
to set precision higher than 16 for double, then 7 digits should be handled
equally for single precision. OTOH, if output_precision is used for built-in
floating-point formats only and output precision will be capped anyway
(starting with Octave 4.4), it is of no use to set a value higher than 16.

I assume that the only reason for capping the output precisions of both number
formats has been for Matlab compatibility of the “format long” command.

Thus, I'd like to suggest the following improvement (which would resolve the
regression): Internally, there should be two output precisions, one for single
and one for double. The output_precision function should have the following

 · output_precision () or output_precision ("double") return double output
 · output_precision ("single") return single output precision
 · output_precision (p, "single") set single output precision
 · output_precision (p, "double") set double output precision
 · output_precision (p) set both single and double output precision (like in
Octave 4.2 and earlier)

The number of digits displayed should never be capped (to 7 or 16 digits
respectively). Instead, for Matlab compatibility the “format long” command
should set each internal precision accordingly. “format short” (the
default format) should set both precisions to 5.


Reply to this item at:


  Message sent via/by Savannah

reply via email to

[Prev in Thread] Current Thread [Next in Thread]