[gnuastro-devel] [task #14299] Not writing NaN in ASCII (plain or FITS) tables

[Mohammad Akhlaghi]

URL:
  <http://savannah.gnu.org/task/?14299>

                 Summary: Not writing NaN in ASCII (plain or FITS) tables
                 Project: GNU Astronomy Utilities
            Submitted by: makhlaghi
            Submitted on: Wed 04 Jan 2017 09:01:24 PM JST
         Should Start On: Wed 04 Jan 2017 12:00:00 AM JST
   Should be Finished on: Wed 04 Jan 2017 12:00:00 AM JST
                Category: Libraries
                Priority: 5 - Normal
              Item Group: Enhancement
                  Status: Postponed
                 Privacy: Public
        Percent Complete: 0%
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any
                  Effort: 0.00

    _______________________________________________________

Details:

While I was working on reading and writing of FITS ASCII tables, I got in
touch with William Pence (author of CFITSIO) for some issues I had and
fortunately with his guidance they were all fixed.

One issue that he raised was that of printing NaN values in an ASCII table and
that some compilers cannot correctly read ASCII printd `nan' (or `NaN') as a
floating point number (IEEE 754's NaN). So he suggested using some extreme
floating point number instead and also noted that CFITSIO comes with
`FLOATNULLVALUE' and `DOUBLENULLVALUE' for this purpose.

So I am opening this task as a discussion (and hopefully later,
implementation) to account for this issue. One way that comes to my mind now
is that we can add an argument/flag to the table producing library, that when
the table is in text format (FITS ASCII or plain text), it doesn't print NaN
values, but uses extreme numbers. Since those numbers will be included in the
column's meta-data (both in FITS ASCII and also plain text, see below), the
readers won't have any problem.

I had a look at the portability issues with strtod
<https://www.gnu.org/software/gnulib/manual/html_node/strtod.html> in Gnulib
which confirms William Pence's point and the many issues different
systems/compilers have with reading floating points from printed characters.
So in order to avoid all these problems in Gnuastro (when compiled on those
systems), Gnulib's `strtod' module is now also imported into Gnuastro during
bootstrapping.

About FITS ASCII tables, I don't think there will be a problem, because
according to the FITS standrard, the `TNULLn' keywords for FITS ASCII are
strings, not numbers (and all the columns have a fixed width), so the column
string can be compared with the string given to the `TNULLn' keyword *before*
it is read as a number, so the string can be anything (including `nan'). 

So the problem is mainly in plain text tables, and thanks to Gnulib, Gnuastro
won't have many of the problems other programs might have. But still, people
generate plain text for portability, not to use with one particular program
(like Gnuastro), so we need to give them the ability to account for this
deficiancy in other software.

Until we find a solution, people who can't read `nan' values can do use tools
like SED <https://www.gnu.org/software/sed/> to chance all occurances of `nan'
in the plain text tables to some out-of-range number they like.

Just as a side note, Gnuastro's plain text table format is now fully described
in the Gnuastro text table format
<https://gitlab.com/makhlaghi/gnuastro/blob/datastruct/doc/gnuastro.texi#L5150>
of the book. Just note that since this version of the book is not yet
released, this link points to a line number in the source, which might change
with updates to the book. So search for the section name if it has changed.




    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/task/?14299>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/