[gnuastro-commits] master 5128a3f: FITS (program and library): Parsing FITS date formatted strings

[Mohammad Akhlaghi]

branch: master
commit 5128a3f3a886126e33d0cdb59961528bc4f598af
Author: Mohammad Akhlaghi <address@hidden>
Commit: Mohammad Akhlaghi <address@hidden>

    FITS (program and library): Parsing FITS date formatted strings
    
    Two new functions were added in the FITS library to allow parsing of FITS
    date formatted strings. `gal_fits_key_date_to_struct_tm' will convert it to
    C's broken-down time sturcture and `gal_fits_key_date_to_seconds' will
    convert it to the Unix epoch time. The latter is now called by the Fits
    program through the new `--datetosec' option to enable easy access to this
    feature which is very useful in using the date-related FITS keywords.
---
 NEWS                |  21 +++++++---
 bin/fits/args.h     |  13 ++++++
 bin/fits/keywords.c |  41 +++++++++++++++++++
 bin/fits/main.h     |   1 +
 bin/fits/ui.c       |  10 ++++-
 bin/fits/ui.h       |   3 +-
 bootstrap.conf      |   4 ++
 doc/gnuastro.texi   |  71 +++++++++++++++++++++++++++++++++
 doc/plotsrc/README  |  19 ++++++++-
 lib/fits.c          | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/gnuastro/fits.h |   8 ++++
 11 files changed, 294 insertions(+), 9 deletions(-)

diff --git a/NEWS b/NEWS
index d40a2e0..8456a8f 100644
--- a/NEWS
+++ b/NEWS
@@ -17,7 +17,7 @@ GNU Astronomy Utilities NEWS                          -*- 
outline -*-
      operand into a file. They can be very handy in debugging/understanding
      an Arithmetic command (especially as it gets complicated), or to
      produce multiple files/extensions with a single call to Arithmetic.
-   - Four new operators have beed added to allow coadding multiple datasets
+   - Four new operators have been added to allow coadding multiple datasets
      into one using sigma-clipping: `sigclip-number', `sigclip-mean',
      `sigclip-median', and `sigclip-std'. These are very useful when
      several inputs have strong outliers that affect the median, or the
@@ -28,7 +28,7 @@ GNU Astronomy Utilities NEWS                          -*- 
outline -*-
      `sigclip-median', `sigclip-mean', `sigclip-std'.
    --wcsfile and --wcshdu: these two options can be used to specify a
      different file for reading the WCS of the output. This is useful when
-     the default (theh WCS of the first dataset that is read) is not the
+     the default (the WCS of the first dataset that is read) is not the
      required one.
    --interpmetric: new option that is necessary with the
      `interpolate-medianngb' operator. For more, see the description of
@@ -39,10 +39,17 @@ GNU Astronomy Utilities NEWS                          -*- 
outline -*-
      "title" is composed of two keyword records: a blank one, followed by
      another starting with `/' and ending in any string given to this
      option. Classifying the keywords into contextually similar groups
-     greatly helps in visual inspection and is ecouraged.
+     greatly helps in visual inspection and is encouraged.
    - Calculate and write `CHECKSUM' and `DATASUM' integrity keywords into
      the specified header using `--write=checksum' (for both) or
      `--write=datasum' (only for `DATASUM').
+   --datetosec: Convert the FITS date format (old or new) to number of
+     seconds since since the Unix epoch time (1970-01-01,00:00:00). The
+     FITS date format (for example `YYYY-MM-DDThh:mm:ss') is hard to use
+     for automatic processing (requires calendar peculiarities like number
+     of days in each month, or leap years and etc). With this option a
+     single integer is returned that can be used for example to sort FITS
+     files by date keywords without worrying about calendar peculiarities.
    --verify: confirm if the `DATASUM' and `CHECKSUM' keyword values agree
      with the specified HDU's content and/or data.
    --copykeys: Copy several keyword records (in a given range) from one
@@ -54,7 +61,7 @@ GNU Astronomy Utilities NEWS                          -*- 
outline -*-
      any of the columns of the second using the special `_all' name of
      `--outcols'. For example the output of `--outcols=a_all,b5' will
      contain all the columns from the first input and the 5th column of the
-     second input. This greatly simplifies the mergining of different table
+     second input. This greatly simplifies the merging of different table
      columns into one.
    --coord: manually specify coordinates to match on the
      command-line. Until now, if you only wanted to make check a specific
@@ -93,19 +100,21 @@ GNU Astronomy Utilities NEWS                          -*- 
outline -*-
    --range: Limit the output rows to those with a value within the given
      numeric range with this format: `--range=COL,low,high'. This is very
      useful when only certain rows of the input are required not the
-     output. The advantage over pipeing to AWK is that you can save the
+     output. The advantage over piping to AWK is that you can save the
      output directly to FITS (preserving the metadata). See the book for
      more.
    --sort: Sort the output rows based on the value of the given column in
      ascending order.
    --descending: When called with `--sort', will arrange the output rows in
-     decending order.
+     descending order.
 
   Library:
     GAL_BLANK_LONG: new macro for the `long' type blank value.
     GAL_BLANK_ULONG: new macro for the `unsigned long' type blank value.
     gal_blank_number: Return the number of blank elements in a dataset.
     gal_dimension_dist_radial: Radial distance between two coordinates.
+    gal_fits_key_date_to_struct_tm: FITS date format to C broken-down time.
+    gal_fits_key_date_to_seconds: FITS date format to Unix epoch time.
     gal_qsort_index_single_TYPE_i: Set of functions to sort indexs ascending.
     gal_qsort_index_single_TYPE_d: Set of functions to sort indexs descending.
     gal_statistics_outlier_cumulative: Uses flatness of the cumulative
diff --git a/bin/fits/args.h b/bin/fits/args.h
index 281a65b..2a1360b 100644
--- a/bin/fits/args.h
+++ b/bin/fits/args.h
@@ -240,6 +240,19 @@ struct argp_option program_options[] =
       GAL_OPTIONS_NOT_MANDATORY,
       GAL_OPTIONS_NOT_SET
     },
+    {
+      "datetosec",
+      UI_KEY_DATETOSEC,
+      "STR",
+      0,
+      "FITS date to sec from 1970/01/01T00:00:00",
+      UI_GROUP_KEYWORD,
+      &p->datetosec,
+      GAL_TYPE_STRING,
+      GAL_OPTIONS_RANGE_ANY,
+      GAL_OPTIONS_NOT_MANDATORY,
+      GAL_OPTIONS_NOT_SET
+    },
 
 
 
diff --git a/bin/fits/keywords.c b/bin/fits/keywords.c
index c7e6fe2..2220113 100644
--- a/bin/fits/keywords.c
+++ b/bin/fits/keywords.c
@@ -397,6 +397,39 @@ keywords_copykeys(struct fitsparams *p, char *inkeys, 
size_t numinkeys)
 
 
 
+static void
+keywords_date_to_seconds(struct fitsparams *p, fitsfile *fptr)
+{
+  int status=0;
+  double subsec;
+  size_t seconds;
+  char *subsecstr;
+  char fitsdate[FLEN_KEYWORD];
+
+  /* Read the requested FITS keyword. */
+  fits_read_key(fptr, TSTRING, p->datetosec, &fitsdate, NULL, &status);
+
+  /* Return the number of seconds (and subseconds) that it corresponds
+     to. */
+  seconds=gal_fits_key_date_to_seconds(fitsdate, &subsecstr, &subsec);
+
+  /* Print the result (for the sub-seconds, print everything after the */
+  if( !p->cp.quiet )
+    {
+      printf("%s (hdu %s), key `%s': %s\n", p->filename, p->cp.hdu,
+             p->datetosec, fitsdate);
+      printf("Seconds since 1970/01/01 (00:00:00): %zu%s\n\n", seconds,
+             subsecstr);
+      printf("(To suppress verbose output, run with `-q')\n");
+    }
+  else
+    printf("%ld%s\n", seconds, subsecstr);
+}
+
+
+
+
+
 
 
 
@@ -554,6 +587,14 @@ keywords(struct fitsparams *p)
     }
 
 
+  /* Convert the FITS date string into seconds. */
+  if(p->datetosec)
+    {
+      keywords_open(p, &fptr, READONLY);
+      keywords_date_to_seconds(p, fptr);
+    }
+
+
   /* Close the FITS file */
   if(fits_close_file(fptr, &status))
     gal_fits_io_error(status, NULL);
diff --git a/bin/fits/main.h b/bin/fits/main.h
index 6d27a8c..460a067 100644
--- a/bin/fits/main.h
+++ b/bin/fits/main.h
@@ -72,6 +72,7 @@ struct fitsparams
   gal_list_str_t *comment;     /* COMMENT value.                        */
   uint8_t         *verify;     /* Verify the CHECKSUM and DATASUM keys. */
   char          *copykeys;     /* Range of keywords to copy in output.  */
+  char         *datetosec;     /* Convert FITS date to seconds.         */
   uint8_t     quitonerror;     /* Quit if an error occurs.              */
 
   /* Internal: */
diff --git a/bin/fits/ui.c b/bin/fits/ui.c
index fdbe637..c0c14d7 100644
--- a/bin/fits/ui.c
+++ b/bin/fits/ui.c
@@ -319,7 +319,7 @@ ui_read_check_only_options(struct fitsparams *p)
      mode flag to keyword-mode. */
   if( p->date || p->comment || p->history || p->asis || p->delete
       || p->rename || p->update || p->write || p->verify || p->printallkeys
-      || p->copykeys )
+      || p->copykeys || p->datetosec )
     {
       /* Set the mode. */
       p->mode=FITS_MODE_KEY;
@@ -360,6 +360,14 @@ ui_read_check_only_options(struct fitsparams *p)
                                                    "_ext.fits");
     }
 
+  /* Currently `datetosec' must be called alone. */
+  if( p->datetosec
+      && (p->date || p->comment || p->history || p->asis || p->delete
+          || p->rename || p->update || p->write || p->verify
+          || p->printallkeys || p->copykeys || p->mode==FITS_MODE_HDU) )
+    error(EXIT_FAILURE, 0, "`--datetosec' cannot currently be called with "
+          "any other option");
+
   /* If no options are given, go into HDU mode, which will print the HDU
      information when nothing is asked. */
   if(p->mode==FITS_MODE_INVALID)
diff --git a/bin/fits/ui.h b/bin/fits/ui.h
index 582a03c..229cc61 100644
--- a/bin/fits/ui.h
+++ b/bin/fits/ui.h
@@ -43,7 +43,7 @@ enum program_args_groups
 
 /* Available letters for short options:
 
-   b e f g i j l m s x y z
+   b e f g i j l m x y z
    A B E G J L O W X Y
  */
 enum option_keys_enum
@@ -64,6 +64,7 @@ enum option_keys_enum
   UI_KEY_DATE         = 't',
   UI_KEY_VERIFY       = 'v',
   UI_KEY_QUITONERROR  = 'Q',
+  UI_KEY_DATETOSEC    = 's',
 
 
   /* Only with long version (start with a value 1000, the rest will be set
diff --git a/bootstrap.conf b/bootstrap.conf
index 82baa12..8814be2 100644
--- a/bootstrap.conf
+++ b/bootstrap.conf
@@ -215,18 +215,22 @@ gnulib_modules="
     func
     math
     argp
+    time
     fcntl
     regex
     error
     nproc
     stdint
     strtod
+    mktime
     getline
     strcase
     gendocs
     gpl-3.0
     mbstok_r
     inttypes
+    sys_time
+    strptime
     faccessat
     system-posix
     secure_getenv
diff --git a/doc/gnuastro.texi b/doc/gnuastro.texi
index a01efc9..d86077d 100644
--- a/doc/gnuastro.texi
+++ b/doc/gnuastro.texi
@@ -10485,6 +10485,31 @@ Quit if any of the operations above are not 
successful. By default if
 an error occurs, Fits will warn the user of the faulty keyword and
 continue with the rest of actions.
 
address@hidden -s STR
address@hidden --datetosec STR
address@hidden Unix epoch time
address@hidden Epoch time, Unix
+Interpret the value of the given keyword in the FITS date format (most
+generally: @code{YYYY-MM-DDThh:mm:ss.ddd...}) and return the corresponding
+Unix epoch time (number of seconds that have passed since 00:00:00
+Thursday, January 1st, 1970). The @code{Thh:mm:ss.ddd...} section
+(specifing the time of day), and also the @code{.ddd...} (specifying the
+fraction of a second) are optional. The value to this option must be the
+FITS keyword name that contains the requested date, for example
address@hidden
+
address@hidden GNU C Library
+This option can also interpret the older FITS date format
+(@code{DD/MM/YYThh:mm:ss.ddd...}) where only two characters are given to
+the year. In this case (following the GNU C Library), this option will make
+the following assumption: values 68 to 99 correspond to the years 1969 to
+1999, and values 0 to 68 as the years 2000 to 2068.
+
+This is a very useful option for operations on the FITS date values, for
+example sorting FITS files by their dates, or finding the time difference
+between two FITS files. The advantage of working with the Unix epoch time
+is that you don't have to worry about calendar details (for example the
+number of days in different months, or leap years, and etc).
 @end table
 
 
@@ -26071,6 +26096,52 @@ spaces). This function will do this within the 
allocated space of the
 string.
 @end deftypefun
 
address@hidden {char *} gal_fits_key_date_to_struct_tm (char @code{*fitsdate}, 
struct tm @code{*tp})
address@hidden Date: FITS format
+Parse @code{fitsdate} as a FITS date format string (most generally:
address@hidden:mm:ss.ddd...}) into the C library's broken-down time
+structure, or @code{struct tm} (declared in @file{time.h}) and return a
+pointer to the remainder of the string (containing the optional sub-second
+portion of @code{fitsdate}, or the @code{.ddd...} of the format).
+
+The returned @code{char *} points to part of the @code{fitsdate} string, so
+it must not be freed. For example, if @code{fitsdate} contains no
+sub-second portion, then the returned @code{char *} will point to the
+NULL-character of @code{fitsdate}.
+
+Note that the FITS date format mentioned above is the most complete
+representation. The following two formats are also acceptable:
address@hidden:mm:ss} and @code{YYYY-MM-DD}. This option can also
+interpret the older FITS date format where only two characters are given to
+the year and the date format is reversed
+(@code{DD/MM/YYThh:mm:ss.ddd...}). In this case (following the GNU C
+Library), this option will make the following assumption: values 68 to 99
+correspond to the years 1969 to 1999, and values 0 to 68 as the years 2000
+to 2068.
address@hidden deftypefun
+
address@hidden size_t gal_fits_key_date_to_seconds (char @code{*fitsdate}, char 
@code{**subsecstr}, double @code{*subsec})
address@hidden Unix epoch time
address@hidden Epoch time, Unix
+Return the Unix epoch time (number of seconds that have passed since
+00:00:00 Thursday, January 1st, 1970) corresponding to the FITS date format
+string @code{fitsdate} (see description of
address@hidden above).
+
+The Unix epoch time is in units of seconds, but the FITS date format allows
+sub-second accuracy. The las two arguments are for the (optional)
+sub-second portion. If @code{fitsdate} contains sub-second accuracy, then
+the starting of the sub-second part is stored in the @code{char *} pointer
+that @code{subsecstr} points to, and @code{subsec} will the corresponding
+numerical value (between 0 and 1, in double precision floating point).
+
+This is a very useful function for operations on the FITS date values, for
+example sorting FITS files by their dates, or finding the time difference
+between two FITS files. The advantage of working with the Unix epoch time
+is that you don't have to worry about calendar details (for example the
+number of days in different months, or leap years, and etc).
address@hidden deftypefun
+
 @deftypefun void gal_fits_key_read_from_ptr (fitsfile @code{*fptr}, gal_data_t 
@code{*keysll}, int @code{readcomment}, int @code{readunit})
 
 Read the list of keyword values from a FITS pointer. The input should be a
diff --git a/doc/plotsrc/README b/doc/plotsrc/README
index 37eb656..befdda4 100644
--- a/doc/plotsrc/README
+++ b/doc/plotsrc/README
@@ -5,4 +5,21 @@ formats.
 
 To do everything, simply run:
 
-    make
\ No newline at end of file
+    make
+
+
+
+
+
+PRACTICAL NOTES
+---------------
+
+1. When creating the final PDF files here (usually at the end of the
+boostrapping step), you might get this error:
+
+convert: attempt to perform an operation not allowed by the security policy 
`EPS' @ error/constitute.c/IsCoderAuthorized/408.
+
+If you do, go to `/etc/ImageMagick-7/policy.xml', and uncomment the line
+with the EPS,PDF,etc.... policy limitations. It was apparently due to a bug
+in Ghostscript that has been fixed since version 9.24
+(https://www.kb.cert.org/vuls/id/332928/).
\ No newline at end of file
diff --git a/lib/fits.c b/lib/fits.c
index 0261c30..91a2115 100644
--- a/lib/fits.c
+++ b/lib/fits.c
@@ -892,6 +892,118 @@ gal_fits_key_clean_str_value(char *string)
 
 
 
+/* Fill the `tm' structure (defined in `time.h') with the values derived
+   from a FITS format date-string and return the (optional) sub-second
+   information as a double.*/
+char *
+gal_fits_key_date_to_struct_tm(char *fitsdate, struct tm *tp)
+{
+  char *c=NULL, *cf;
+  int hasT=0, hassq=0, usesdash=0, usesslash=0;
+
+  /* Initialize the `tm' structure to all-zero elements. In particular, The
+     FITS standard times are written in UTC, so, the time zone (`tm_zone'
+     element, which specifies number of seconds to shift for the time zone)
+     has to be zero. The day-light saving flag (`isdst' element) also has
+     to be set to zero. */
+  tp->tm_sec=tp->tm_min=tp->tm_hour=tp->tm_mday=tp->tm_mon=tp->tm_year=0;
+  tp->tm_wday=tp->tm_yday=tp->tm_isdst=tp->tm_gmtoff;
+  tp->tm_zone=NULL;
+
+  /* According to the FITS standard the `T' in the middle of the date and
+     time of day is optional (the time is not mandatory). */
+  cf=(c=fitsdate)+strlen(fitsdate);
+  do
+    switch(*c)
+      {
+      case 'T':  hasT=1;      break; /* With `T' HH:MM:SS are defined.    */
+      case '-':  usesdash=1;  break; /* Day definition: YYYY-MM-DD.       */
+      case '/':  usesslash=1; break; /* Day definition(old): DD/MM/YY.    */
+      case '\'': hassq=1;     break; /* Wholly Wrapped in a single-quote. */
+      }
+  while(++c<cf);
+
+  /* Convert this date into seconds since 1970/01/01, 00:00:00. */
+  c = ( (usesdash==0 && usesslash==0)
+        ? NULL
+        : ( usesdash
+            ? ( hasT
+                ? strptime(fitsdate, hassq?"'%FT%T'":"%FT%T", tp)
+                : strptime(fitsdate, hassq?"'%F'"   :"%F"   , tp))
+            : ( hasT
+                ? strptime(fitsdate, hassq?"'%d/%m/%yT%T'":"%d/%m/%yT%T", tp)
+                : strptime(fitsdate, hassq?"'%d/%m/%y'"   :"%d/%m/%y"   , tp)
+                )
+            )
+        );
+
+  /* The value might have sub-seconds. In that case, `c' will point to a
+     `.' and we'll have to parse it as double. */
+  if( c==NULL || (*c!='.' && *c!='\0') )
+    error(EXIT_FAILURE, 0, "`%s' isn't in the FITS date format.\n\n"
+          "According to the FITS standard, the date must be in one of "
+          "these formats:\n"
+          "   YYYY-MM-DD\n"
+          "   YYYY-MM-DDThh:mm:ss\n"
+          "   DD/MM/YY               (Note the `YY', see *)\n"
+          "   DD/MM/YYThh:mm:ss      (Note the `YY', see *)\n\n"
+          "[*]: Gnuastro's FITS library (this program), interprets the "
+          "older (two character for year) format, year values 68 to 99 as "
+          "the years 1969 to 1999 and values 0 to 68 as the years 2000 to "
+          "2068.", fitsdate);
+
+  /* Return the subsecond value. */
+  return c;
+}
+
+
+
+
+
+/* Convert the FITS standard date format (as a string, already read from
+   the keywords) into number of seconds since 1970/01/01, 00:00:00. Very
+   useful to avoid calendar issues like number of days in a different
+   months or leap years and etc. The remainder of the format string
+   (sub-seconds) will be put into the two pointer arguments: `subsec' is in
+   double-precision floating point format and  */
+size_t
+gal_fits_key_date_to_seconds(char *fitsdate, char **subsecstr,
+                             double *subsec)
+{
+  time_t t;
+  char *tmp;
+  struct tm tp;
+  void *outptr=subsec;
+
+  /* Fill in the `tp' elements with values read from the string. */
+  tmp=gal_fits_key_date_to_struct_tm(fitsdate, &tp);
+
+  /* If the user cared about the remainder (sub-second string), then set it
+     and convert it to a double type. */
+  if(subsecstr)
+    {
+      /* Set the output pointer. */
+      *subsecstr=tmp;
+
+      /* Convert the remainder string to double-precision floating point
+         (if the given pointer isn't NULL). */
+      if(subsec)
+        if( gal_type_from_string(&outptr, tmp, GAL_TYPE_FLOAT64) )
+          error(EXIT_FAILURE, 0, "%s: the sub-second portion of `%s' (or "
+                "`%s') couldn't be read as a number", __func__, fitsdate,
+                tmp);
+    }
+
+  /* Convert the `tm' structure to `time_t'. */
+  t=mktime(&tp);
+
+  /* Return the value and set the output pointer. */
+  return (size_t)t;
+}
+
+
+
+
 
 /* Read the keyword values from a FITS pointer. The input should be a
    linked list of `gal_data_t'. Before calling this function, you just have
diff --git a/lib/gnuastro/fits.h b/lib/gnuastro/fits.h
index 3816b39..9071294 100644
--- a/lib/gnuastro/fits.h
+++ b/lib/gnuastro/fits.h
@@ -37,6 +37,7 @@ along with Gnuastro. If not, see 
<http://www.gnu.org/licenses/>.
 /* Include other headers if necessary here. Note that other header files
    must be included before the C++ preparations below */
 #include <math.h>
+#include <time.h>
 #include <float.h>
 
 #include <fitsio.h>
@@ -166,6 +167,13 @@ gal_fits_key_img_blank(uint8_t type);
 void
 gal_fits_key_clean_str_value(char *string);
 
+char *
+gal_fits_key_date_to_struct_tm(char *fitsdate, struct tm *tp);
+
+size_t
+gal_fits_key_date_to_seconds(char *fitsdate, char **subsecstr,
+                             double *subsec);
+
 void
 gal_fits_key_read_from_ptr(fitsfile *fptr, gal_data_t *keysll,
                            int readcomment, int readunit);