gnuastro-commits
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[gnuastro-commits] master a40ff0b 1/2: --checkconfig prints parsing of c


From: Mohammad Akhlaghi
Subject: [gnuastro-commits] master a40ff0b 1/2: --checkconfig prints parsing of configuration files
Date: Mon, 14 Jan 2019 20:48:25 -0500 (EST)

branch: master
commit a40ff0b5fff521604e5ee3865c3a6cfb0bf30497
Author: Mohammad Akhlaghi <address@hidden>
Commit: Mohammad Akhlaghi <address@hidden>

    --checkconfig prints parsing of configuration files
    
    With this option, its now possible to check/debug how options are parsed by
    any Gnuastro program in the same order that they are parsed.
    
    This idea for this option actually came up when I was checking how
    `--minmapsize' is being parsed by the Fits program. I found that
    `--minmapsize' is being ignored and that there is no check when its value
    is zero (it is needed for allocating some datasets). To address the first,
    its no longer in the list of hidden options for Fits and for the second, we
    there is now a warning to inform the users of unexpected behavior.
---
 NEWS                               |   7 ++
 bin/fits/fits.c                    |   1 +
 bin/fits/ui.c                      |   1 -
 doc/gnuastro.texi                  |  65 ++++++++++-----
 lib/gnuastro-internal/commonopts.h |  14 ++++
 lib/gnuastro-internal/options.h    |   6 ++
 lib/options.c                      | 163 +++++++++++++++++++++++++++++++++++--
 7 files changed, 228 insertions(+), 29 deletions(-)

diff --git a/NEWS b/NEWS
index 3b1ea23..8c9a2a6 100644
--- a/NEWS
+++ b/NEWS
@@ -5,6 +5,13 @@ GNU Astronomy Utilities NEWS                          -*- 
outline -*-
 
 ** New features
 
+  All programs:
+   --checkconfig: print the names and values given to options as they are
+     parsed on the command-line or in various configuration files (the
+     configuration file name is also printed). This can be very useful in
+     debugging (finding which configuration file is responsible for a given
+     option's value).
+
   Fits:
    - Add "title" to group FITS keywords with `--write=/,"title name". This
      "title" is composed of two keyword records: a blank one, followed by
diff --git a/bin/fits/fits.c b/bin/fits/fits.c
index 2220a9d..24e58ec 100644
--- a/bin/fits/fits.c
+++ b/bin/fits/fits.c
@@ -359,6 +359,7 @@ fits(struct fitsparams *p)
 
     /* HDU, functions defined here. */
     case FITS_MODE_HDU:
+
       /* Options that must be called alone. */
       if(p->numhdus)
         fits_hdu_number(p);
diff --git a/bin/fits/ui.c b/bin/fits/ui.c
index 04defd6..fdbe637 100644
--- a/bin/fits/ui.c
+++ b/bin/fits/ui.c
@@ -121,7 +121,6 @@ ui_initialize_options(struct fitsparams *p,
         case GAL_OPTIONS_KEY_TABLEFORMAT:
         case GAL_OPTIONS_KEY_DONTDELETE:
         case GAL_OPTIONS_KEY_LOG:
-        case GAL_OPTIONS_KEY_MINMAPSIZE:
         case GAL_OPTIONS_KEY_NUMTHREADS:
         case GAL_OPTIONS_KEY_STDINTIMEOUT:
           cp->coptions[i].flags=OPTION_HIDDEN;
diff --git a/doc/gnuastro.texi b/doc/gnuastro.texi
index d669b08..ff92d1e 100644
--- a/doc/gnuastro.texi
+++ b/doc/gnuastro.texi
@@ -7918,11 +7918,26 @@ command-line or in the configuration files. In any 
case, it will be
 immediately read (before parsing the rest of the options on the
 command-line, or lines in a configuration file).
 
-Note that by definition, later options on the command-line still take
-precedence over those in these in any configuration file, including the
-file(s) given to this option. Also see @option{--lastconfig} and
address@hidden on how this option can be used for reproducible
-results.
+Note that by definition, options on the command-line still take precedence
+over those in these in any configuration file, including the file(s) given
+to this option if they are called before it. Also see @option{--lastconfig}
+and @option{--onlyversion} on how this option can be used for reproducible
+results. You can use @option{--checkconfig} (below) to check/confirm the
+parsing of configuration files.
+
address@hidden --checkconfig
+Print options and their values, within the command-line or configuration
+files, as they are parsed (see @ref{Configuration file precedence}). If an
+option has already been set, or is ignored by the program, this option will
+also inform you with special values like @code{--ALREADY-SET--}. Only
+options that are parsed after this option are printed, so to see the
+parsing of all input options, it is recommended to put this option
+immediately after the program name before any other options.
+
address@hidden Debug
+This is a very good option to confirm where the value of each option is has
+been defined in scenarios where there are multiple configuration files (for
+debugging).
 
 @item -S
 @itemx --setdirconf
@@ -8258,21 +8273,8 @@ The option values in all the programs of Gnuastro will 
be filled in the
 following order. If an option only takes one value which is given in an
 earlier step, any value for that option in a later step will be
 ignored. Note that if the @option{lastconfig} option is specified in any
-step below, all later files will be ignored (see @ref{Operating mode
-options}). The basic idea behind setting this progressive state of checking
-for parameter values is that separate users of a computer or separate
-folders in a user's file system might need different values for some
-parameters.
-
-In each step, there can also be a configuration file containing the common
-options in all the programs: @file{gnuastro.conf} (see @ref{Common
-options}). If options specific to one program are specified in this file,
-there will be unrecognized option errors, or unexpected behavior if the
-option has different behavior in another program. On the other hand, there
-is no problem with @file{astprogname.conf} containing common
address@hidden an example, the @option{--setdirconf} and
address@hidden options will also write the common options they have
-read in their produced @file{astprogname.conf}.}.
+step below, no other configuration files will be parsed (see @ref{Operating
+mode options}).
 
 @enumerate
 @item
@@ -8306,6 +8308,29 @@ system-wide installation directory (see @ref{System 
wide} for
 
 @end enumerate
 
+The basic idea behind setting this progressive state of checking for
+parameter values is that separate users of a computer or separate folders
+in a user's file system might need different values for some
+parameters.
+
address@hidden
address@hidden
address@hidden the order:} You can confirm/check the order of parsing
+configuration files using the @option{--checkconfig} option with any
+Gnuastro program, see @ref{Operating mode options}. Just be sure to place
+this option immediately after the program name, before any other option.
address@hidden cartouche
+
+As you see above, there can also be a configuration file containing the
+common options in all the programs: @file{gnuastro.conf} (see @ref{Common
+options}). If options specific to one program are specified in this file,
+there will be unrecognized option errors, or unexpected behavior if the
+option has different behavior in another program. On the other hand, there
+is no problem with @file{astprogname.conf} containing common
address@hidden an example, the @option{--setdirconf} and
address@hidden options will also write the common options they have
+read in their produced @file{astprogname.conf}.}.
+
 @cartouche
 @noindent
 @strong{Manipulating the order:} You can manipulate this order or add new
diff --git a/lib/gnuastro-internal/commonopts.h 
b/lib/gnuastro-internal/commonopts.h
index 1939e69..4420947 100644
--- a/lib/gnuastro-internal/commonopts.h
+++ b/lib/gnuastro-internal/commonopts.h
@@ -395,6 +395,20 @@ struct argp_option gal_commonopts_options[] =
       gal_options_call_parse_config_file
     },
     {
+      "checkconfig",
+      GAL_OPTIONS_KEY_CHECKCONFIG,
+      0,
+      0,
+      "List all config files and variables read.",
+      GAL_OPTIONS_GROUP_OPERATING_MODE,
+      &cp->checkconfig,
+      GAL_OPTIONS_NO_ARG_TYPE,
+      GAL_OPTIONS_RANGE_0_OR_1,
+      GAL_OPTIONS_NOT_MANDATORY,
+      GAL_OPTIONS_NOT_SET,
+      gal_options_check_config
+    },
+    {
       "setdirconf",
       GAL_OPTIONS_KEY_SETDIRCONF,
       0,
diff --git a/lib/gnuastro-internal/options.h b/lib/gnuastro-internal/options.h
index 211df64..ba22bd3 100644
--- a/lib/gnuastro-internal/options.h
+++ b/lib/gnuastro-internal/options.h
@@ -114,6 +114,7 @@ enum options_common_keys
   GAL_OPTIONS_KEY_CONFIG,
   GAL_OPTIONS_KEY_SEARCHIN,
   GAL_OPTIONS_KEY_LASTCONFIG,
+  GAL_OPTIONS_KEY_CHECKCONFIG,
   GAL_OPTIONS_KEY_TABLEFORMAT,
   GAL_OPTIONS_KEY_ONLYVERSION,
   GAL_OPTIONS_KEY_WORKOVERCH,
@@ -205,6 +206,7 @@ struct gal_options_common_params
   uint8_t           setdirconf; /* To write the directory config file.    */
   uint8_t           setusrconf; /* To write teh user config config file.  */
   uint8_t           lastconfig; /* This is the last configuration file.   */
+  uint8_t          checkconfig; /* Check config files and values.         */
 
   /* Output files. */
   gal_fits_list_key_t  *okeys;  /* Configuration as FITS keys in output.  */
@@ -257,6 +259,10 @@ gal_options_print_citation(struct argp_option *option, 
char *arg,
                            char *filename, size_t lineno, void *pa);
 
 void *
+gal_options_check_config(struct argp_option *option, char *arg,
+                         char *filename, size_t lineno, void *junk);
+
+void *
 gal_options_read_type(struct argp_option *option, char *arg,
                       char *filename, size_t lineno, void *junk);
 
diff --git a/lib/options.c b/lib/options.c
index cd7fd32..133f427 100644
--- a/lib/options.c
+++ b/lib/options.c
@@ -340,6 +340,62 @@ gal_options_print_citation(struct argp_option *option, 
char *arg,
 
 
 void *
+gal_options_check_config(struct argp_option *option, char *arg,
+                          char *filename, size_t lineno, void *junk)
+{
+  char *str;
+
+  /* First see if we are reading or writing. */
+  if(lineno==-1)
+    {
+      gal_checkset_allocate_copy("1", &str);
+      return str;
+    }
+
+  /* Check if the given value is different from this version. */
+  else
+    {
+      /* If its already set then ignore it. */
+      if(option->set) return NULL;
+
+      /* Activate the option and let the user know its activated. */
+      (*(uint8_t *)(option->value)) = 1;
+      printf("-----------------\n"
+             "Parsing of options AFTER `--checkconfig'.\n\n"
+             "IMPORTANT: Any option that was parsed before encountering "
+             "`--checkconfig', on the command-line or in a configuration "
+             "file, is not shown here. It is thus recommended to use this "
+             "option before calling any other option.\n"
+             "-----------------\n");
+
+      /* Print where this option was first seen: if `checkconfig' is called
+         within a configuration file, `filename!=NULL' and has an argument
+         (=="1"). But on the command-line, it has no argument or
+         filename. */
+      if(filename)
+        printf("%s:\n", filename);
+      else
+        {
+          if(arg)
+            error(EXIT_FAILURE, 0, "%s: a bug! Please contact us at %s to "
+                  "fix the problem. `filename==NULL', but `arg!=NULL'",
+                  __func__, PACKAGE_BUGREPORT);
+          else
+            printf("Command-line:\n");
+        }
+
+      /* Just to avoid compiler warnings for unused variables. The program
+         will never reach this point! */
+      arg=filename=NULL; lineno=0; option=NULL; junk=NULL;
+    }
+  return NULL;
+}
+
+
+
+
+
+void *
 gal_options_read_type(struct argp_option *option, char *arg,
                       char *filename, size_t lineno, void *junk)
 {
@@ -1204,6 +1260,12 @@ gal_options_read_check(struct argp_option *option, char 
*arg, char *filename,
          unset). */
       option->func(option, arg, filename, lineno, topass);
       if(option->key!=GAL_OPTIONS_KEY_CONFIG) option->set=GAL_OPTIONS_SET;
+
+      /* The `--config' option is printed for `--checkconfig' by its
+         function (`gal_options_call_parse_config_file'), so must be
+         ignored here. */
+      if(cp->checkconfig && option->key!=GAL_OPTIONS_KEY_CONFIG)
+        printf("  %-25s%s\n", option->name, arg?arg:"ACTIVATED");
       return;
     }
 
@@ -1217,7 +1279,12 @@ gal_options_read_check(struct argp_option *option, char 
*arg, char *filename,
       else
         {
           /* If the option is already set, ignore the given value. */
-          if(option->set==GAL_OPTIONS_SET) return;
+          if(option->set==GAL_OPTIONS_SET)
+            {
+              if(cp->checkconfig)
+                printf("  %-25s--ALREADY-SET--\n", option->name);
+              return;
+            }
 
           /* Read the string argument into the value. */
           if( gal_type_from_string(&option->value, arg, option->type) )
@@ -1245,7 +1312,12 @@ gal_options_read_check(struct argp_option *option, char 
*arg, char *filename,
   else
     {
       /* If the option is already set, ignore the given value. */
-      if(option->set==GAL_OPTIONS_SET) return;
+      if(option->set==GAL_OPTIONS_SET)
+        {
+          if(cp->checkconfig)
+            printf("  %-25s--ALREADY-SET--\n", option->name);
+          return;
+        }
 
       /* Make sure the option has the type set for options with no
          argument. So, give it a value of 1. */
@@ -1261,6 +1333,12 @@ gal_options_read_check(struct argp_option *option, char 
*arg, char *filename,
     }
 
 
+  /* If the user wanted to check the value. */
+  if(cp->checkconfig)
+    printf("  %-25s%s\n", option->name,
+           (arg && option->type!=GAL_OPTIONS_NO_ARG_TYPE)?arg:"ACTIVATED");
+
+
   /* Flip the `set' flag to `GAL_OPTIONS_SET'. */
   option->set=GAL_OPTIONS_SET;
 }
@@ -1537,7 +1615,13 @@ options_set_from_name(char *name, char *arg,  struct 
argp_option *options,
           if( options[i].flags==OPTION_HIDDEN
               || ( options[i].set
                    && !gal_type_is_list(options[i].type ) ) )
-            return 0;
+            {
+              if(cp->checkconfig)
+                printf("  %-25s%s\n", name, ( options[i].flags==OPTION_HIDDEN
+                                              ? "--IGNORED--"
+                                              : "--ALREADY-SET--" ) );
+              return 0;
+            }
 
           /* Read the value into the option and do a sanity check. */
           gal_options_read_check(&options[i], arg, filename, lineno, cp);
@@ -1610,6 +1694,11 @@ options_parse_file(char *filename,  struct 
gal_options_common_params *cp,
     }
 
 
+  /* If necessary, print the configuration file name. */
+  if(cp->checkconfig)
+    printf("%s:\n", filename);
+
+
   /* Allocate the space necessary to keep a copy of each line as we parse
      it. Note that `getline' is going to later `realloc' this space to fit
      the line length. */
@@ -1661,11 +1750,26 @@ options_parse_file(char *filename,  struct 
gal_options_common_params *cp,
 /* This function will be used when the `--config' option is called. */
 void *
 gal_options_call_parse_config_file(struct argp_option *option, char *arg,
-                                   char *filename, size_t lineno, void *cp)
+                                   char *filename, size_t lineno, void *c)
 {
+  struct gal_options_common_params *cp=(struct gal_options_common_params *)c;
+
+  /* The `--config' option is a special function when it comes to
+     `--checkconfig': we'll have to write its value before interpretting
+     it. */
+  if(cp->checkconfig)
+    {
+      printf("  %-25s%s\n", option->name, arg);
+      printf("............................\n");
+    }
+
   /* Call the confguration file parser. */
   options_parse_file(arg, cp, 1);
 
+  /* Ending boundary of this file's options. */
+  if(cp->checkconfig)
+      printf("............................\n");
+
   /* Just to avoid compiler warnings, then return, note that all pointers
      are just copies. */
   option=NULL; filename=NULL; lineno=0;
@@ -1770,6 +1874,51 @@ options_reverse_lists_check_mandatory(struct 
gal_options_common_params *cp,
 
 
 
+void
+gal_options_read_low_level_checks(struct gal_options_common_params *cp)
+{
+  size_t suggested_mmap=10000000;
+
+  /* If `numthreads' is 0, use the number of threads available to the
+     system. */
+  if(cp->numthreads==0)
+    cp->numthreads=gal_threads_number();
+
+  /* If `minmapsize==0' and quiet isn't given, print a warning. */
+  if(cp->minmapsize==0)
+    {
+      fprintf(stderr, "\n\n"
+              "========= WARNING =========\n"
+              "Minimum size to map an allocated space outside of RAM is "
+              "not set, or set to zero. This can greatly slow down the "
+              "processing of a program or cause strange crashes (recall "
+              "that the number of files that can be memory-mapped is "
+              "limited).\n\n"
+
+              "On modern systems (with RAM larger than a giga-byte), it "
+              "should be fine to set it to %zu (10 million bytes or 10Mb) "
+              "with the command below. In this manner, only arrays that "
+              "are larger than this will be memory-mapped and smaller "
+              "arrays (which are much more numerous) will be allocated and "
+              "freed in the RAM.\n\n"
+
+              "     --minmapsize=%zu\n\n"
+
+              "[This warning can be disabled with the `--quiet' (or `-q') "
+              "option.]\n"
+              "===========================\n\n", suggested_mmap,
+              suggested_mmap);
+    }
+
+  /* If the user wanted to check the parsing of configuration files, then
+     the program must stop here. */
+  if(cp->checkconfig) exit(0);
+}
+
+
+
+
+
 /* Read all configuration files and set common options */
 void
 gal_options_read_config_set(struct gal_options_common_params *cp)
@@ -1786,10 +1935,8 @@ gal_options_read_config_set(struct 
gal_options_common_params *cp)
   /* Abort if any of the mandatory options are not set. */
   gal_options_abort_if_mandatory_missing(cp);
 
-  /* If `numthreads' is 0, use the number of threads available to the
-     system. */
-  if(cp->numthreads==0)
-    cp->numthreads=gal_threads_number();
+  /* Low-level/basic checks before passing control back to program. */
+  gal_options_read_low_level_checks(cp);
 }
 
 



reply via email to

[Prev in Thread] Current Thread [Next in Thread]