[gnuastro-commits] master 41d0bfa: Table: new --noblankend option to allow removing blanks from input

[Mohammad Akhlaghi]

branch: master
commit 41d0bfab845bcfba9321043de46c3db61cd4f4db
Author: Mohammad Akhlaghi <mohammad@akhlaghi.org>
Commit: Mohammad Akhlaghi <mohammad@akhlaghi.org>

    Table: new --noblankend option to allow removing blanks from input
    
    Until now, the '--noblank' option would only apply to the output table (for
    example after doing column arithmetic or adding columns from other
    tables). However, logically, it is very similar to the '--range' or
    '--equal' options (that operate on the input table before column arithmetic
    or addign columns). As a result, this differing behavior could cause
    confusion for the users!
    
    With this commit, the '--noblank' operator now operates on the input table
    (and doesn't necessarily have to be in the output table!). To keep the old
    functionality, the new '--noblankend' option has been added that has the
    same behavior as the old '--noblank', except that it can be called multiple
    times, not just once with all the columns.
    
    This issue was reported by Samane Raji.
---
 NEWS                             | 22 +++++++++
 bin/table/args.h                 | 20 ++++++--
 bin/table/asttable-complete.bash |  4 +-
 bin/table/main.h                 |  5 +-
 bin/table/table.c                | 42 ++++++++++-------
 bin/table/ui.c                   | 98 ++++++++++++++++++++++++++++++++++++----
 bin/table/ui.h                   |  3 +-
 doc/announce-acknowledge.txt     |  1 +
 doc/gnuastro.texi                | 56 +++++++++++++++++------
 9 files changed, 202 insertions(+), 49 deletions(-)

diff --git a/NEWS b/NEWS
index 1f661ca..67c8d3f 100644
--- a/NEWS
+++ b/NEWS
@@ -45,6 +45,12 @@ See the end of the file for license conditions.
      operations. This was suggested by Raúl Infante-Sainz and Manuel
      Sánchez Benavente.
    --catrowhdu: The HDU(s) of the FITS file(s) given to '--catrowfile'.
+   --noblankend: remove blank rows from selected columns of the _output_
+     table, for example after after adding columns by column arithmetic or
+     from other tables (this is the new name for the old '--noblank'
+     option, see the changed features below). Like the new '--noblank',
+     this option can also be called multiple times, so '--noblankend=1
+     --noblankend=2' is equivalent to '--noblankend=1,2'.
 
   Match:
    - k-d tree based matching has been added for finding the matching rows,
@@ -86,6 +92,22 @@ See the end of the file for license conditions.
      of the inputs and timings of important steps to the standard output
      (on the command-line).
 
+  Table:
+   - The '--noblank' option's logic is very similar to other row-selection
+     (by value) options (like '--range', '--polygon' or
+     '--equal'). However, until now, it was applied at the end of Table,
+     just before writing the output (so it would apply to the output table,
+     that may have new/different column names/orders, not the input table
+     like the other similar options). This was very inconvenient and
+     non-intuitive. From this version, '--noblank' option now behaves
+     similarly to the other input row-selection (by value) options. So it
+     can be called multiple times (to give many columns in multiple calls)
+     and it can be called for a column that is not necessarily one of the
+     output columns (for example you want the RA and Dec of all rows
+     without a blank magnitude, so you don't need to have the magnitude
+     column any more!). The old behavior of this option is now available in
+     the new '--noblankend' option. This point was raised by Samane Raji.
+
   Library:
    - gal_fits_tab_read: now takes the number of threads to read the desired
      columns in parallel (this will greatly improve reading speed when
diff --git a/bin/table/args.h b/bin/table/args.h
index bf039da..5e5cd4c 100644
--- a/bin/table/args.h
+++ b/bin/table/args.h
@@ -388,12 +388,24 @@ struct argp_option program_options[] =
       0,
       "Remove rows with blank in given columns.",
       GAL_OPTIONS_GROUP_INPUT,
-      &p->noblank,
-      GAL_TYPE_STRING,
+      &p->noblankll,
+      GAL_TYPE_STRLL,
       GAL_OPTIONS_RANGE_ANY,
       GAL_OPTIONS_NOT_MANDATORY,
-      GAL_OPTIONS_NOT_SET,
-      gal_options_parse_csv_strings
+      GAL_OPTIONS_NOT_SET
+    },
+    {
+      "noblankend",
+      UI_KEY_NOBLANKEND,
+      "STR[,STR]",
+      0,
+      "Sim. to --noblank, at end (e.g., after arith.).",
+      GAL_OPTIONS_GROUP_INPUT,
+      &p->noblankend,
+      GAL_TYPE_STRLL,
+      GAL_OPTIONS_RANGE_ANY,
+      GAL_OPTIONS_NOT_MANDATORY,
+      GAL_OPTIONS_NOT_SET
     },
 
 
diff --git a/bin/table/asttable-complete.bash b/bin/table/asttable-complete.bash
index 920166b..508da82 100644
--- a/bin/table/asttable-complete.bash
+++ b/bin/table/asttable-complete.bash
@@ -66,13 +66,13 @@ _gnuastro_autocomplete_asttable_option_value(){
     case "$option_name" in
 
         # Options that take a columns from the main argument.
-        
--column|--noblank|--inpolygon|--outpolygon|--colmetadata|--equal|--notequal)
+        
--column|--noblank|--noblankend|--inpolygon|--outpolygon|--colmetadata|--equal|--notequal)
 
             # The '--column' and '--noblank' options can (and usually
             # will!) take more than one column name as value.
             local continuematch=""
             case "$option_name" in
-                --column|--noblank) continuematch=yes;;
+                --column|--noblank|--noblankend) continuematch=yes;;
             esac
 
             # Find the suggestions.
diff --git a/bin/table/main.h b/bin/table/main.h
index e1c2eee..5777163 100644
--- a/bin/table/main.h
+++ b/bin/table/main.h
@@ -44,6 +44,7 @@ enum select_types
  SELECT_TYPE_OUTPOLYGON,
  SELECT_TYPE_EQUAL,
  SELECT_TYPE_NOTEQUAL,
+ SELECT_TYPE_NOBLANK,
 
  /* This marks the total number of row-selection criteria. */
  SELECT_TYPE_NUMBER,
@@ -100,6 +101,8 @@ struct tableparams
   gal_data_t         *polygon;  /* Values of vertices of the polygon.   */
   gal_data_t           *equal;  /* Values to keep in output.            */
   gal_data_t        *notequal;  /* Values to not include in output.     */
+  gal_list_str_t   *noblankll;  /* Remove rows with blank values.       */
+  gal_list_str_t  *noblankend;  /* Similar to noblank but at end.       */
   char                  *sort;  /* Column name or number for sorting.   */
   uint8_t          descending;  /* Sort columns in descending order.    */
   size_t                 head;  /* Output only the no. of top rows.     */
@@ -107,7 +110,6 @@ struct tableparams
   gal_data_t        *rowlimit;  /* Output rows in row-counter range.    */
   size_t            rowrandom;  /* Number of rows to show randomly.     */
   uint8_t             envseed;  /* Use the environment for random seed. */
-  gal_data_t         *noblank;  /* Remove rows that have blank.         */
   gal_list_str_t *catcolumnfile; /* Filename to concat column wise.     */
   gal_list_str_t *catcolumnhdu;  /* HDU/extension for the catcolumn.    */
   gal_list_str_t  *catcolumns;  /* List of columns to concatenate.      */
@@ -118,6 +120,7 @@ struct tableparams
 
   /* Internal. */
   struct column_pack *outcols;  /* Output column packages.              */
+  gal_data_t         *noblank;  /* Remove rows that have blank.         */
   gal_data_t           *table;  /* Linked list of output table columns. */
   struct wcsprm          *wcs;  /* WCS structure for conversion.        */
   int                    nwcs;  /* Number of WCS structures.            */
diff --git a/bin/table/table.c b/bin/table/table.c
index 1b4d770..fa5083f 100644
--- a/bin/table/table.c
+++ b/bin/table/table.c
@@ -431,6 +431,10 @@ table_select_by_value(struct tableparams *p)
           addmask=table_selection_equal_or_notequal(p, tmp->col, 1);
           break;
 
+        case SELECT_TYPE_NOBLANK:
+          addmask = gal_arithmetic(GAL_ARITHMETIC_OP_ISBLANK, 1, 0, tmp->col);
+          break;
+
         default:
           error(EXIT_FAILURE, 0, "%s: a bug! Please contact us at %s "
                 "to fix the problem. The code %d is not a recognized "
@@ -438,8 +442,8 @@ table_select_by_value(struct tableparams *p)
                 tmp->type);
         }
 
-      /* Remove any blank elements. */
-      if(gal_blank_present(tmp->col, 1))
+      /* Remove any blank elements (incase we are on a noblank column. */
+      if(tmp->type!=SELECT_TYPE_NOBLANK && gal_blank_present(tmp->col, 1))
         {
           blmask = gal_arithmetic(GAL_ARITHMETIC_OP_ISBLANK, 1, 0, tmp->col);
           addmask=gal_arithmetic(GAL_ARITHMETIC_OP_OR, 1, inplace,
@@ -1042,16 +1046,20 @@ table_colmetadata(struct tableparams *p)
 
 
 void
-table_noblank(struct tableparams *p)
+table_noblankend(struct tableparams *p)
 {
   int found;
+  gal_list_str_t *tmp;
   size_t i, j, *index;
   gal_data_t *tcol, *flag;
-  char **strarr=p->noblank->array;
   gal_list_sizet_t *column_indexs=NULL;
 
+  /* Merge all possible calls to '--noblankend' into one list. */
+  gal_options_merge_list_of_csv(&p->noblankend);
+
   /* See if all columns should be checked, or just a select few. */
-  if( p->noblank->size==1 && !strcmp(strarr[0],"_all") )
+  if( gal_list_str_number(p->noblankend)==1
+      && !strcmp(p->noblankend->v,"_all") )
     {
       for(i=0;i<gal_list_data_number(p->table);++i)
         gal_list_sizet_add(&column_indexs, i);
@@ -1059,7 +1067,7 @@ table_noblank(struct tableparams *p)
 
   /* Only certain columns should be checked, so find/add their index. */
   else
-    for(i=0;i<p->noblank->size;++i)
+    for(tmp=p->noblankend; tmp!=NULL; tmp=tmp->next)
       {
         /* First go through the column names and if they match, add
            them. Note that we don't want to stop once a name is found, in
@@ -1069,7 +1077,7 @@ table_noblank(struct tableparams *p)
         found=0;
         for(tcol=p->table; tcol!=NULL; tcol=tcol->next)
           {
-            if( tcol->name && !strcmp(tcol->name, strarr[i]) )
+            if( tcol->name && !strcmp(tcol->name, tmp->v) )
               {
                 found=1;
                 gal_list_sizet_add(&column_indexs, j);
@@ -1083,26 +1091,28 @@ table_noblank(struct tableparams *p)
           {
             /* Parse the given index. */
             index=NULL;
-            if( gal_type_from_string((void **)(&index), strarr[i],
+            if( gal_type_from_string((void **)(&index), tmp->v,
                                      GAL_TYPE_SIZE_T) )
               error(EXIT_FAILURE, 0, "column '%s' didn't match any of the "
                     "final column names and can't be parsed as a column "
-                    "counter (starting from 1) either", strarr[i]);
+                    "counter (starting from 1) either", tmp->v);
 
             /* Make sure its not zero (the user counts from 1). */
             if(*index==0)
               error(EXIT_FAILURE, 0, "the column number (given to the "
-                    "'--noblank' option) should start from 1, but you have "
+                    "'--noblankend' option) should start from 1, but you have "
                     "given 0.");
 
             /* Make sure that the index falls within the number (note that
                it still counts from 1).  */
             if(*index > gal_list_data_number(p->table))
-              error(EXIT_FAILURE, 0, "the final output table only has %zu "
-                    "columns, but you have given column %zu to '--noblank'. "
-                    "Recall that '--noblank' operates on the output columns "
-                    "and that you can also use output column names (if they "
-                    "have any)",
+              error(EXIT_FAILURE, 0, "the final output table only has "
+                    "%zu columns, but you have given column %zu to "
+                    "'--noblankend'. Recall that '--noblankend' operates "
+                    "at the end (on the output columns) and that you "
+                    "can also use output column names (if they have "
+                    "any). In case you meant a column from the input "
+                    "table, you should use '--noblank'",
                     gal_list_data_number(p->table), *index);
 
             /* Everything is fine, add the index to the list of columns to
@@ -1174,7 +1184,7 @@ table(struct tableparams *p)
   if(p->colmetadata) table_colmetadata(p);
 
   /* When any columns with blanks should be removed. */
-  if(p->noblank) table_noblank(p);
+  if(p->noblankend) table_noblankend(p);
 
   /* Write the output. */
   gal_table_write(p->table, NULL, NULL, p->cp.tableformat, p->cp.output,
diff --git a/bin/table/ui.c b/bin/table/ui.c
index aa07616..a997f95 100644
--- a/bin/table/ui.c
+++ b/bin/table/ui.c
@@ -730,6 +730,64 @@ ui_check_select_sort_read_col_ind(char *string)
 
 
 
+/* Fill replace '--noblank' with all columns (if it is given '_all'). */
+static void
+ui_noblank_prepare(struct tableparams *p, size_t numcols)
+{
+  size_t i, zero=0;
+  gal_list_str_t *tmp;
+  char name[50]; /* No table will have a 50 digit number of columns! */
+
+  /* Merge all the possible calls of 'noblank' into one. */
+  gal_options_merge_list_of_csv(&p->noblankll);
+
+  /* Initialize the 'noblank' list of (empty!) datasets (we only need the
+     column names, this is done to conform with the general selection
+     mechanism, where some selection methods need values). */
+  p->noblank=NULL;
+
+  /* In case it is given an '_all' option (for all columns to be
+     checked). */
+  if(gal_list_str_number(p->noblankll)==1
+     && !strcmp(p->noblankll->v,"_all"))
+    {
+      /* Add all the columns to the final list of datasets. */
+      for(i=0;i<numcols;++i)
+        {
+          sprintf(name, "%zu", i+1);
+          gal_list_data_add_alloc(&p->noblank, NULL, GAL_TYPE_FLOAT64, 1,
+                                  &zero, NULL, 0, -1, 1, name, NULL, NULL);
+        }
+    }
+
+  /* Only certain columns are requested (we should just put the strings in
+     the desired format of a list of datasets with names, not a single
+     dataset with type string and many elements). */
+  else
+    {
+      for(tmp=p->noblankll; tmp!=NULL; tmp=tmp->next)
+        gal_list_data_add_alloc(&p->noblank, NULL, GAL_TYPE_FLOAT64, 1,
+                                &zero, NULL, 0, -1, 1, tmp->v, NULL, NULL);
+    }
+
+  /* Reverse the final output list of columns. */
+  gal_list_data_reverse(&p->noblank);
+
+  /* Clean up the list that we don't need any more. */
+  gal_list_str_free(p->noblankll, 1);
+  p->noblankll=NULL;
+
+  /* For a check:
+  gal_data_t *dtmp;
+  for(dtmp=p->noblank;dtmp!=NULL;dtmp=dtmp->next)
+    printf("col: %s\n", dtmp->name);
+  exit(0);
+  */
+}
+
+
+
+
 
 /* See if row selection or sorting needs any extra columns to be read. */
 static void
@@ -749,10 +807,11 @@ ui_check_select_sort_before(struct tableparams *p, 
gal_list_str_t *lines,
   gal_data_t *dtmp, *allcols, *inpolytmp=NULL, *outpolytmp=NULL;
 
   /* Important note: these have to be in the same order as the 'enum
-     select_types' in 'main.h'. We'll fill the two polygon elements
-     later. */
+     select_types' in 'main.h'. We'll fill the NULL components
+     afterwards. */
   gal_data_t *select[SELECT_TYPE_NUMBER]={p->range, NULL, NULL,
-                                          p->equal, p->notequal};
+                                          p->equal, p->notequal,
+                                          NULL};
 
 
   /* The inpolygon dataset is currently a single dataset with two elements
@@ -778,14 +837,32 @@ ui_check_select_sort_before(struct tableparams *p, 
gal_list_str_t *lines,
     }
 
 
+  /* Get all the input table's column information. */
+  allcols=gal_table_info(p->filename, p->cp.hdu, lines, &numcols,
+                         &numrows, &tableformat);
+
+
+  /* If '--noblank' is given an '_all', we should replace it with the
+     counters of all the input table columns. */
+  if(p->noblankll)
+    {
+      ui_noblank_prepare(p, numcols);
+      select[SELECT_TYPE_NOBLANK]=p->noblank;
+    }
+
+
   /* Allocate necessary spaces. */
   if(p->selection)
     {
+      /* Find the number of columns that are needed for selection. */
       *nselect = ( gal_list_data_number(p->range)
                    + gal_list_data_number(inpolytmp)
                    + gal_list_data_number(outpolytmp)
                    + gal_list_data_number(p->equal)
-                   + gal_list_data_number(p->notequal) );
+                   + gal_list_data_number(p->notequal)
+                   + gal_list_data_number(p->noblank) );
+
+      /* Allocate the necessary arrays. */
       selectind=gal_pointer_allocate(GAL_TYPE_SIZE_T, *nselect, 0,
                                      __func__, "selectind");
       selecttype=gal_pointer_allocate(GAL_TYPE_SIZE_T, *nselect, 0,
@@ -794,7 +871,12 @@ ui_check_select_sort_before(struct tableparams *p, 
gal_list_str_t *lines,
                                         __func__, "selectindout");
       selecttypeout=gal_pointer_allocate(GAL_TYPE_SIZE_T, *nselect, 0,
                                          __func__, "selecttypeout");
+
+      /* Initialize them to blank. */
       sf=(s=selectindout)+*nselect; do *s++=GAL_BLANK_SIZE_T; while(s<sf);
+
+      /* Set the output pointers (they have already been initialized to
+         NULL by the caller function). */
       *selectindout_out=selectindout;
       *selecttypeout_out=selecttypeout;
     }
@@ -813,11 +895,6 @@ ui_check_select_sort_before(struct tableparams *p, 
gal_list_str_t *lines,
         }
 
 
-  /* Get all the column information. */
-  allcols=gal_table_info(p->filename, p->cp.hdu, lines, &numcols,
-                         &numrows, &tableformat);
-
-
   /* If the values are numbers, we'll check if the given value is less than
      the total number of columns. Just note that the indexs count from
      zero. */
@@ -1073,7 +1150,7 @@ ui_preparations(struct tableparams *p)
 
   /* If any kind of row-selection is requested set 'p->selection' to 1. */
   p->selection = ( p->range || p->inpolygon || p->outpolygon || p->equal
-                   || p->notequal );
+                   || p->notequal || p->noblankll );
 
 
   /* If row sorting or selection are requested, see if we should read any
@@ -1305,6 +1382,7 @@ ui_free_report(struct tableparams *p)
   ui_outcols_free(p->outcols);
   gal_list_data_free(p->table);
   if(p->wcshdu) free(p->wcshdu);
+  gal_list_data_free(p->noblank);
   gal_list_str_free(p->columns, 1);
   if(p->colarray) free(p->colarray);
   gal_list_data_free(p->colmetadata);
diff --git a/bin/table/ui.h b/bin/table/ui.h
index 9ee9136..b62944e 100644
--- a/bin/table/ui.h
+++ b/bin/table/ui.h
@@ -42,7 +42,7 @@ enum program_args_groups
 /* Available letters for short options:
 
    a d f g j k l p t v x y z
-   A B E G J O Q X Y
+   A B G J O Q X Y
 */
 enum option_keys_enum
 {
@@ -60,6 +60,7 @@ enum option_keys_enum
   UI_KEY_HEAD            = 'H',
   UI_KEY_TAIL            = 't',
   UI_KEY_NOBLANK         = 'b',
+  UI_KEY_NOBLANKEND      = 'E',
   UI_KEY_CATCOLUMNS      = 'C',
   UI_KEY_CATCOLUMNHDU    = 'u',
   UI_KEY_CATCOLUMNFILE   = 'L',
diff --git a/doc/announce-acknowledge.txt b/doc/announce-acknowledge.txt
index 7f77cb0..45a9b2d 100644
--- a/doc/announce-acknowledge.txt
+++ b/doc/announce-acknowledge.txt
@@ -11,6 +11,7 @@ Zahra Hosseini
 Raúl Infante-Sainz
 Alejandro Lumbreras Calle
 Sebastian Luna-Valero
+Samane Raji
 Manuel Sánchez-Benavente
 Peter Teuben
 Jesús Varela
diff --git a/doc/gnuastro.texi b/doc/gnuastro.texi
index 54ad8d1..9e4d168 100644
--- a/doc/gnuastro.texi
+++ b/doc/gnuastro.texi
@@ -11070,6 +11070,8 @@ If the datatypes are different, you can use the type 
conversion operators of Tab
 @option{--equal}: only keep rows with specified value in given column.
 @item
 @option{--notequal}: only keep rows without specified value in given column.
+@item
+@option{--noblank}: only keep rows that aren't blank in the given column(s).
 @end itemize
 These options take certain column(s) as input and remove some rows from the 
full table (all columns), based on the given limitations.
 They can be called any number of times (to limit the final rows based on 
values in different columns for example).
@@ -11110,19 +11112,20 @@ For more on column arithmetic, see @ref{Column 
arithmetic}.
 @item Column metadata (@option{--colmetadata})
 Changing column metadata is necessary after column arithmetic or adding new 
columns from other tables (that were done above).
 
-@item Row selection (@option{--noblank})
-Theoretically, this method of row selection by value should have been done 
with the other logically similar options like @option{--range} or 
@option{--equal}.
-However, those options are applied to the raw (input) column value.
-In some scenarios, you need to apply arithmetic operations on the columns 
(through @ref{Column arithmetic}) before rejecting the undesired rows.
-After the arithmetic operation is done, you can use the @code{where} operator 
to set the non-desired columns to NaN/blank and use @option{--noblank} to 
remove them at the end.
-See the example below for applying any generic value-based row selection based 
on @option{--noblank}.
+@item Output row selection (@option{--noblankend})
+Only keep the output rows that don't have a blank value in the given column(s).
+For example, you may need to apply arithmetic operations on the columns 
(through @ref{Column arithmetic}) before rejecting the undesired rows.
+After the arithmetic operation is done, you can use the @code{where} operator 
to set the non-desired columns to NaN/blank and use @option{--noblankend} 
option to remove them just before writing the output.
+In other scenarios, you may want to remove blank values based on columns in 
another table.
+You can also use the modified metadata of the previous steps to use updated 
names!
+See the example below for applying any generic value-based row selection based 
on @option{--noblankend}.
 @end table
 
 As an example, let's review how Table interprets the command below.
 We are assuming that @file{table.fits} contains at least three columns: 
@code{RA}, @code{DEC} and @code{PARAM} and you only want the RA and Dec of the 
rows where @mymath{p\times 2<5} (@mymath{p} is the value of each row in the 
@code{PARAM} column).
 
 @example
-asttable table.fits -cRA,DEC --noblank=MULTIP \
+asttable table.fits -cRA,DEC --noblankend=MULTIP \
          -c'arith PARAM 2 x set-i i i 5 gt nan where' \
          --colmetadata=3,MULTIP,unit,"Description of column"
 @end example
@@ -11142,7 +11145,7 @@ Updating column metadata (with @option{--colmetadata}) 
is then done to give a na
 During the process, besides a name, we also set a unit and description for the 
new column.
 These metadata entries are @emph{very important}, so always be sure to add 
metadata after doing column arithmetic.
 @item
-The lowest precedence operation is @option{--noblank=MULTIP}.
+The lowest precedence operation is @option{--noblankend=MULTIP}.
 So only rows that aren't blank/NaN in the @code{MULTIP} column are kept.
 @item
 Finally, the output table (with three columns) is written to the command-line.
@@ -11159,7 +11162,7 @@ For example, let's assume that you want to sort the 
output table from the exampl
 Since sorting is done prior to column arithmetic, you can't do it in one 
command, but you can circumvent this limitation by simply piping the output 
(including metadata) to another call to Table:
 
 @example
-asttable table.fits -cRA,DEC --noblank=MULTIP --colinfoinstdout \
+asttable table.fits -cRA,DEC --noblankend=MULTIP --colinfoinstdout \
          -c'arith PARAM 2 x set-i i i 5 gt nan where' \
          --colmetadata=3,MULTIP,unit,"Description of column" \
          | asttable --sort=MULTIP --output=selected.fits
@@ -11461,6 +11464,23 @@ This option can also be called multiple times, so 
@option{--notequal=ID,4,5 --no
 Be very careful if you want to use the non-equality with floating point 
numbers, see the special note under @option{--equal} for more.
 This option also works when the given column has a string type, see the 
description under @option{--equal} (above) for more.
 
+@item -b STR[,STR[,STR]]
+@itemx --noblank=STR[,STR[,STR]]
+Only output rows that are @emph{not} blank in the given column of the 
@emph{input} table.
+Like above, the columns can be specified by their name or number (counting 
from 1).
+This option can be called multiple times, so @option{--noblank=MAG 
--noblank=PHOTOZ} is equivalent to @option{--noblank=MAG,PHOTOZ}.
+For the precedence of this operation in relation to others, see @ref{Operation 
precedence in Table}.
+
+For example if @file{table.fits} has blank values (NaN in floating point 
types) in the @code{magnitude} and @code{sn} columns, with 
@code{--noblank=magnitude,sn}, the output will not contain any rows with blank 
values in these two columns.
+
+If you want @emph{all} columns to be checked, simply set the value to 
@code{_all} (in other words: @option{--noblank=_all}).
+This mode is useful when there are many columns in the table and you want a 
``clean'' output table (with no blank values in any column): entering their 
name or number one-by-one can be buggy and frustrating.
+In this mode, no other column name should be given.
+For example if you give @option{--noblank=_all,magnitude}, then Table will 
assume that your table actually has a column named @code{_all} and 
@code{magnitude}, and if it doesn't, it will abort with an error.
+
+If you want to change column values using @ref{Column arithmetic} (and set 
some to blank, to later remove), or you want to select rows based on columns 
that you have imported from other tables, you should use the 
@option{--noblankend} option described below.
+Also, see @ref{Operation precedence in Table}.
+
 @item -s STR
 @itemx --sort=STR
 Sort the output rows based on the values in the @code{STR} column (can be a 
column name or number).
@@ -11526,17 +11546,23 @@ Read the random number generator seed from the 
@code{GSL_RNG_SEED} environment v
 This is useful if you want a reproducible random selection of the input rows.
 For more, see @ref{Generating random numbers}.
 
-@item -b STR[,STR[,STR]]
-@itemx --noblank=STR[,STR[,STR]]
-Remove all rows in the given @emph{output} columns that have a blank value.
+@item -E STR[,STR[,STR]]
+@itemx --noblankend=STR[,STR[,STR]]
+Remove all rows in the requested @emph{output} columns that have a blank value.
 Like above, the columns can be specified by their name or number (counting 
from 1).
+This option can be called multiple times, so @option{--noblank=MAG 
--noblank=PHOTOZ} is equivalent to @option{--noblank=MAG,PHOTOZ}.
+For the precedence of this operation in relation to others, see @ref{Operation 
precedence in Table}.
 
-For example if @file{table.fits} has blank values (NaN in floating point 
types) in the @code{magnitude} and @code{sn} columns, with 
@code{--noblank=magnitude,sn}, the output will not contain any rows with blank 
values in these columns.
+For example if your final output table (possibly after column arithmetic, or 
adding new columns) has blank values (NaN in floating point types) in the 
@code{magnitude} and @code{sn} columns, with @code{--noblankend=magnitude,sn}, 
the output will not contain any rows with blank values in these two columns.
 
-If you want @emph{all} columns to be checked, simply set the value to 
@code{_all} (in other words: @option{--noblank=_all}).
+If you want blank values to be removed from the main input table _before_ any 
further processing (like adding columns, sorting or column arithmetic), you 
should use the @option{--noblank} option.
+With the @option{--noblank} option, the column(s) that is(are) given doesn't 
necessarily have to be in the output (it is just temporarily used for reading 
the inputs and selecting rows, but doens't necessarily need to be present in 
the output).
+However, the column(s) given to this option should exist in the output.
+
+If you want @emph{all} columns to be checked, simply set the value to 
@code{_all} (in other words: @option{--noblankend=_all}).
 This mode is useful when there are many columns in the table and you want a 
``clean'' output table (with no blank values in any column): entering their 
name or number one-by-one can be buggy and frustrating.
 In this mode, no other column name should be given.
-For example if you give @option{--noblank=_all,magnitude}, then Table will 
assume that your table actually has a column named @code{_all} and 
@code{magnitude}, and if it doesn't, it will abort with an error.
+For example if you give @option{--noblankend=_all,magnitude}, then Table will 
assume that your table actually has a column named @code{_all} and 
@code{magnitude}, and if it doesn't, it will abort with an error.
 
 This option is applied just before writing the final table (after 
@option{--colmetadata} has finished).
 So in case you changed the column metadata, or added new columns, you can use 
the new names, or the newly defined column numbers.