gnuastro-commits
[Top][All Lists]

## [gnuastro-commits] master 307e155: Book: new section on precedence of Ta

 From: Mohammad Akhlaghi Subject: [gnuastro-commits] master 307e155: Book: new section on precedence of Table operations Date: Sun, 18 Jul 2021 20:44:12 -0400 (EDT)

branch: master
commit 307e155bdbb7befc0d1da659f29fb281865056b2

Book: new section on precedence of Table operations

Until now there was no official place to see which table operation is done
before/after which operation. Describing this under each option was hard
and could be confusing.

With this commit, a new section has been added in the book to give a nice
general view on the various operations that can be done and also show which
one happens before/after which. The logic/reason behind the precedence is
also explained. The section finishes with an example command and a
description of how Table interprets it.

In the process (while checking the precedence on a test table), several
other issues were done:

- I noticed that during column arithmetic we aren't reseting the internal
flags of each column. As a result, the '--noblank' would now work in
some scenarios. So with this commit, upon finishing column arithmetic
the internal flags of the column are reset to zero.

- Column concatenation has been moved before all row-selection or reording
operations. This is much more logically and practically useful.

- In the book, the description of reverse polish notation section has been
greatly improved and edited to help new users.
---
NEWS                   |  13 ++
bin/table/arithmetic.c |   6 +-
bin/table/table.c      |   9 +-
doc/gnuastro.texi      | 320 +++++++++++++++++++++++++++++++++++++++----------
4 files changed, 277 insertions(+), 71 deletions(-)

diff --git a/NEWS b/NEWS
index de5cb51..d537eed 100644
--- a/NEWS
+++ b/NEWS
@@ -15,6 +15,13 @@ See the end of the file for license conditions.
as your paper/slide, while being in vector graphics quality, and not
requiring hundreds of dependencies (only LaTeX is necessary beyond
Gnuastro).
+   - New section to describe the precedence of Table operators in one call
+     to Table. Since the number of operations on table columns/rows has
+     greatly increased in the last few versions, knowing this precedence
+     can minimize the number of calls to Table.
+   - The section on reverse polish notation (used in Arithmetic, for
+     images, and column arithmetic in Table) has been edited and extended
+     to be more useful for a new user.

Arithmetic:
- New operands (also available in Table's column arithmetic):
@@ -76,6 +83,12 @@ See the end of the file for license conditions.
table with the rows of the first. Until this version, this would cause
an error.

+  Table
+   - Column concatenation (adding columns from other files) is now done
+     before all row selection or row re-ording operations (if
+     called). Until now, adding columns from other files was done after all
+     row selection or re-ordering, but this was not too practically useful.
+
** Bugs fixed
bug #60725: MakeCatalog doesn't put comment on --halfsumsb column.
bug #60776: Radial profile script not using standard deviation image,
diff --git a/bin/table/arithmetic.c b/bin/table/arithmetic.c
index 3da9a50..d226a23 100644
--- a/bin/table/arithmetic.c
+++ b/bin/table/arithmetic.c
@@ -929,8 +929,10 @@ arithmetic_reverse_polish(struct tableparams *p, struct
column_pack *outpack)
"single value, but other columns have also been requested "
"which have more elements/rows");

-      /* Set 'single->next' to NULL so it isn't treated as a list and
+      /* Set 'single->next' to NULL so it isn't treated as a list, and set
+         all flags to zero (the arithmetic operation may have changed what
+         the flags pointed to). */
+      single->flag=0;
single->next=NULL;
}
diff --git a/bin/table/table.c b/bin/table/table.c
index 0f49814..3fa627a 100644
--- a/bin/table/table.c
+++ b/bin/table/table.c
@@ -747,8 +747,7 @@ table_select_by_position(struct tableparams *p)

-/*This function concatenates two table column wise .
- It  attaches catcolumn table at the back of first table */
+/* Import columns from another file/table into the working table. */
static void
table_catcolumn(struct tableparams *p)
{
@@ -993,6 +992,9 @@ table_noblank(struct tableparams *p)
void
table(struct tableparams *p)
{
+  /* Concatenate the columns of tables (if required)*/
+  if(p->catcolumnfile) table_catcolumn(p);
+
/* Apply ranges based on row values (if required). */
if(p->selection) table_select_by_value(p);

@@ -1010,9 +1012,6 @@ table(struct tableparams *p)
if(p->outcols)
arithmetic_operate(p);

-  /* Concatenate the columns of tables (if required)*/
-  if(p->catcolumnfile) table_catcolumn(p);
-
/* When column metadata should be updated. */

diff --git a/doc/gnuastro.texi b/doc/gnuastro.texi
index ef7face..79335d3 100644
--- a/doc/gnuastro.texi
+++ b/doc/gnuastro.texi
@@ -414,6 +414,7 @@ Annotations for figure in paper
Table

* Column arithmetic::           How to do operations on table columns.
+* Operation precedence in Table::  Order of running options in Table.
* Invoking asttable::           Options and arguments to Table.

Query
@@ -10743,46 +10744,52 @@ Note that this behavior is ideal for gray-scale
images, if you want a color imag
@node Table, Query, ConvertType, Data containers
@section Table

-Tables are the products of processing astronomical images and spectra.
-For example in Gnuastro, MakeCatalog will process the defined pixels over an
object and produce a catalog (see @ref{MakeCatalog}).
-For each identified object, MakeCatalog can print its position on the image or
sky, its total brightness and many other information that is deducible from the
given image.
+Tables are the high-level products of processing on low-leveler data like
images or spectra.
+For example in Gnuastro, MakeCatalog will process the pixels over an object
and produce a catalog (or table) with the properties of each object like
magnitudes, positions and etc (see @ref{MakeCatalog}).
Each one of these properties is a column in its output catalog (or table) and
for each input object, we have a row.

When there are only a small number of objects (rows) and not too many
properties (columns), then a simple plain text file is mainly enough to store,
transfer, or even use the produced data.
-However, to be more efficient in all these aspects, astronomers have defined
the FITS binary table standard to store data in a binary (0 and 1) format, not
plain text.
-This can offer major advantages in all those aspects: the file size will be
greatly reduced and the reading and writing will be faster (because the RAM and
CPU also work in binary).
-
-The FITS standard also defines a standard for ASCII tables, where the data are
stored in the human readable ASCII format, but within the FITS file structure.
-These are mainly useful for keeping ASCII data along with images and possibly
binary data as multiple (conceptually related) extensions within a FITS file.
+However, to be more efficient, astronomers have defined the FITS binary table
standard to store data in a binary format (which cannot be seen in a text
editor text).
+This can offer major advantages: the file size will be greatly reduced and the
reading and writing will also be faster (because the RAM and CPU also work in
binary).
The acceptable table formats are fully described in @ref{Tables}.

@cindex AWK
@cindex GNU AWK
-Binary tables are not easily readable by human eyes.
+Binary tables are not easily readable with basic plain-text editors.
There is no fixed/unified standard on how the zero and ones should be
interpreted.
-The Unix-like operating systems have flourished because of a simple fact:
communication between the various tools is based on human readable
characters@footnote{In The art of Unix programming'', Eric Raymond makes this
suggestion to programmers: When you feel the urge to design a complex binary
file format, or a complex binary application protocol, it is generally wise to
lie down until the feeling passes.''.
+Unix-like operating systems have flourished because of a simple fact:
communication between the various tools is based on human readable
characters@footnote{In The art of Unix programming'', Eric Raymond makes this
suggestion to programmers: When you feel the urge to design a complex binary
file format, or a complex binary application protocol, it is generally wise to
lie down until the feeling passes.''.
This is a great book and strongly recommended, give it a look if you want to
truly enjoy your work/life in this environment.}.
So while the FITS table standards are very beneficial for the tools that
recognize them, they are hard to use in the vast majority of available software.
This creates limitations for their generic use.

-Table' is Gnuastro's solution to this problem.
-With Table, FITS tables (ASCII or binary) are directly accessible to the
Unix-like operating systems power-users (those working the command-line or
shell, see @ref{Command-line interface}).
+Table is Gnuastro's solution to this problem.
+Table has a large set of operations that you can directly do on any recognized
table (like selecting certain rows, doing arithmetic on the columns and etc).
+For operations that Table doesn't do internally, FITS tables (ASCII or binary)
are directly accessible to the users of Unix-like operating systems (in
particular those working the command-line or shell, see @ref{Command-line
interface}).
With Table, a FITS table (in binary or ASCII formats) is only one command away
from AWK (or any other tool you want to use).
Just like a plain text file that you read with the @command{cat} command.
You can pipe the output of Table into any other tool for higher-level
processing, see the examples in @ref{Invoking asttable} for some simple
examples.

+In the sections below we describe how to effectively use the Table program.
+We start with @ref{Column arithmetic}, where the basic concept and methods of
applying arithmetic operations on one or more columns are discussed.
+Afterwards, in @ref{Operation precedence in Table}, we review the various
types of operations available and their precedence in an instance of calling
Table.
+This is a good place to get a general feeling of all the things you can do
with Table.
+Finally, in @ref{Invoking asttable}, we give some examples and describe each
option in Table.
+
* Column arithmetic::           How to do operations on table columns.
+* Operation precedence in Table::  Order of running options in Table.
* Invoking asttable::           Options and arguments to Table.

-@node Column arithmetic, Invoking asttable, Table, Table
+@node Column arithmetic, Operation precedence in Table, Table, Table
@subsection Column arithmetic

-After reading the requested columns from the input table, you can also do
operations/arithmetic on the columns and save the resulting values as new
column(s) in the output table (possibly in between other requested columns).
-To enable column arithmetic, the first 6 characters of the value to
@option{--column} (@code{-c}) should be the arithmetic activation word
@option{arith }' (note the space character in the end, after @code{arith}').
+In many scenarios, you want to apply some kind of operation on the columns and
save them in another table or feed them into another program.
+With Table you can do a rich set of operations on the contents of one or more
columns in a table, and save the resulting values as new column(s) in the
output table.
+For seeing the precedence of Column arithmetic in relation to other Table
operators, see @ref{Operation precedence in Table}.

-After the activation word, you can use the reverse polish notation to identify
the operators and their operands, see @ref{Reverse polish notation}.
+To enable column arithmetic, the first 6 characters of the value to
@option{--column} (@code{-c}) should be the activation word @option{arith }'
(note the space character in the end, after @code{arith}').
+After the activation word, you can use reverse polish notation to identify the
operators and their operands, see @ref{Reverse polish notation}.
Just note that white-space characters are used between the tokens of the
arithmetic expression and that they are meaningful to the command-line
environment.
Therefore the whole expression (including the activation word) has to be
quoted on the command-line or in a shell script (see the examples below).

@@ -10791,8 +10798,8 @@ When you are giving a column number, it is necessary to
prefix the number with a
Otherwise the number is not distinguishable from a constant number to use in
the arithmetic operation.

For example with the command below, the first two columns of @file{table.fits}
will be printed along with a third column that is the result of multiplying the
first column with @mymath{10^{10}} (for example to convert wavelength from
Meters to Angstroms).
-Note that without the @key{$}', it is not possible to distinguish between 1'' as a column-counter, or as a constant number to use in the arithmetic operation. -Also note that because of the significance of @key{$} for the command-line
environment, the single-quotes are used here (as in an AWK expression), not
double-quotes.
+Note that without the @key{$}', it is not possible to distinguish between 1'' as a column-counter, or 1'' as a constant number to use in the arithmetic operation. +Also note that because of the significance of @key{$} for the command-line
environment, the single-quotes are the recommended quoting method (as in an AWK
expression), not double-quotes (for the significance of using single quotes see
the box below).

@example
$asttable table.fits -c1,2 -c'arith$1 1e10 x'
@@ -10828,13 +10835,13 @@ $asttable table.fits -cAWAV,SPECTRUM -c'arith AWAV 1e10 x' Comparison of the two commands above clearly shows why it is recommended to use column names instead of numbers. When the columns have descriptive names, the command/script actually becomes much more readable, describing the intent of the operation. -It is also independent of the low-level table structure: for the second command, the position of the @code{AWAV} and @code{SPECTRUM} columns in @file{table.fits} is irrelevant. +It is also independent of the low-level table structure: for the second command, the column numbers of the @code{AWAV} and @code{SPECTRUM} columns in @file{table.fits} is irrelevant. -By nature, column arithmetic changes the values of the data within the column. +Column arithmetic changes the values of the data within the column. So the old column meta data can't be used any more. -By default the new column created for the arithmetic operation will be given generic metadata (for example its name will be @code{ARITH_1}, which is hardly useful!). +By default the output column of the arithmetic operation will be given a generic metadata (for example its name will be @code{ARITH_1}, which is hardly useful!). But meta data are critically important and it is good practice to always have short, but descriptive, names for each columns, units and also some comments for more explanation. -To add metadata to a column, you can use the @option{--colmetadata} option that is described in @ref{Invoking asttable}. +To add metadata to a column, you can use the @option{--colmetadata} option that is described in @ref{Invoking asttable} and @ref{Operation precedence in Table}. Finally, since the arithmetic expressions are a value to @option{--column}, it doesn't necessarily have to be a separate option, so the commands above are also identical to the command below (note that this only has one @option{-c} option). Just be very careful with the quoting! @@ -10844,12 +10851,10 @@$ asttable table.fits -cAWAV,SPECTRUM,'arith AWAV
1e10 x'
@end example

Almost all the arithmetic operators of @ref{Arithmetic operators} are also
supported for column arithmetic in Table.
-In particular, the few that are not present in the Gnuastro library aren't yet
supported.
-For a list of the Gnuastro library arithmetic operators, please see the macros
starting with @code{GAL_ARITHMETIC_OP} and ending with the operator name in
@ref{Arithmetic on datasets}.
+In particular, the few that are not present in the Gnuastro
library@footnote{For a list of the Gnuastro library arithmetic operators,
please see the macros starting with @code{GAL_ARITHMETIC_OP} and ending with
the operator name in @ref{Arithmetic on datasets}.} aren't yet supported for
column arithmetic.
Besides the operators in @ref{Arithmetic operators}, several operators are
only available in Table to use on table columns.

-
@cindex WCS: World Coordinate System
@cindex World Coordinate System (WCS)
@table @code
@@ -10970,14 +10975,142 @@ $astfits *.fits --keyvalue=DATE-OBS --colinfoinstdout \ If you don't need to see the Unix-seconds any more, you can add a @option{-cFILENAME} (short for @option{--column=FILENAME}) at the end. For more on @option{--keyvalue}, see @ref{Keyword inspection and manipulation}. +@end table + +@node Operation precedence in Table, Invoking asttable, Column arithmetic, Table +@subsection Operation precedence in Table + +The Table program can do many operations on the rows and columns of the input tables and they aren't always applied in the order you call the operation on the command-line. +In this section we will describe which operation is done before/after which operation. +Knowing this precedence table is important to avoid confusion when you ask for more than one operation. +For a description of each option, please see @ref{Invoking asttable}. + +@table @asis +@item Column information (@option{--information} or @option{-i}) +When given this option, the column data are not read at all. +Table simply reads the column metadata (name, units, numeric data type and comments), and the number of rows and prints them. +Table then terminates and no other operation is done. +This can therefore be called at the end of an arbitrarily long Table command only to remember the column metadata, then deleted to continue writing the command (using the shell's history to retrieve the previous command with an up-arrow key). + +@item Columns from other files (@option{--catcolumns} and @option{--catcolumnfile}) +With this feature, you can import columns from other tables (in other files). +The rest of the operations below are done on the rows, therefore you can merge the columns of various tables into one table, then start limiting the rows to have in the output. + +If any of the row-based operations below are requested in the same Table command, they will also be applied to the rows of these added columns. +However, the conditions to keep/reject rows can only be applied to the rows of the main input table. + +@item Row selection by value in a column +@itemize +@item +@option{--range}: only keep rows within a certain interval in given column. +@item +@option{--inpolygon}: only keep rows within the polygon of @option{--polygon}. +@item +@option{--outpolygon}: only keep rows outside the polygon of @option{--polygon}. +@item +@option{--equal}: only keep rows with specified value in given column. +@item +@option{--notequal}: only keep rows without specified value in given column. +@end itemize +These options take certain column(s) as input and remove some rows from the full table (all columns), based on the given limitations. +They can be called any number of times (to limit the final rows based on values in different columns for example). +Since these are row-rejection operations, their internal order is irrelevant. +In other words, it makes no difference if @option{--equal} is called before or after @option{--range} for example. + +As a side-effect, because NaN/blank values are defined to fail on any condition, these operations will also remove rows with NaN/blank values in the specified column they are checking. +Also, the columns that are used for these operations don't necessarily have to be in the final output table (you may not need the column after doing the selection based on it). +Even though these options are applied after merging columns from other tables, currently their condition-columns can only come from the main input table. +In other words, even though the rows of the added columns (from another file) will also be selected with these options, the condition to keep/reject rows cannot be taken from the newly added columns. + +These options are applied first because the speed of later operations can be greatly affected by the number of rows. +For example, if you also call the @option{--sort} option, and your row selection will result in 50 rows (from an input of 1000 rows), limiting the number of rows can greatly speed up the sorting in your final output. + +@item Sorting (@option{--sort}) +Sort of the rows based on values in a certain column. +The column to sort by can only come from the main input table columns (not columns that may have been added with @option{--catcolumnfile}). + +@item Row selection (by position) +@itemize +@item +@option{--head}: keep only requested number of top rows. +@item +@option{--tail}: keep only requested number of bottom rows. +@item +@option{--rowrandom}: keep only a random number of rows. +@item +@option{--rowlimit}: keep only rows within a certain positional interval. +@end itemize + +These options limit/select rows based on their position within the table (not their value in any certain column). + +@item Column arithmetic +Once the final rows are selected in the requested order, column arithmetic is done (if requested). +For more on column arithmetic, see @ref{Column arithmetic}. + +@item Column metadata (@option{--colmetadata}) +Changing column metadata is necessary after column arithmetic or adding new columns from other tables (that were done above). + +@item Row selection (@option{--noblank}) +Theoretically, this method of row selection by value should have been done with the other logically similar options like @option{--range} or @option{--equal}. +However, those options are applied to the raw (input) column value. +In some scenarios, you need to apply arithmetic operations on the columns (through @ref{Column arithmetic}) before rejecting the undesired rows. +After the arithmetic operation is done, you can use the @code{where} operator to set the non-desired columns to NaN/blank and use @option{--noblank} to remove them at the end. +See the example below for applying any generic value-based row selection based on @option{--noblank}. @end table +As an example, let's review how Table interprets the command below. +We are assuming that @file{table.fits} contians atleast three columns: @code{RA}, @code{DEC} and @code{PARAM} and you only want the RA and Dec of the rows where @mymath{p\times 2<5} (@mymath{p} is the value of each row in the @code{PARAM} column). + +@example +asttable table.fits -cRA,DEC --noblank=MULTIP \ + -c'arith PARAM 2 x set-i i i 5 gt nan where' \ + --colmetadata=3,MULTIP,unit,"Description of column" +@end example + +@noindent +Due to the precedence described in this section, Table does these operations (which are independent of the order of the operations written on the command-line): + +@enumerate +@item +At the start (with @code{-cRA,DEC}), Table reads the @code{RA} and @code{DEC} columns. +@item +In between all the operations in the command above, Column arithmetic (with @option{-c'arith ...'}) has the highest precedence. +So the arithmetic operation is done and stored as a new (third) column. +In this arithmetic operation, we multiply all the values of the @code{PARAM} column by 2, then set all those with a value larger than 5 to NaN (for more on understanding this operation, see the @code{set-}' and @code{where}' operators in @ref{Arithmetic operators}). +@item +Updating column metadata (with @option{--colmetadata}) is then done to give a name (@code{MULTIP}) to the newly calculated (third) column. +During the process, besides a name, we also set a unit and description for the new column. +These metadata entries are @emph{very important}, so always be sure to add metadata after doing column arithmetic. +@item +The lowest precedence operation is @option{--noblank=MULTIP}. +So only rows that aren't blank/NaN in the @code{MULTIP} column are kept. +@item +Finally, the output table (with three columns) is written to the command-line. +If you also want to print the column metadata, you can use the @option{--colinfoinstdout} option. +Alternatively, if you want the output in a file, you can use the @option{--output} option to save the table in FITS or plain-text format. +@end enumerate -@node Invoking asttable, , Column arithmetic, Table +@cartouche +@noindent +@strong{Out of precedence:} It may happen that your desired operation needs a separate precedence. +In this case you can pipe the output of Table into another call of Table and use the @option{--colinfoinstdout} option to preserve the metadata between the two calls. + +For example, let's assume that you want to sort the output table from the example command above based on the new @code{MULTIP} column. +Since sorting is done prior to column arithmetic, you can't do it in one command, but you can circumvent this limitation by simply piping the output (including metadata) to another call to Table: + +@example +asttable table.fits -cRA,DEC --noblank=MULTIP --colinfoinstdout \ + -c'arith PARAM 2 x set-i i i 5 gt nan where' \ + --colmetadata=3,MULTIP,unit,"Description of column" \ + | asttable --sort=MULTIP --output=selected.fits +@end example +@end cartouche + +@node Invoking asttable, , Operation precedence in Table, Table @subsection Invoking Table -Table will read/write, select, convert, or show the information of the columns in FITS ASCII table, FITS binary table and plain text table files, see @ref{Tables}. +Table will read/write, select, modify, or show the information of the rows and columns in recognized Table formats (including FITS binary, FITS ASCII, and plain text table files, see @ref{Tables}). Output columns can also be determined by number or regular expression matching of column names, units, or comments. The executable name is @file{asttable} with the following general template @@ -11030,11 +11163,12 @@ echo "113.64812416667 31.88732" \ | asttable -c'arith$1 degree-to-ra $2 degree-to-dec' @end example -Table's input dataset can be given either as a file or from Standard input (see @ref{Standard input}). +Table's input dataset can be given either as a file or from Standard input (piped from another program, see @ref{Standard input}). In the absence of selected columns, all the input's columns and rows will be written to the output. +The full set of operations Table can do are described in detail below, but for a more high-level introduction to the various operations, and their precedence, see @ref{Operation precedence in Table}. + If any output file is explicitly requested (with @option{--output}) the output table will be written in it. When no output file is explicitly requested the output table will be written to the standard output. - If the specified output is a FITS file, the type of FITS table (binary or ASCII) will be determined from the @option{--tabletype} option. If the output is not a FITS file, it will be printed as a plain text table (with space characters between the columns). When the columns are accompanied by meta-data (like column name, units, or comments), this information will also printed in the plain text file before the table, as described in @ref{Gnuastro text table format}. @@ -11042,12 +11176,12 @@ When the columns are accompanied by meta-data (like column name, units, or comme For the full list of options common to all Gnuastro programs please see @ref{Common options}. Options can also be stored in directory, user or system-wide configuration files to avoid repeating on the command-line, see @ref{Configuration files}. Table does not follow Automatic output that is common in most Gnuastro programs, see @ref{Automatic output}. -Thus, in the absence of an output file, the selected columns will be printed on the command-line with no column information, ready for redirecting to other tools like AWK or sort, similar to the examples above. +Thus, in the absence of an output file, the selected columns will be printed on the command-line with no column information, ready for redirecting to other tools like @command{awk}. @cartouche @noindent @strong{Sexagesimal coordinates as floats in plain-text tables:} -When a column is determined to be a floating point type (32-bit or 64-bit) in a plain-text table, it can contain sexagesimal values in the format of @code{_h_m_s}' (for RA) and @code{_d_m_s}' (for Dec). +When a column is determined to be a floating point type (32-bit or 64-bit) in a plain-text table, it can contain sexagesimal values in the format of @code{_h_m_s}' (for RA) and @code{_d_m_s}' (for Dec), where the @code{_}'s are place-holders for numbers. In this case, the string will be immediately converted to a single floating point number (in units of degrees) and stored in memory with the rest of the column or table. Besides being useful in large tables, with this feature, conversion to sexagesimal coordinates to degrees becomes very easy, for example: @example @@ -11061,6 +11195,9 @@ operators: echo "113.64812416667 31.88732" \ | asttable -c'arith$1 degree-to-ra \$2 degree-to-dec'
@end example
+
+@noindent
+If you want to preserve the sexagesimal contents of a column, you should store
that column as a string, see @ref{Gnuastro text table format}.
@end cartouche

@table @option
@@ -11072,9 +11209,9 @@ Each column's information (number, name, units, data
Note that the FITS standard only requires the data type (see @ref{Numeric data
types}), and in plain text tables, no meta-data/information is mandatory.
Gnuastro has its own convention in the comments of a plain text table to store
and transfer this information as described in @ref{Gnuastro text table format}.

-This option will take precedence over the @option{--column} option, so when it
is called along with requested columns, the latter will be ignored.
+This option will take precedence over all other operations in Table, so when
it is called along with other operations, they will be ignored, see
@ref{Operation precedence in Table}.
This can be useful if you forget the identifier of a column after you have
already typed some on the command-line.
-You can simply add a @option{-i} and run Table to see the whole list and
remember.
changing anything) and run Table, to see the whole list of column names and
information.
Then you can use the shell history (with the up arrow key on the keyboard),
and retrieve the last command with all the previously typed columns present,

@cindex AWK
@@ -11083,11 +11220,11 @@ Then you can use the shell history (with the up arrow
key on the keyboard), and
@itemx --column=STR/INT
Set the output columns either by specifying the column number, or name.
For more on selecting columns, see @ref{Selecting table columns}.
-If a value of this option starts with @code{arith }', this option will do the
requested operations/arithmetic on the specified columns and output the result
in that place (among other requested columns).
+If a value of this option starts with @code{arith }', column arithmetic will
be activated, allowing you to edit/manipulate column contents.
For more on column arithmetic see @ref{Column arithmetic}.

-To ask for multiple columns this option can be used in two way: 1) multiple
calls to this option, 2) using a comma between each column specifier in one
call to this option.
-These different solutions may be mixed in one call to Table: for example,
@option{-cRA,DEC -cMAG}, or @option{-cRA -cDEC -cMAG} are both equivalent to
@option{-cRA -cDEC -cMAG}.
+To ask for multiple columns this option can be used in two ways: 1) multiple
calls to this option, 2) using a comma between each column specifier in one
call to this option.
+These different solutions may be mixed in one call to Table: for example,
@option{-cRA,DEC,MAG}', or @option{-cRA,DEC -cMAG}' are both equivalent to
@option{-cRA -cDEC -cMAG}'.
The order of the output columns will be the same order given to the option or
in the configuration files (see @ref{Configuration file precedence}).

This option is not mandatory, if no specific columns are requested, all the
input table columns are output.
@@ -11095,26 +11232,23 @@ When this option is called multiple times, it is
possible to output one column m

@item -w FITS
@itemx --wcsfile=FITS
-FITS file that contains the WCS to be used in the @code{wcstoimg} and
@code{imgtowcs} operators of @option{--column} (see above).
+FITS file that contains the WCS to be used in the @code{wcstoimg} and
@code{imgtowcs} operators of @ref{Column arithmetic}.
The extension name/number within the FITS file can be specified with
@option{--wcshdu}.

-If the value to this option is @option{none}, no WCS will be written in the
output.
+If the value to this option is @option{none}', no WCS will be written in the
output.

@item -W STR
@itemx --wcshdu=STR
-FITS extension/HDU that contains the WCS to be used in the @code{wcstoimg} and
@code{imgtowcs} operators of @option{--column} (see above).
-The FITS file name can be specified with @option{--wcsfile}.
+FITS extension/HDU in the FITS file given to @option{--wcsfile} (see the
description of @option{--wcsfile} for more).

@item -L FITS/TXT
@itemx --catcolumnfile=FITS/TXT
Concatenate (or add, or append) the columns of this option's value (a
filename) to the output columns.
This option may be called multiple times (to add columns from more than one
file into the final output), the columns from each file will be added in the
same order that this option is called.
+The number of rows in the file(s) given to this option has to be the same as
the input table (before any type of row-selection), see @ref{Operation
precedence in Table}.

By default all the columns of the given file will be appended, if you only
want certain columns to be appended, use the @option{--catcolumns} option to
specify their name or number (see @ref{Selecting table columns}).
-Note that the columns given to @option{--catcolumns} must be present in all
the given files (if this option is called more than once).
-
-The concatenation is done after any column selection (for example with
@option{--column}) or row selection (for example with @option{--range}) is
applied to the main input table given to Table.
-The number of rows in the file(s) given to this option has to be the same as
the final output table if this option wasn't given.
+Note that the columns given to @option{--catcolumns} must be present in all
the given files (if this option is called more than once with more than one
file).

If the file given to this option is a FITS file, its necessary to also define
the corresponding HDU/extension with @option{--catcolumnhdu}.
Also note that no operation (for example row selection, arithmetic or etc) is
applied to the table given to this option.
@@ -11122,7 +11256,7 @@ Also note that no operation (for example row selection,
arithmetic or etc) is ap
If the appended columns have a name, the column names of each file will be
appended with a @code{-N}, where @code{N} is a counter starting from 1 for each
appended file.
This is done because when concatenating columns from multiple tables (more
than two) into one, they may have the same name, and its not good practice to
have multiple columns with the same name.
You can disable this feature with @option{--catcolumnrawname}.
-To have full control over the concatenated column names, you can use the
+Generally, you can use the @option{--colmetadata} option to update column

For example, let's assume you have two catalogs of the same objects (same
number of rows) in different filters.
Such that @file{f160w-cat.fits} has a @code{MAGNITUDE} column that has the
magnitude of each object in the @code{F160W} filter and similarly
@file{f105w-cat.fits}, also has a @code{MAGNITUDE} column, but for the
@code{F105W} filter.
@@ -11135,6 +11269,9 @@ asttable f160w-cat.fits --output=both.fits \
@end example

+@noindent
+For a more complete example, see @ref{Working with catalogs estimating colors}.
+
@item -u STR/INT
@itemx --catcolumnhdu=STR/INT
The HDU/extension of the FITS file(s) that should be concatenated, or
appended, with @option{--catcolumnfile}.
@@ -11154,14 +11291,16 @@ Don't modify the names of the concatenated (appended)
columns, see description i
@itemx --colinfoinstdout
@cindex Standard output
Add column metadata when the output is printed in the standard output.
-Usually the standard output is used for a fast visual check or to pipe into
other program for further processing.
+Usually the standard output is used for a fast visual check, or to pipe into
other metadata-agnostic programs (like AWK) for further processing.
So by default meta-data aren't included.
+But when piping to other Gnuastro programs (where metadata can be interpreted
and used) it is recommended to use this option and use column names in the next
program.

@item -r STR,FLT:FLT
@itemx --range=STR,FLT:FLT
Only output rows that have a value within the given range in the @code{STR}
column (can be a name or counter).
Note that the range is only inclusive in the lower-limit.
For example with @code{--range=sn,5:20} the output's columns will only contain
rows that have a value in the @code{sn} column (not case-sensitive) that is
greater or equal to 5, and less than 20.
+For the precedence of this operation in relation to others, see @ref{Operation
precedence in Table}.

This option can be called multiple times (different ranges for different
columns) in one run of the Table program.
This is very useful for selecting the final rows from multiple
criteria/columns.
@@ -11175,6 +11314,7 @@ For one example of using this option, see the example
under
@item --inpolygon=STR1,STR2
Only return rows where the given coordinates are inside the polygon specified
by the @option{--polygon} option.
The coordinate columns are the given @code{STR1} and @code{STR2} columns, they
can be a column name or counter (see @ref{Selecting table columns}).
+For the precedence of this operation in relation to others, see @ref{Operation
precedence in Table}.

Note that the chosen columns doesn't have to be in the output columns (which
are specified by the @code{--column} option).
For example if we want to select rows in the polygon specified in @ref{Dataset
inspection and cropping}, this option can be used like this (you can remove the
double quotations and write them all in one line if you remove the white-spaces
around the colon separating the column vertices):
@@ -11191,7 +11331,7 @@ asttable table.fits --inpolygon=RA,DEC      \
@noindent
@strong{Flat/Euclidean space: } The @option{--inpolygon} option assumes a
flat/Euclidean space so it is only correct for RA and Dec when the polygon size
is very small like the example above.
If your polygon is a degree or larger, it may not return correct results.
-We are working on other options for this.
+Please get in touch if you need such a feature (see @ref{Suggest new feature}).
@end cartouche

@item --outpolygon=STR1,STR2
@@ -11207,6 +11347,8 @@ This option behaves identically to the same option in
the Crop program, so for m
@itemx --equal=STR,INT/FLT,...
Only output rows that are equal to the given number(s) in the given column.
The first argument is the column identifier (name or number, see
@ref{Selecting table columns}), after that you can specify any number of values.
+For the precedence of this operation in relation to others, see @ref{Operation
precedence in Table}.
+
For example @option{--equal=ID,5,6,8} will only print the rows that have a
value of 5, 6, or 8 in the @code{ID} column.
This option can also be called multiple times, so @option{--equal=ID,4,5
--equal=ID,6,7} has the same effect as @option{--equal=4,5,6,7}.

@@ -11234,9 +11376,10 @@ Be very careful if you want to use the non-equality
with floating point numbers,
This option also works when the given column has a string type, see the
description under @option{--equal} (above) for more.

@item -s STR
-@item --sort=STR
+@itemx --sort=STR
Sort the output rows based on the values in the @code{STR} column (can be a
column name or number).
By default the sort is done in ascending/increasing order, to sort in a
descending order, use @option{--descending}.
+For the precedence of this operation in relation to others, see @ref{Operation
precedence in Table}.

The chosen column doesn't have to be in the output columns.
This is good when you just want to sort using one column's values, but don't
need that column anymore afterwards.
@@ -11251,6 +11394,7 @@ Only print the given number of rows from the @emph{top}
of the final table.
Note that this option only affects the @emph{output} table.
For example if you use @option{--sort}, or @option{--range}, the printed rows
are the first @emph{after} applying the sort sorting, or selecting a range of
the full input.
This option cannot be called with @option{--tail}, @option{--rowlimit} or
@option{--rowrandom}.
+For the precedence of this operation in relation to others, see @ref{Operation
precedence in Table}.

@cindex GNU Coreutils
If the given value to @option{--head} is 0, the output columns won't have any
rows and if its larger than the number of rows in the input table, all the rows
are printed (this option is effectively ignored).
@@ -11266,6 +11410,7 @@ This option cannot be called with @option{--head},
@option{--rowlimit} or @optio
Only return the rows within the requested positional range (inclusive on both
sides).
Therefore, @code{--rowlimit=5,7} will return 3 of the input rows, row 5, 6 and
7.
This option will abort if any of the given values is larger than the total
number of rows in the table.
+For the precedence of this operation in relation to others, see @ref{Operation
precedence in Table}.

With the @option{--head} or @option{--tail} options you can only see the top
or bottom few rows.
However, with this option, you can limit the returned rows to a contiguous set
of rows in the middle of the table.
@@ -11279,6 +11424,7 @@ This option is applied @emph{after} the value-based
selection options (like @opt
On the other hand, only the row counters are randomly selected, this option
doesn't change the order.
Therefore, if @option{--rowrandom} is called together with @option{--sort},
the returned rows are still sorted.
This option cannot be called with @option{--head}, @option{--tail}, or
@option{--rowlimit}.
+For the precedence of this operation in relation to others, see @ref{Operation
precedence in Table}.

This option will only have an effect if @code{INT} is larger than the number
of rows when it is activated (after the value-based selection options have been
applied).
When there are fewer rows, a warning is printed, saying that this option has
no effect.
@@ -11298,6 +11444,7 @@ For more, see @ref{Generating random numbers}.
@itemx --noblank=STR[,STR[,STR]]
Remove all rows in the given @emph{output} columns that have a blank value.
Like above, the columns can be specified by their name or number (counting
from 1).
+
For example if @file{table.fits} has blank values (NaN in floating point
types) in the @code{magnitude} and @code{sn} columns, with
@code{--noblank=magnitude,sn}, the output will not contain any rows with blank
values in these columns.

If you want @emph{all} columns to be checked, simply set the value to
@code{_all} (in other words: @option{--noblank=_all}).
@@ -11307,11 +11454,14 @@ For example if you give
@option{--noblank=_all,magnitude}, then Table will assum

This option is applied just before writing the final table (after
So in case you changed the column metadata, or added new columns, you can use
the new names, or the newly defined column numbers.
+For the precedence of this operation in relation to others, see @ref{Operation
precedence in Table}.

@item -m STR/INT,STR[,STR[,STR]]
Update the specified column metadata in the output table.
This option is applied after all other column-related operations are complete,
for example column arithmetic, or column concatenation.
+For the precedence of this operation in relation to others, see @ref{Operation
precedence in Table}.
+
The first value (before the first comma) given to this option is the column's
identifier.
It can either be a counter (positive integer, counting from 1), or a name (the
column's name in the output if this option wasn't called).

see @ref{Invoking astarith
@cindex Post-fix notation
@cindex Reverse Polish Notation
The most common notation for arithmetic operations is the
@url{https://en.wikipedia.org/wiki/Infix_notation, infix notation} where the
operator goes between the two operands, for example @mymath{4+5}.
-While the infix notation is the preferred way in most programming languages,
currently the Gnuastro's program (in particular Arithmetic and Table, when
doing column arithmetic) do not use it.
-This is because it will require parenthesis which can complicate the
implementation of the code.
-In the near future we do plan to also allow this
the time being (due to time constraints on the developers), arithmetic
operations can only be done in the post-fix notation (also known as
@url{https://en.wikipedia.org/wiki/Reverse_Polish_notation, reverse polish
notation}).
-The Wikipedia article provides some excellent explanation on this notation but
here we will give a short summary here for self-sufficiency.
-
-In the post-fix notation, the operator is placed after the operands, as we
will see below this removes the need to define parenthesis for most ordinary
operators.
+The infix notation is the preferred way in most programming languages which
come with scripting features for large programs.
+This is because the infix notation requires a way to define precedence when
more than one operator is involved.
+
+For example consider the statement @code{5 + 6 / 2}.
+Should 6 first be divided by 2, then added by 5?
+Or should 5 first be added with 6, then divided by 2?
+Therefore we need parenthesis to show precedence: @code{5+(6/2)} or
@code{(5+6)/2}.
+Furthermore, if you need to leave a value for later processing, you will need
to define a variable for it; for example @code{a=(5+6)/2}.
+
+Gnuastro provides libraries where you can also use infix notation in C or C++
programs.
+However, Gnuastro's programs are primarily designed to be run on the
command-line and the level of complexity that infix notation requires can be
annoying/confusing to write on the command-line (where they can get confused
with the shell's parenthesis or variable definitions.
+Therefore Gnuastro's Arithmetic and Table (when doing column arithmetic)
programs use the the post-fix notation, also known as
@url{https://en.wikipedia.org/wiki/Reverse_Polish_notation, reverse polish
notation}.
For example, instead of writing @command{5+6}, we write @command{5 6 +}.
-To easily understand how this notation works, you can think of each operand as
a node in a last-in-first-out'' stack.
-Every time an operator is confronted, the operator pops the number of operands
it needs from the top of the stack (so they don't exist in the stack any more),
does its operation and pushes the result back on top of the stack.
+
+The Wikipedia article on the reverse polish notation provides some excellent
explanation on this notation but here we will give a short summary here for
self-sufficiency.
+In short, in the reverse polish notation, the operator is placed after the
operands.
+As we will see below this removes the need to define parenthesis and lets you
use previous values without needing to define a variable.
do plan to also optionally allow infix notation when arithmetic operations on
datasets are desired, but due to time constraints on the developers we can't do
it immediately.
+
+To easily understand how the reverse polish notation works, you can think of
each operand (@code{5} and @code{6} in the example above) as a node in a
last-in-first-out'' stack.
+One such stack in daily life is a stack of dishes in the kitchen: you put a
clean dish, on the top of a stack of dishes when its ready for later usage.
+Later, when you need a dish, you pick the top one (hence the last'' dish
placed in'' the stack is the first'' dish that comes out'' when
necessary).
+
+Each operator will need a certain number of operators (in the example above,
the @code{+} operator needs two operands).
+In the kitchen metaphor, an operator can be an oven.
+Every time an operator is confronted, the operator takes (or pops'') the
number of operands it needs from the top of the stack (so they don't exist in
the stack any more), does its operation, and places (or pushes'') the result
back on top of the stack.
So if you want the average of 5 and 6, you would write: @command{5 6 + 2 /}.
The operations that are done are:

@enumerate
@item
-@command{5} is an operand, so it is pushed to the top of the stack (which is
initially empty).
+@command{5} is an operand, so Arithmetic pushes it to the top of the stack
(which is initially empty).
+In the kitchen metaphoe, you can visualize this as taking a new dish from the
cabinet, putting the number 5 inside of the dish, and putting the dish on top
of the (empy) cooking table infront of you.
+You now have a stack of one dish on the table infront of you.
@item
-@command{6} is an operand, so it is pushed to the top of the stack.
+@command{6} is also an operand, so it is pushed to the top of the stack.
+Like before, you can visualize this as taking a new dish from the cabinet,
putting the number 6 in it and placing it on top of the previous dish.
+You now have a stack of two dishes on the table infront of you.
@item
@command{+} is a @emph{binary} operator, so it will pop the top two elements
of the stack out of it, and perform addition on them (the order is @mymath{5+6}
in the example above).
The result is @command{11} which is pushed to the top of the stack.
+
+To visualize this, you can think of the @code{+} operator as an oven with a
place for two dishes.
+You pick up the top-most dish (that has the number 6 in it) and put it in the
over.
+Then you pick the next dish (that has the number 5) and also put it in the
oven and close the oven door.
+When the oven has finished its cooking, it produces a single output (in one
dish, with the number 11 inside of it).
+You take that output dish and put it back on the table.
+You now have a stack of one dish on the table infront of you.
@item
@command{2} is an operand so push it onto the top of the stack.
+In the kitchen metaphor, you again go to the cabinet, pick up a dish and put
the number 2 inside of it and put the dish over the previous dish (that has the
number 11).
+You now have a stack of two dishes on the table infront of you.
@item
-@command{/} is a binary operator, so pull out the top two elements of the
stack (top-most is @command{2}, then @command{11}) and divide the second one by
the first.
+@command{/} (division) is a binary operator, so pull out the top two elements
of the stack (top-most is @command{2}, then @command{11}) and divide the second
one by the first.
+In the kitchen metaphor, the @command{/} operator can be visualized as a
microwave that takes two dishes.
+But unlike the oven (@code{+} operator) before, the order of inputs matters
(they are on top of each other: with the top dish holder being the nominator
and the bottom one being the denominator).
+Again, you look to your stack of dishes on the table.
+
+You pick up the top one (with value 2 inside of it) and put it in the
microwave's bottom (denominator) dish holder.
+Then you go back to your stack of dishes on the table and pick up the top dish
(with value 11 inside of it) and put that in the top (nominator) disho holder.
+The microwave will do its work and when its finished, returns a new dish with
the single value 5.5 inside of it.
+You pick up the dish from the microwave and place it back on the table.
+
+@item
+There are no more operands or operators, so simply return the remaining
operand in the output.
+In the kitchen metaphor, you see that your recipe has no more steps, so you
just pick up the remaining dish and take it to the dining room to enjoy a good
dinner.
@end enumerate

-In the Arithmetic program, the operands can be FITS images or numbers (see
@ref{Invoking astarithmetic}).
-In Table's column arithmetic, they can be any column or a number (see
@ref{Column arithmetic}).
+In the Arithmetic program, the operands can be FITS images of any
dimensionality, or numbers (see @ref{Invoking astarithmetic}).
+In Table's column arithmetic, they can be any column in the table (a series of
numbers in an array) or a single number (see @ref{Column arithmetic}).

With this notation, very complicated procedures can be created without the
need for parenthesis or worrying about precedence.
Even functions which take an arbitrary number of arguments can be defined in
this notation.
@@ -20087,10 +20279,10 @@ See the description of @option{--config} in
@ref{Operating mode options}.}.
Please see @ref{Sufi simulates a detection} for a very complete tutorial
explaining how one could use MakeProfiles in conjunction with other Gnuastro's
programs to make a complete simulated image of a mock galaxy.

-* MakeProfiles catalog::           Required catalog properties.
+* MakeProfiles catalog::        Required catalog properties.
* MakeProfiles profile settings::  Configuration parameters for all profiles.
-* MakeProfiles output dataset::    The canvas/dataset to build profiles over.
-* MakeProfiles log file::          A description of the optional log file.
+* MakeProfiles output dataset::  The canvas/dataset to build profiles over.
+* MakeProfiles log file::       A description of the optional log file.