[gnuastro-commits] master ea30904: Updated NEWS file, small corrections in the manual

[Mohammad Akhlaghi]

branch: master
commit ea30904fbc37eeac2a671efef84042dc1b83c2b8
Author: Mohammad Akhlaghi <address@hidden>
Commit: Mohammad Akhlaghi <address@hidden>

    Updated NEWS file, small corrections in the manual
    
    The NEWS file was updated for the pending alpha-release. The Hubble
    tutorial section was also reviewed and corrected where necessary.
---
 NEWS              | 312 ++++++++++++++++++++++++++++++++++++++++++++++--------
 doc/gnuastro.texi |  96 ++++++++---------
 2 files changed, 314 insertions(+), 94 deletions(-)

diff --git a/NEWS b/NEWS
index 32f070b..b7092de 100644
--- a/NEWS
+++ b/NEWS
@@ -1,57 +1,128 @@
 GNU Astronomy Utilities NEWS                          -*- outline -*-
 
-* Noteworthy changes in release 0.2.28 (library 1.0.0) (2016-10-23) [alpha]
+* Noteworthy changes in release 0.2.206 (library 1.0.0) (2017-05-05) [alpha]
+
+  This is a full re-write of Gnuastro. Most importantly, Gnuastro now has a
+  new generic data contaihtner (`gal_data_t'). This new container can now
+  deal natively with all standard numeric data types, work in RAM or
+  HDD/SSD, keep data in any dimensions and has enabled many other very
+  useful features. Some of the most prominent of the new features are
+  discussed below. It is strongly recommended to review the respective
+  section of the Gnuastro manual/book for a better feeling of all the new
+  features.
+
+  As discussed below, some program names have changed, if you have a
+  previous version of Gnuastro installed from source, it is recommended to
+  uninstall it first (with `make uninstall' using the corresponding
+  tarball), then install this new version. Building Gnuastro can be slow,
+  so please build in parallel with Make's `-j8' option (to build on 8
+  threads).
+
+** New programs or library features
+
+  Library functions that deal with datasets now use this generic data
+  container for inputs and outputs, significantly simplifying their
+  API. Nearly all library functions have been re-written with much more
+  clear names, argument lists and individual purpose. Major new library
+  features are:
+
+     -- `gal_table_read' and `gal_table_write' will read and write data to
+        plain text, FITS ASCII and FITS Binary formats.
+
+     -- `gal_fits_img_read' and `gal_fits_img_write' can read a FITS image
+        to memory or write a FITS image from memory.
+
+  Gnuastro now defines a simple comment line format to keep basic
+  information in a plain text table, see the "Gnuastro text table format"
+  section of the book. In short for every column, a comment line like below
+  can be used to give a name, units, comments, or a type to a column. This
+  allows a FITS binary table for eample to be written to plain text and
+  converted back to binary without loosing any information (except for very
+  small floating point errors if not enough decimals are printed).
+
+      # Column N: NAME [UNIT, TYPE, BLANK] COMMENT
+
+  The new Fits program replaces the old Header program. But besides
+  reading/checking FITS header keywords, it can now also work on FITS
+  extensions/HDUs. For example with no options, it will list all the HDUs
+  in a FITS file along with basic information. It can copy a whole HDU to
+  another file, or delete a HDU from a FITS file. To get the previous
+  behavior of listing all the keywords in a FITS HDU, you can run it with
+  the `-p' option.
+
+  All programs now write data into the second HDU of a FITS file to allow a
+  clean first HDU. Note that following CFITSIO, HDU counting still starts
+  from zero, so FITS images and tables written by Gnuastro in a new file
+  can always be accessed with the `--hdu=1' option (which is now the
+  default).
+
+  If any program is run within a Git version controlled directory, a
+  `COMMIT' header keyword will be added to the created FITS files, see the
+  "Output headers" section of the book for a discussion on the usefulness
+  of this new feature.
+
+  Gnuastro now has a new program called "BuildProgram" to easily compile,
+  link and run a C program you have written with Gnuastro's libraries
+  without having to worry about which libraries (Gnuastro dependencies)
+  your program needs. For example if your C source code is called
+  `myprog.c', you can compile, link and run it with this simple command and
+  let GNU Libtool worry about the linking.
+
+      $ astbuildprog myprog.c
+
+** Removed programs or library features
+
+  The following program names have been renamed:
+
+    ImageCrop       --> Crop
+    ImageWarp       --> Warp
+    ImageStatistics --> Statistics
+    Header          --> Fits
+
+  SubtractSky has been removed. The Statistics program now has tools to
+  estimate the Sky value and it can be subtracted with Arithmetic.
 
-** Bug fixes
-
-  Using `%zu' to print `size_t' variables for clean build on 32-bit
-  systems.
-
-  Crash in Table for some operating systems due to memory is now fixed (bug
-  #49347).
-
-  Table's man-page is now created and installed (bug #49418).
-
-  Fixes in the documentation (sr #109170, bug #49419).
-
-** New programs, functions, or macros
+** New features
 
-  `gal_wcs_pixel_scale_deg'
-    Returns the pixel scale of the input WCS structure in degrees.
+  All Gnuastro programs that read and write tables can now do so in plain
+  text table format or in FITS ASCII or FITS Binary tables. Depending on
+  the filename or with the new `--tableformat' common option to all
+  programs.
 
-  `gal_wcs_angular_distance_deg'
-    This has replaced `gal_wcs_angular_distance'. That function would take
-    the angles in radians and also return its value in radians. Radians are
-    very inconvenient and uncommon in celestial coordinates, so its input
-    and output was modified to accept angles in degrees and also return the
-    angular distance in degrees.
+  The option management system in all Gnuastro programs has been completely
+  re-written with many new features, some of the most important ones are
+  listed below. For developers, you will notice that there is no more usage
+  of macros and adding new options has become much more easier.
 
-  `gnuastro/git.h'
-    New header for wrappers on the Git library.
+     -- All programs will now also look for a `gnuastro.conf' configuration
+        file to keep common options for all programs in every directory.
 
-  `gal_git_describe'
-    Bascially a wrapper on Git's describe functions to return the string
-    for easy inclusion.
+     -- The `--lastconfig' opion can be used on the command-line or in any
+        configuration file to stop parsing any further configuration files.
 
-  `gal_fits_datatype_size
-    Return the size (in bytes) of the given datatype.
+     -- The `--config' option can now be used to identify any arbitrary
+        file to be parsed as a configuration file. Any file that is given
+        to this option is parsed immediately.
 
-  `GAL_GNUASTRO_HAVE_PTHREAD_BARRIER'
-    This is a new name for the old `GAL_GNUASTRO_PTHREAD_BARRIER' to
-    conform better with other such marcos.
+     -- The `--printparams' option now also prints the short documentation
+        of each option (same description in `--help') after its value.
 
-  `GAL_GNUASTRO_HAVE_LIBGIT2'
-    Has a value of 1 if the host system has the Git library (`libgit2').
+  It is now possible to choose columns in tables based on column names as
+  well as column numbers. It is also possible to search for columns based
+  on searching in their units or comments. When using column numbers,
+  counting now starts from 1 (one), not 0 (as before). See the new
+  "Selecting table columns" section for more on these new features.
 
-** Removed programs, functions, macros
+  Where relevant, all programs now accept a `--type' option that you can
+  use to specify the numerical datatype of the output.
 
-  `gal_wcs_angular_distance'
-    See the explanation above for `gal_wcs_angular_distance_deg'.
+  With the new common option `--minmapsize', you can specify a minimum size
+  of an array (in bytes) to store data in SSD/HDD and not in RAM. This can
+  be instrumental when you are dealing with large datasets, or even smaller
+  ones, but when your RAM is getting full.
 
-  `GAL_GNUASTRO_PTHREAD_BARRIER'
-    See the explanation above for `GAL_GNUASTRO_HAVE_PTHREAD_BARRIER'.
-
-** New features
+  Making a log file is now optional and users have to explicitly ask for it
+  with the `--log' option.
 
   ImageWarp can now align the image with the celestial coordiates using the
   `--align' option.
@@ -63,14 +134,169 @@ GNU Astronomy Utilities NEWS                          -*- 
outline -*-
   the same order to create one warping matrix. By default the FITS
   coordinates will also be corrected.
 
+  Arithmetic now does all its operations in the native data type of the
+  dataset. Until now, it would convert the data internally to double
+  precision floating point, do the requested operation and write the data
+  back in the proper type. So this new implementation is much more
+  efficient than the 0.2 version.
+
+  Arithmetic now accepts two new classes of operators: type conversion
+  operators to all standard types and integer-only operators (all C
+  bit-wise operators and the modulo operator).
+
+  Arithmetic does the binary operators (e.g., plus or multiply) also in the
+  native type of the input dataset. Doing so for all the different
+  combinations of types, greatly slows down the initial compilation of
+  Gnuastro (after running `make'). So for every type there is now a
+  `--enable-bin-op-*' configure time option. Only for a the binary
+  operators, when the dataset's type isn't compiled, it will be converted
+  to a compiled type and then converted back.
+
+  Convolve now has the `--minsharpspec' option to specify the minimum
+  spectrum value to use in deconvolution (matching PSFs).
+
+  MakeProfile can now accept the radial function of profiles as
+  human-readable strings instead of a code for each profile (which was very
+  cryptic, although codes are also still acceptable). For example in the
+  profile column you can now write `sersic' instead of the code `1'. Also
+  compared to the previous versions where the codes started from `0', not
+  they start from `1'.
+
+  Crop can now read the name of the crop based on a value in a given in a
+  table column. This allows your object's IDs to be directly used as the
+  crop's file name.
+
+  ConvertType can also print the input dataset to the command-line
+  (`stdout'). To use this feature set the output filename to `stdout'.
+
+  The Statistics program can now read table columns as well as images and
+  do basic operations on them. It can also only work on a certain range of
+  the data. Alternatively, you can define the range on another reference
+  column, but use values of the main column.
+
+  Statistics can also do all its single-valued measurements on a
+  tessellation (tile grid) over the input dataset.
+
+  Statistics now has features to estimate the Sky value on the image using
+  the mode's quantile similar to what NoiseChisel does to find its initial
+  threshold.
+
+  Statistics now has several new single valued calculations: `--quantile',
+  `--quantfunc' (quantile function), `--mode', `--modequant', `--modesym',
+  and `--modesymvalue'.
+
+  When in WCS mode, Crop can still only work on aligned images. However,
+  very small floating point errors in writing the WCS (for example 10e-13
+  degrees) are now acceptable. Until now these would cause Crop to complain
+  and abort.
+
+  NoiseChisel now has a new `--cleandilated' option to remove dilated
+  objects that have a low S/N (it is mainly useful on very clean or mock
+  images), for non-clean noise, it will result in a decrease of
+  completeness. With this new option, NoiseChisel will also print detection
+  S/N values when run with the `--checkdetsn' option.
+
 ** Changes in behavior
 
   When a 2 by 2 matrix is given to ImageWarp, the FITS pixel positions
   (which define the center of a pixel as an integer) are automatically
   implemented internally, see "Invoking ImageWarp" in the manual for more.
 
-  `gal_fits_write_keys_version' will now also write the Git commit
-  description when the Git library is installed on the host system.
+  To select column(s) by regular expression searching, the name now has to
+  be put in `/ /' (similar to AWK). If a value isn't in `/ /', the programs
+  will only select a column with the exact match.
+
+  Crop's `--section' syntax is now inclusive in both bounds.
+
+  Mask image options have been removed from all programs. Instead, all
+  programs can work directly on data with blank values. So when some pixels
+  must be masked, the Arithmetic program's `where' operator can be used to
+  select special pixels and set them to blank. In particular bit-wise
+  operations are now also available in Arithmetic which can be used with
+  Arithmetic's `where' also. Managing all these different choices in every
+  program would only confuse the user (with too many options).
+
+  Crop only checks if the center of a crop is filled when the crop was
+  defined by its center (for example with `--ra' and `--dec'). The verbose
+  outputs of Crop are also not cryptic 0s or 1s. The are human readable
+  text.
+
+  Crop doesn't have the separate `--imgmode' and `--wcsmode' options any
+  more. There is now a single `--mode' option which accepts values of `img'
+  or `wcs'.
+
+  Convolve's old `--frequency' and `--spatial' options have been removed
+  and are replaced by `--domain' which accepts values of `frequency' and
+  `spatial'.
+
+  Convolve's old `--viewfreqsteps' was changed to `--checkfreqsteps' to fit
+  with the general style of such check images in all Gnuastro's programs.
+
+  Warp's old `--nowcscorrection' option has been given a more clear name of
+  `--keepwcs'. With this option, Warp will not apply the warp the input's
+  WCS structure.
+
+  Warp's old `--maxblankfrac' option has been changed to
+  `--coveredfrac'. Until now, Warp would only look for the fraction of
+  input blank/NaN pixel area over the output pixel. But this would be
+  useless on the edges of the image. So the new `--coveredfrac' option
+  takes the acceptable fraction of output pixel area that must be covered
+  by input pixels in order to give that output pixel a value. You can use
+  this to set edge pixels that are not fully covered in the new grid to
+  blank and have a flat warped image.
+
+  Warp's old `--nofitscorrect' option has been changed to
+  `--centeroncorner' to be more clear. The new option is now more general
+  than before and also works on warping with a matrix, not just on modular
+  warpings.
+
+  MakeProfiles' old `--inputascanvas' is now called `--clearcanvas'.
+
+  Until now, MakeProfiles would abort with an error when the input columns
+  had blank values. But for masking, it might happen that you set a blank
+  magnitude. So this check has now been removed when reading the magnitude
+  column.
+
+  Statistics will not make a histogram and cumulative frequency files, or
+  calculate sigma-clipped results by default (with no options). It will
+  just print some basic information.
+
+  NoiseChisel's default value of the `--minskyfrac' option (new name for
+  the old `--minbfrac') is now 0.7 as opposed to 0.5. This will allow much
+  better estimation of noise properties (by default). It may be slightly
+  too high for a crowded field, but the users can change it on the
+  command-line (or in a configuration file) for such datasets.
+
+  When NoiseChisel is run with any of the `--check' options, it will abort
+  after all the check images have been created. This is very useful for
+  checking your parameters until each step and not be distracted (or have
+  to wait) for later steps to finish.
+
+** Bug fixes
+
+  Using `%zu' to print `size_t' variables for clean build on 32-bit
+  systems.
+
+  Crash in Table for some operating systems due to memory is now fixed (bug
+  #49347).
+
+  Table's man-page is now created and installed (bug #49418).
+
+  Fixes in the documentation (sr #109170, bug #49419).
+
+  Check for malloc returning valid pointer (bug #49459).
+
+  Segfault in mesh interpolation corrected (bug #49588).
+
+  Corrected bad status usage in calls to wcsp2s and wcss2p (bug #49752).
+
+  Strickter checking in Crop's polygon point list (bug #48978).
+
+  Correction in alignment and getting pixel scale (bug #50072).
+
+  Decomposing PCi_j and CDELTi matices in output WCS (bug #50073).
+
+  Using image naxes[n] when checkcenter is larger (bug #50099).
 
 
 * Noteworthy changes in release 0.2 (library 0.0.0) (2016-10-03) [stable]
diff --git a/doc/gnuastro.texi b/doc/gnuastro.texi
index 92e1337..2a9571b 100644
--- a/doc/gnuastro.texi
+++ b/doc/gnuastro.texi
@@ -1640,13 +1640,13 @@ can't be modeled (like stars that are always a point) 
easily. So there
 is no better way to distinguish them than to visually inspect them and
 see if it is possible to classify these nebulae or not.
 
-Hubble has stored all the FITS images of the objects he wants to
-visually inspect in his @file{/mnt/data/images} directory. He has also
-stored his catalog of extra Galactic nebulae in
+Hubble has stored all the FITS images of the objects he wants to visually
+inspect in his @file{/mnt/data/images} directory. He has also stored his
+catalog of extra-galactic nebulae in
 @file{/mnt/data/catalogs/extragalactic.txt}. Any normal user on his
-GNU/Linux system (including himself) only has read access to the
-contents of the @file{/mnt/data} directory. He has done this by
-running this command as root:
+GNU/Linux system (including himself) only has read access to the contents
+of the @file{/mnt/data} directory. He has done this by running this command
+as root:
 
 @example
 # chmod -R 755 /mnt/data
@@ -1685,19 +1685,16 @@ $ ls
 
 Hubble has multiple images in @file{/mnt/data/images}, some of his targets
 might be on the edges of an image and so several images need to be stitched
-to give a good view of them. Also his extra Galactic targets belong to
+to give a good view of them. Also his extra-galactic targets belong to
 various pointings in the sky, so they are not in one large
 image. Gnuastro's Crop is just the program he wants. The catalog in
 @file{extragalactic.txt} is a plain text file which stores the basic
-information of all his known 200 extra Galactic nebulae. In its second
+information of all his known 200 extra-galactic nebulae. In its second
 column it has each object's Right Ascension (the first column is a label he
 has given to each object) and in the third the object's declination.
-Having read the Gnuastro manual, he knows that all counting is done
-starting from zero, so the RA and Dec columns have number 1 and 2
-respectively.
 
 @example
-$ astcrop --racol=1 --deccol=2 /mnt/data/images/*.fits     \
+$ astcrop --racol=2 --deccol=3 /mnt/data/images/*.fits     \
              /mnt/data/catalogs/extragalactic.txt
 Crop started on Tue Jun  14 10:18:11 1932
   ---- ./4_crop.fits                  1 1
@@ -1716,11 +1713,11 @@ Crop finished in:  2.429401 (seconds)
 @cindex Asynchronous thread allocation
 @noindent
 Hubble already knows that thread allocation to the the CPU cores is
-asynchronous, so each time you run it the order of which job gets done
+asynchronous. Hence each time you run it, the order of which job gets done
 first differs. When using Crop the order of outputs is irrelevant since
 each crop is independent of the rest. This is why the crops are not
-necessarily created in the same input order. He is content with the default
-width of the outputs (which he inspected by running @code{$ astcrop
+necessarily created in the same input order. He is satisfied with the
+default width of the outputs (which he inspected by running @code{$ astcrop
 -P}). If he wanted a different width for the cropped images, he could do
 that with the @option{--wwidth} option which accepts a value in
 arc-seconds.  When he lists the contents of the directory again he finds
@@ -1732,38 +1729,36 @@ $ ls
 @end example
 
 @cindex GNU Parallel
-The FITS image format was not designed for viewing, but mainly for
-accurate storing of the data. So he chooses to convert the cropped
-images to a more common image format to view them more quickly and
-easily through standard image viewers (which load much faster than
-FITS image viewer). JPEG is one of the most recognized image formats
-that is supported by most image viewers. Fortunately Gnuastro has just
-such a tool to convert various types of file types to and from each
-other: ConvertType. Hubble has already heard of GNU Parallel from one
-of his colleagues at Mount Wilson Observatory. It allows multiple
-instances of a command to be run simultaneously on the system, so he
-uses it in conjunction with ConvertType to convert all the images to
-JPEG.
+The FITS image format was not designed for efficient/fast viewing, but
+mainly for accurate storing of the data. So he chooses to convert the
+cropped images to a more common image format to view them more quickly and
+easily through standard image viewers (which load much faster than FITS
+image viewer). JPEG is one of the most recognized image formats that is
+supported by most image viewers. Fortunately Gnuastro has just such a tool
+to convert various types of file types to and from each other:
+ConvertType. Hubble has already heard of GNU Parallel from one of his
+colleagues at Mount Wilson Observatory. It allows multiple instances of a
+command to be run simultaneously on the system, so he uses it in
+conjunction with ConvertType to convert all the images to JPEG.
 @example
 $ parallel astconvertt -ojpg ::: *_crop.fits
 @end example
 
 @pindex eog
 @cindex Eye of GNOME
-For his graphical user interface Hubble is using GNOME which is the
-default in most distributions in GNU/Linux. The basic image viewer in
-GNOME is the Eye of GNOME, which has the executable file name
address@hidden @footnote{Eye of GNOME is only available for users of
-the GNOME graphical desktop environment which is the default in most
-GNU/Linux distributions. If you use another graphical desktop
-environment, replace @command{eog} with any other image
-viewer.}. Since he has used it before, he knows that once it opens an
-image, he can use the @key{ENTER} or @key{SPACE} keys on the keyboard
-to go to the next image in the directory or the @key{Backspace} key to
-to go the previous image. So he opens the image of the first object
-with the command below and with his cup of coffee in his other hand,
-he flips through his targets very fast to get a good initial
-impression of the morphologies of these extra Galactic nebulae.
+For his graphical user interface Hubble is using GNOME which is the default
+in most distributions in GNU/Linux. The basic image viewer in GNOME is the
+Eye of GNOME, which has the executable file name @command{eog}
address@hidden of GNOME is only available for users of the GNOME graphical
+desktop environment which is the default in most GNU/Linux
+distributions. If you use another graphical desktop environment, replace
address@hidden with any other image viewer.}. Since he has used it before,
+he knows that once it opens an image, he can use the @key{ENTER} or
address@hidden keys on the keyboard to go to the next image in the directory
+or the @key{Backspace} key to to go the previous image. So he opens the
+image of the first object with the command below and with his cup of coffee
+in his other hand, he flips through his targets very fast to get a good
+initial impression of the morphologies of these extra-galactic nebulae.
 
 @example
 $ eog 1_crop.jpg
@@ -1783,16 +1778,15 @@ arms which he associates with code 2 and finally there 
seems to be a
 class of nebulae in between which appear to have a disk but no spiral
 arms, he gives them code 3.
 
-Now he wants to know how many of the nebulae in his extra Galactic
-sample are within each class. Repeating the same process above and
-writing the results on paper is very time consuming and prone to
-errors. Fortunately Hubble knows the basics of GNU Bash shell
-programming, so he writes the following short script with a loop to
-help him with the job. After all, computers are made for us to operate
-and knowing basic shell programming gives Hubble this ability to
-creatively operate the computer as he wants. So using GNU
address@hidden can be done with any text editor} (his favorite
-text editor) he puts the following text in a file named
+Now he wants to know how many of the nebulae in his extra-galactic sample
+are within each class. Repeating the same process above and writing the
+results on paper is very time consuming and prone to errors. Fortunately
+Hubble knows the basics of GNU Bash shell programming, so he writes the
+following short script with a loop to help him with the job. After all,
+computers are made for us to operate and knowing basic shell programming
+gives Hubble this ability to creatively operate the computer as he
+wants. So using GNU address@hidden can be done with any text editor}
+(his favorite text editor) he puts the following text in a file named
 @file{classify.sh}.
 
 @example