Gnuastro 0.14 released

[Mohammad Akhlaghi]

Dear all,

I am happy to announce the availability of Gnuastro 0.14. For the full
list of added and changed/improved features, see the excerpt of the
NEWS file for this release in [1] below.

Gnuastro is an official GNU package, consisting of various
command-line programs and library functions for the manipulation and
analysis of (astronomical) data. All the programs share the same basic
command-line user interface (modeled on GNU Coreutils). For the full
list of Gnuastro's library, programs, and a comprehensive general
tutorial (recommended place to start using Gnuastro), please see the
links below respectively:

https://www.gnu.org/s/gnuastro/manual/html_node/Gnuastro-library.html
https://www.gnu.org/s/gnuastro/manual/html_node/Gnuastro-programs-list.html
https://www.gnu.org/s/gnuastro/manual/html_node/General-program-usage-tutorial.html

The most prominent new feature may be the new Query program (called
with 'astquery'). It allows you to directly query many large
astronomical data centers (currently VizieR, NED, ESA and ASTRON) and
only download your selected columns/rows. For example with the command
below you can download the RA, Dec and Parallax of all stars in the
Gaia eDR3 dataset (from VizieR) that overlap with your
'image.fits'. You just have to change '--dataset' to access any of the
+20,000 datasets within VizieR for example! You can also search in the
dataset metadata from the command-line, and much more.

  astquery vizier --dataset=gaiaedr3 --overlapwith=image.fits \
           --column=RAJ2000,DEJ2000,Plx

See the new "Query" section in the Gnuastro book for more:

https://www.gnu.org/software/gnuastro/manual/html_node/Query.html

Here is the compressed source and the GPG detached signature for this
release. To uncompress Lzip tarballs, see [2]. To check the validity
of the tarballs using the GPG detached signature (*.sig) see [3]:

  https://ftp.gnu.org/gnu/gnuastro/gnuastro-0.14.tar.lz    (3.6MB)
  https://ftp.gnu.org/gnu/gnuastro/gnuastro-0.14.tar.gz    (5.6MB)
  https://ftp.gnu.org/gnu/gnuastro/gnuastro-0.14.tar.gz.sig (833B)
  https://ftp.gnu.org/gnu/gnuastro/gnuastro-0.14.tar.lz.sig (833B)

Here are the MD5 and SHA1 checksums:

30d77e2ad1c03d4946d06e4062252969  gnuastro-0.14.tar.gz
f3ddbc4b5763ec2742f9080d42b69ed3  gnuastro-0.14.tar.lz
cfbcd4b9ae1c5c648c5dc266d638659f0117c816  gnuastro-0.14.tar.gz
4e4c6b678095d2838f77b2faae584ea51df2d33c  gnuastro-0.14.tar.lz

I am very grateful to (in alphabetic order) Pedram Ashofteh Ardakani,
Thérèse Godefroy, Raúl Infante-Sainz, Sachin Kumar Singh, Samane Raji
and Zahra Sharbaf for directly contributing to the source of Gnuastro
since the last alpha-release. It is great that in this release we have
an equal gender balance in the contributors. I sincerely hope this can
continue in the next release :-).

I am also very grateful to (in alphabetic order) Antonio Diaz Diaz,
Paul Eggert, Andrés García-Serra Romero, Thérèse Godefroy, Bruno
Haible, Martin Kuemmel, Javier Licandro, Alireza Molaeinezhad, Javier
Moldon, Sebastian Luna Valero, Samane Raji, Alberto Madrigal, Carlos
Morales Socorro, Francois Ochsenbein, Joanna Sakowska, Zahra Sharbaf,
Sachin Kumar Singh, Ignacio Trujillo and Xiuqin Wu for their very
useful comments, suggestions and bug fixes that have now been
implemented in Gnuastro since the last alpha-release.

If any of Gnuastro's programs or libraries are useful in your work,
please cite _and_ acknowledge them. For citation and acknowledgment
guidelines, run the relevant programs with a `--cite' option (it can
be different for different programs, so run it for all the programs
you use). Citations _and_ acknowledgments are vital for the continued
work on Gnuastro, so please don't forget to support us by doing so.

This tarball was bootstrapped (created) with the tools below. Note
that you don't need these to build Gnuastro from the tarball, these
are the tools that were used to make the tarball itself. They are only
mentioned here to be able to reproduce/recreate this tarball later.
  Texinfo 6.7
  Autoconf 2.70
  Automake 1.16.2
  Help2man 1.47.17
  ImageMagick 7.0.10-59
  Gnulib v0.1-4396-g3b732e789
  Autoconf archives v2019.01.06-98-gefa6f20

The dependencies to build Gnuastro from this tarball on your system
are described here:
  https://www.gnu.org/s/gnuastro/manual/html_node/Dependencies.html

Best wishes,
Mohammad

--
Postdoctoral research fellow,
Instituto de Astrofísica de Canarias (IAC),
Calle Vía Láctea, s/n, E38205,
San Cristóbal de La Laguna, Tenerife, Spain.





[1] Noteworthy changes in release 0.14 (library 12.0.0)

** New features

  Book:
   - Tutorials improved with more useful usage examples, for example
     the "General program usage tutorial" tutorial now includes the
     merging of table columns into one table and the creation of a
     color-magnitude diagram. Also the "Detecting large extended
     targets" now accounts for the recent improvements in NoiseChisel.
   - New sub-section on "Memory management" in the "Common program
     behavior" chapter. It fully describes how to optimally deal with large
     datasets that may exceed your system's RAM.
   - Examples and better description added to many operators in the
     "Arithmetic operators" subsection of the Arithmetic program's section.

  New program:
   - Query ('astquery') is a new program to query to external datasets
     and retrieve the resulting datasets from the command-line. It is
     possible to get list of datasets or column names from the
     databases. The basic spatial query (finding objects in the
     vicinity of a certain point) can be managed without knowing the
     query language of the database: with basic options (like
     '--center' with '--radius' or '--width'), or an input image that
     has WCS. Currently Query supports VizieR (containing +20500
     datasets, making it the largest database in astronomy), NED, as
     ESA's Gaia database and ASTRON. See the new "Query" section of
     the book (under the "Data containers" chapter) for more. Adding
     any new data base that supports the Table Access Protocol (TAP)
     of the International Virtual Observatory Alliance (IVOA,
     https://ivoa.net), is trivial.

  All programs:
   - Plain text table inputs can have floating point columns that are in
     sexagesimal format of '_h_m_s' or '_d_m_s' (where '_' is a
     number). These are the classical format to respectively represent
     Right Ascension (RA) and Declination (Dec). They will be directly read
     as a single floating point number (in units of degrees) into
     memory. Therefore the same column of a plain-text table, can be
     degrees in some rows and sexagesimal in others. Besides large tables,
     with this feature, conversion to sexagesimal coordinates to degrees
     becomes very easy, for example:
         echo "7h34m35.5498 31d53m14.352s" | asttable
     Recall that the inverse can also be done with the more general column
     arithmetic:
         echo "113.64812416667 31.88732" \
              | asttable -c'arith $1 degree-to-ra $2 degree-to-dec'
   - If input is a HEALpix grid (1D table column that represents the 2D
     spherical representation of datasets), the programs will print a
     warning, suggesting to use the 'HPXcvt' utility of WCSLIB.

  Arithmetic:
   - New operators:
     - 'interpolate-maxofregion': interpolate connected blank regions with
       the maximum value that is immediately touching it. This can be used
       to fill the blank centers of saturated stars for example.
     - 'interpolate-minofregion': similar to 'interpolate-maxofregion', but
       for the minimum.
     - 'makenew': new operator to create an empty (zero-valued) new dataset
       with given dimension and size (given as operands).

  Crop:
   --primaryimghdu: Write the final cropped image into the primary (or
     0-th) extension of the output FITS file, so the output only has
     one extension.

  Fits:
   - New '--skycoverage' option will report the area of the input
     image in RA/Dec, both in units of center/width, and box
     minimum/maximum format.

  MakeCatalog:
   --maximum: maximum value of labeled regions pixels (clump/object).
   --areaarcsec2: area of labeled region (clump/object) in arcsec^2.
   --surfacebrightness: the surface brightness of the labeled region.
   --fwhm: observed FWHM in pixels (non-parametric), along the major axis.
   --halfmaxarea: number of pixels with a value larger than half the 
maximum.
   --halfmaxradius: radius of region that is larger than half the maximum.
   --halfmaxsum: sum of pixels with a value larger than half the maximum.
   --halfmaxsb: surf. brightness within half of the maximum.
   --fracmax: fractions to use in '--fracmaxarea1' or '--fracmaxarea2'.
   --fracmaxsum1: sum of pixels brighter than first given fraction of max.
   --fracmaxsum2: sum of pixels brighter than second given fraction of max.
   --fracmaxarea1: number of pixels brighter than first given fraction 
of max.
   --fracmaxarea2: number of pixels brighter than second given fraction 
of max.
   --fracmaxradius1: radius derived from '--fracmaxarea1'.
   --fracmaxradius2: radius derived from '--fracmaxarea2'.
   --halfsumsb: Surface brightness within area reported by '--halfsumarea'.
   --halfsumarea: area containing half of the summed object or clump 
values.
   --halfsumradius: radius derived from '--halfsumarea', underestimates 
r_e.
   --areaminv: the number of pixels that are equal to the minimum value.
   --areamaxv: the number of pixels that are equal to the maximum value.
   - New columns to return the position of pixel with minimum or maximum
     value: '--minvx', '--maxvx', '--minvy', '--maxvy', '--minvz',
     '--maxvz'.

  MakeNoise:
   --bgisbrightness: new option to say that the value of '--background'
     (used to simulate Poisson noise) should be interpreted as brightness,
     not magnitude.

  MakeProfiles:
   - It is now possible to make any custom radial profile with the 'custom'
     profile (with code '8'). A table should be given to the new
     '--customtable' option which will define each radial interval and the
     value to use for that radial interval. See the description of
     '--customtable' for more.

  Statistics:
   - 2D histograms can now be built as a FITS image with a linear WCS that
     contains axis values and box size. This allows using the power of FITS
     viewers for plotting/inspecting distributions of two columns in a
     table relative to each other (for example color-magnitude plots). You
     can also convert these 2D histogram images to PDF or JPEG with
     Gnuastro's ConvertType to directly use in your papers/reports (see the
     newly added "2D histogram as an image" section of the book for more).

  Table:
   - New '--noblank' option will remove all rows in output table that have
     at least one blank value in the specified columns. For example if
     'table.fits' has blank values (NaN in floating point types) in the
     'magnitude' and 'sn' columns, with '--noblank=magnitude,sn', you can
     be sure that all rows with blank values in these columns have been
     removed.
   - New trigonometric operators for column-arithmetic (inputs in units of
     degrees): 'sin', 'cos' and 'tan'. Their inverse trigonometric (outputs
     in units of degrees) have also been added: 'asin', 'acos' and 'atan'.
     The 'atan2' operator (inverse tangent that preserves the quadrant, see
     its description in the book) is also now available.
   - New hyperbolic operators for column-arithmetic: 'sinh', 'cosh' and
     'tanh'. Their inverse has also been added: 'asinh', 'acosh' and
     'atanh'.

  Library:
   - GAL_ARITHMETIC_OP_MAKENEW: new 'makenew' operator.
   - gal_binary_connected_adjacency_list: finding connected components
     without a square adjacency matrix (which can consume major RAM).
   - gal_blank_flag_remove: Remove all flagged elements in a dataset.
   - gal_blank_remove_rows: remove all rows that have at least one blank.
   - gal_dimension_dist_elliptical: Elliptical dist. of a point from 
center.
   - gal_fits_hdu_is_healpix: Return 1 if HDU is a HEALpix grid.
   - gal_fits_hdu_datasum: calculate DATASUM of given HDU in given FITS 
file.
   - gal_fits_hdu_datasum_ptr: calculate DATASUM of opened FITS file 
pointer.
   - gal_fits_key_list_comment_add: add a COMMENT keyword to the 
keyword list.
   - gal_fits_key_list_comment_add_end: add a COMMENT keyword to end of 
list.
   - gal_pointer_allocate_ram_or_mmap: allocate in RAM or memory-mapped 
file.
   - gal_pointer_mmap_free: "free" (actually delete) the memory-mapped 
file.
   - gal_wcs_create: create WCSLIB-compatible WCS from raw values.
   - gal_wcs_coverage: Return the sky coverage of given image HDU.
   - gal_wcs_dimension_name: return the name of the requested WCS dim.

** Removed features
   - Nothing has been removed from Gnuastro in this release.

** Changed features

  All programs:
   - Memory management: Until now, an internal array was only allocated in
     the RAM when its size was smaller (in bytes) than the value given to
     the '--minmapsize' option. But this was annoying/buggy when the system
     has enough RAM to keep large files. From this version, all Gnuastro
     programs will first attempt to write the array in RAM, only when it
     fails (there is no more RAM left), will they use a memory-mapped file
     (which can dramatically slow down the program). Please see the newly
     added "Memory management" section of the book for a complete
     explanation of Gnuastro's new memory management strategy.
   - When an array needs to be memory-mapped (read into a file on HDD/SSD,
     not RAM, usually due to RAM limitations), it is written in a
     'gnuastro_mmap' directory of the running directory, not the hidden
     '.gnuastro_mmap' directory. Since the files in this directory are
     usually large, it is better that the user sees them by default (in
     case the program crashes and they aren't deleted).
   --interpnumngb: the default value has been increased to 15 (from 9). The
     reason for this is that we now have a more robust outlier removal
     algorithm (see description under "NoiseChisel & Statistics").

  Crop:
   - When cropping a single image in WCS mode, there is no longer any
     limitation on the WCS. Until now for all WCS mode crops, it was
     necessary for the WCS to be aligned to the celestial coordinates. But
     from this version, this is only necessary when cropping from many
     files (and stitching them together where necessary). For WCS-mode
     crops of a single image, any WCS that is recognized by WCSLIB is fine.

  Fits:
   - The '--pixelscale' option also prints the pixel area (for 2D inputs,
     or 2D slices of 3D inputs) and the voxel volume (for 3D inputs). Until
     now, it would only print the pixel scale along each dimension.
   - When printing FITS file HDU information (no options given), a new
     "Comments" column may be printed for each HDU in the end of the line.
     It will be printed if special situations are found (for example a 2D
     HEALPix grid, that is usually stored as a 1D array/column).

  NoiseChisel & Statistics:
   - New algorithm used to reject outlying tiles. In NoiseChisel this is
     done when estimating the quantile threshold, the pseudo-detection
     threshold and the final Sky value. In Statistics, its just the Sky
     value. Unlike the previous method that used the global distribution of
     tile values to identify outliers, the new algorithm uses a relative
     measure. See the book for more. Since outliers are now rejected more
     robustly, the default value of '--meanmedqdiff' has been increased to
     0.01 (was 0.005) and '--outliersigma' has been decreased to '5' (was
     10). Also 'smoothwidth' has been increased from '3' to '5' to have
     smoother tessellation values.

  Statistics:
   - The '--histogram2d' now takes a string argument: either 'image' or
     'table'. For the old behavior please use '--histogram2d=table'. See
     the new features above for the 'image' mode.

  Table:
   - Column arithmetic operators 'degree-to-ra' and 'degree-to-dec' will
     return the sexagesimal format of '_h_m_s' and '_d_m_s'
     respectively. Until this version, they would both use colons as
     delimiters ('_:_:_').

  Library:
   - gal_pointer_mmap_allocate: new name for 'gal_pointer_allocate_mmap'.
   - gal_threads_dist_in_threads: now accounts for billions of threads,
     thus includes memory management options.
   - gal_threads_spin_off: now accounts for memory management.
   - gal_units_degree_to_ra: new 'usecolon' argument to optionally format
     output string with colons as delimiters ('_:_:_'). When this option is
     zero, the string will be in the '_h_m_s' format.
   - gal_units_degree_to_dec: similar to 'gal_units_degree_to_ra', but when
     'usecolon' is zero, the string will be in the '_d_m_s' format.

** Bugs fixed
  bug #59017: Segment's object IDs are not thread-safe (i.e., 
reproducible).
  bug #59105: Column arithmetic operator degree-to-ra, returning to dec.
  bug #59136: Makeprofiles with --replace is not thread-safe.
  bug #59155: Match cannot find the proper row when coordinates have NaN.
  bug #59371: MakeCatalog crash with clumps on non-contiguous object 
labels.
  bug #59400: CosmicCalculator fails --printparams when redshift isn't 
given.
  bug #59459: Unclear WCS when both PC and CD exist in input, but conflict.
  bug #59625: MakeProfiles uses last --kernel, if it is called more 
than once.
  bug #59700: Segment's excessive RAM usage when many clumps over a 
detection.
  bug #59765: MakeCatalog crash when to-be-subtracted Sky is a 
single-element
              per tile tessellation (e.g., NoiseChisel's 
'--oneelempertile').





[2] Lzip has better compression ratio and archival features compared
to the `.gz' or `.xz' formats. Therefore Gnuastro's alpha/test
releases are only in this format, but for historical reasons we also
include `.gz' tarballs in the official releases. If you don't have
Lzip (you can check with `lzip --version' command), download and
install it from its webpage:

  https://www.nongnu.org/lzip/lzip.html

If Lzip is present and you use GNU Tar, then the single command below
should uncompress and un-pack the tarball:

  $ tar xf gnuastro-0.14.tar.lz

If the command above doesn't work, you have to un-compress and un-pack
it with two separate commands (or use a pipe to feed the output of the
first into the second: `lzip -cd gnuastro-0.14.tar.lz | tar -xf -'):

  $ lzip -d gnuastro-0.14.tar.lz
  $ tar xf gnuastro-0.14.tar





[3] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify gnuastro-0.14.tar.gz.sig

If that command fails because you don't have the required public key,
then run this command to import it:

  gpg --keyserver keys.gnupg.net --recv-keys 71E899012D174B66

and rerun the 'gpg --verify' command.