[gnuastro-commits] master 65f87a3 3/3: Completed one full edit of the tutorial

[Mohammad Akhlaghi]

branch: master
commit 65f87a3febc1540e358d8f7d9500294cacf4fe2b
Author: Mohammad Akhlaghi <address@hidden>
Commit: Mohammad Akhlaghi <address@hidden>

    Completed one full edit of the tutorial
    
    Along with the previous commits, the tutorial section has been re-read and
    re-checked to be more clear.
---
 doc/gnuastro.texi | 301 +++++++++++++++++++++++++++++++++++-------------------
 1 file changed, 194 insertions(+), 107 deletions(-)

diff --git a/doc/gnuastro.texi b/doc/gnuastro.texi
index 70e7d37..1330b83 100644
--- a/doc/gnuastro.texi
+++ b/doc/gnuastro.texi
@@ -2839,73 +2839,83 @@ $ mkdir noisechisel
 $ astnoisechisel flat-ir/xdf-deep.fits -onoisechisel/xdf-deep.fits
 @end example
 
-In some cases, you might want to use a different kernel (and not the
-default value). To do that, you can use MakeProfiles (see
address@hidden) in the following manner to build a 2D Gaussian kernel
-with a FWHM of 4 pixels that extends 3 times the FWHM. You can then feed
-this new kernel into NoiseChisel.
-
address@hidden
-$ astmkprof --kernel=gaussian,3,5 --oversample=1 -okernel-g-3-5.fits
-$ astnoisechisel flat-ir/xdf-deep.fits --kernel=kernel-g-4-3.fits    \
-                 --output=lab-my-kernel.fits
address@hidden example
+Read what NoiseChisel writes on the command-line. The curious thing you
+will notice from the information NoiseChisel prints for you on the
+command-line is that while there were more than 3000 pseudo detections to
+find the pseudo-detection S/N, there was only slightly more than 100 clumps
+to find the false clump S/N. We will see what caused this below.
 
-But we'll stick with the default parameters for the rest of this
-processing.  NoiseChisel's output is a single file containing multiple
-extensions. You can get basic information about the extensions in a FITS
-file with Gnuastro's Fits program (see @ref{Fits}) as shown below. You will
-see that it contains 6 extensions with the first one being blank (counted
-as zero).
+NoiseChisel's output is a single file containing multiple extensions. You
+can get basic information about the extensions in a FITS file with
+Gnuastro's Fits program (see @ref{Fits}) as shown below. You will see that
+it contains 6 extensions with the first one being blank (counted as zero).
 
 @example
 $ astfits noisechisel/xdf-deep.fits
 @end example
 
 NoiseChisel puts some general information on its outputs in the FITS header
-of the respective section. To see them, you can again use Gnuastro's Fits
-program and specify which extension/HDU that contains your expected
-keywords. You can also use the extension number (as listed in the output
-above) for example @option{-h2}.
+of the respective extension. To see the full list of keywords, you can
+again use Gnuastro's Fits program and specify the extension/HDU. You can
+also use the extension number (as listed in the output above) for example
address@hidden instead of @option{-hOBJECTS}.
 
 @example
 $ astfits noisechisel/xdf-deep.fits -hOBJECTS
 @end example
 
-The @code{NUMLABS} keyword contains the number of objects that was found by
-NoiseChisel with the given parameters. But in the general output of the
-command above, you see the full header, which contains many keywords that
-you are not interested in at this point. To make things easier, you can
-pipe the output of the command above into @code{grep} (a program for
-matching lines and is available on almost all Unix-like systems).
address@hidden GNU Grep
+The @code{NUMLABS} keyword in NoiseChisel's @code{OBJECTS} extension
+contains the number of objects that was found by NoiseChisel in that
+run. Try to find it in the outputs of the command above. To make things
+easier, you can pipe the output of the command above into @code{grep} (a
+program for matching lines which is available on almost all Unix-like
+operating systems).
 
 @example
 $ astfits noisechisel/xdf-deep.fits -hOBJECTS | grep NUMLABS
 @end example
 
-If you just want the number and not the full FITS keyword line, you can use
-AWK. In the example below, AWK will print the third word (separated by
-white space characters) in any line with a first column value
address@hidden You can also try the command above for the third HDU
-(called @code{CLUMPS}) to see the number of clumps.
address@hidden GNU Grep
+If you just want the value of the keyword and not the full FITS keyword
+line, you can use AWK. In the example below, AWK will print the third word
+(separated by white space characters) in any line that has a first column
+value of @code{NUMLABS}. You can also try the command above for the third
+HDU (called @code{CLUMPS}) to see the number of clumps.
 
 @example
 $ astfits noisechisel/xdf-deep.fits -h2 | awk '$1=="NUMLABS" @{print 
address@hidden'
 @end example
 
-Now, you can open the image with SAO DS9 and see the actual outputs. Just
-note that since there are multiple extensions, the easiest way is to open
-the file as a ``Multi-extension data cube'' through the command above. If
-you use GNOME for your GUI, please see @ref{Viewing multiextension FITS
-images} to open DS9 in multi-extension cube mode.
+Grep and AWK are simple, but very powerfull software for processing text
+files. Learning them properly can greatly simplify your processing and
+improve your creativity. When you get time, it is highly recommended to go
+through their manuals and learn them. You don't have to read the whole
+manual at once, they both start with great generic introductions to get you
+going fast. The most common implementation of both is from GNU. Like almost
+all GNU software, both GNU Grep and GNU AWK have wonderful manuals which
+come with the program for free can either read online or on the
+command-line.
+
address@hidden
+$ info awk
+$ info grep
address@hidden example
+
+You can now open NoiseChisel's output with SAO DS9 and visually inspect
+them. Just note that since there are multiple extensions, the easiest way
+to view the whole file is to open it as a ``Multi-extension data cube''
+through the command below. If you use GNOME for your GUI, please see
address@hidden multiextension FITS images} to open DS9 in multi-extension
+cube mode by default when using the GUI (double clicking on the file).
 
 @example
 $ ds9 -mecube noisechisel/xdf-deep.fits
 @end example
 
-Using Gnuastro's Fits program, you can also pull out a specific HDU. So if
-you want to have a FITS file with only the detected objects, you can run
-this command:
+Using Gnuastro's Fits program, you can also copy a HDU from one FITS file
+to another (for more, see @ref{HDU manipulation}). So if you want to have a
+FITS file with only the detected objects, you can run this command:
 
 @example
 $ astfits noisechisel/xdf-deep.fits --copy=OBJECTS -oobjects.fits
@@ -2913,32 +2923,100 @@ $ astfits noisechisel/xdf-deep.fits --copy=OBJECTS 
-oobjects.fits
 
 One good way to see if you have missed any signal is to mask all the
 detected pixels. For this, you can use Gnuastro's Arithmetic program (in
-particular its @code{where} operator as shown below. With this command, all
-pixels that have a value larger than zero in the @code{OBJECTS} extension
-will be set to NaN and the output will be written in
address@hidden If you change the @code{gt} (for ``greater than'')
-operator to @code{eq} (for ``equal''), all the un-detected (sky) pixels
-will be masked and you can see the detections.
+particular its @code{where} operator as shown below). With this command,
+all pixels that have a value larger than zero in the @code{OBJECTS}
+extension will be set to NaN in the NoiseChisel input and the output will
+be written in @file{det-masked.fits}. If you change the @code{gt} (for
+``greater than'') operator to @code{eq} (for ``equal''), all the
+un-detected (sky) pixels will be masked and you can see the detections.
 
 @example
 $ astarithmetic noisechisel/xdf-deep.fits                         \
                 noisechisel/xdf-deep.fits 0 gt nan where -h1 -h2  \
-                --output=det-masked.fits
+                --output=nc-masked.fits
 @end example
 
 NoiseChisel can produce ``Check images'' to help you visualize how it does
 its processing. You can see all the check images it produces with this
-command. The check images are also mult-extension FITS files, so after each
-check image is produced, open it like NoiseChisel's output and flip through
-the extensions to see its processing in detail. It is strongly encouraged
-to play with the different parameters and use the check images to see how
-the change of option values affects the final result. Try running the
-second command multiple times with different
+command.
 
 @example
 $ astnoisechisel --help | grep check
 @end example
 
+The check images are also multi-extension FITS files, so after each check
+image is produced, open it like NoiseChisel's output and flip through the
+extensions to see its processing in detail. It is @emph{strongly}
+encouraged to play with the different parameters and use the respective
+check images to see which step is affected by your change.
+
+In some cases, you might want to use a different kernel (and not the
+default one). To do that, you can use MakeProfiles (see @ref{MakeProfiles})
+in the following manner to build a 2D Gaussian kernel with a FWHM of 3
+pixels that extends 5 times the FWHM. This new kernel can later be fed into
+NoiseChisel with the @option{--kernel} option.
+
address@hidden
+$ astmkprof --kernel=gaussian,3,5 --oversample=1 -okernel-g-3-5.fits
+$ astnoisechisel flat-ir/xdf-deep.fits --kernel=kernel-g-3-5.fits    \
+                 --output=nc-my-kernel.fits
address@hidden example
+
+After the initial run of NoiseChisel, there was one curious situation: the
+number of false clumps to find an S/N limit was very small number (given
+the extent of this image). This is bad, because we use quantiles in
+NoiseChisel and such a small number will result in a relatively large
+scatter. Since this is a segmenation issue, let's see why this happens with
address@hidden
+
address@hidden
+$ astnoisechisel flat-ir/xdf-deep.fits --checksegmentation
address@hidden example
+
+To help you get a result faster, when check images are requested,
+NoiseChisel doesn't finish its processing (unless you also call
address@hidden). NoiseChisel aborts with an explanation of
+why it stopped without finishing and the file name of the check image that
+it produced instead. The first five extensions are: the input image, the
+convolved image, the initially labeled detected regions, all the sky region
+(false) clumps, and those that will be used for S/N. The sky clumps are
+found over NoiseChisel's ``large tiles'' independently. But it is
+interesting how NoiseChisel has ignored most large tiles and only used the
+few that we see, mostly on the edge.
+
+The reason that almost all internal large tiles are ignored is that
+galaxies are extended. Thanks to the PSF (see @ref{PSF}), no object will
+have a sharp truncation. We have also not seen any abrupt break in the
+light profile of any galaxy: galaxies are always larger when we get deeper
+datasets. Therefore, in a noisy image, some light will always be left
+un-detected. To be less affected by this un-detected light, NoiseChisel has
+the @option{--minskyfrac} option (see @ref{General NoiseChisel options}):
+any tile that has a larger fraction of detected pixels will be ignored. So
+let's see what the default value of this option is (recall that with
address@hidden, you can list all the options with their values in Gnuastro's
+programs):
+
address@hidden
+$ astnoisechisel -P | grep minskyfrac
address@hidden example
+
address@hidden
+Try decreasing this value (keep the check image) and see the effect on the
+fraction of large tiles that will be used for finding false/sky
+clumps. Play a little with this parameter (give it different values and
+check the result).
+
address@hidden
+$ astnoisechisel flat-ir/xdf-deep.fits --minskyfrac=0.5           \
+                 --checksegmentation
address@hidden example
+
+The smaller the value to @option{--minskyfrac}, the more probable that
+un-detected light in the wings of galaxies will bias/affect the derived
+false clump S/N. So it is always important to find a good balance between a
+larger @option{--minskyfrac} while having a sufficient number of resulting
+clumps (to avoid scatter).
+
 NoiseChisel doesn't just find the labelled pixels, it also finds the Sky
 value and the Sky standard deviation. To generate a catalog of the colors,
 we will be using the labeled images from the NoiseChisel output on the deep
@@ -2952,15 +3030,15 @@ $ astnoisechisel flat-ir/xdf-f160w.fits 
-onoisechisel/xdf-f160w.fits
 @end example
 
 Now, we are ready to make a catalog. As shown above, we want the object and
-clump labels from the deep image and the input, Sky and Sky standard
-deviation images from each filter's image. So, we'll run MakeCatalog on
-NoiseChisel's output on each filter, but through the @option{--objectsfile}
-and @option{--clumpsfile} options, we'll tell it to read the objects and
-clumps labeled images from the deep image. For each filter, we'll ask for
-the ID, RA, Dec, Magnitude, Magnitude error and upper-limit magnitude (see
address@hidden measurement limits}) for each object or clump. To see a
-list of all the parameters that MakeCatalog can measure for you, run it
-with @option{--help} option.
+clump labels from the deep image. But the input, Sky and Sky standard
+deviation images come from each filter. So, we'll run MakeCatalog on
+NoiseChisel's output in each filter. However, through the
address@hidden and @option{--clumpsfile} options, we'll tell it to
+read the objects and clumps labeled images from NoiseChiel's output on the
+deep image. For each filter, we'll ask for the ID, RA, Dec, Magnitude and
+Magnitude error (see @ref{Quantifying measurement limits}). To see a list
+of all the parameters that MakeCatalog can measure for you, run it with
address@hidden option.
 
 @example
 $ mkdir catalog
@@ -2969,98 +3047,107 @@ $ astmkcatalog noisechisel/xdf-f105w.fits 
--zeropoint=26.27     \
                --objectsfile=noisechisel/xdf-deep.fits          \
                --clumpsfile=noisechisel/xdf-deep.fits           \
                --ids --ra --dec --magnitude --magnitudeerr      \
-               --upperlimitmag --output=catalog/xdf-f105w.fits
+               --output=catalog/xdf-f105w.fits
 
 $ astmkcatalog noisechisel/xdf-f160w.fits --zeropoint=25.94     \
                --objectsfile=noisechisel/xdf-deep.fits          \
                --clumpsfile=noisechisel/xdf-deep.fits           \
                --ids --ra --dec --magnitude --magnitudeerr      \
-               --upperlimitmag --output=catalog/xdf-f160w.fits
+               --output=catalog/xdf-f160w.fits
 @end example
 
 By default MakeCatalog's outputs are in FITS binary table
 format. MakeCatalog can also produce catalogs in plain text format. Please
-run the commands above again and replace the @file{.fits} suffix in the
-value to @option{--output} with @file{.txt}. The comments in MakeCatalog's
-output (in FITS headers or lines starting with @code{#} in plain text)
-contain some important information about the dataset that can be
-useful.
+run the commands above again and replace the @file{.fits} suffix, in the
+value to @option{--output}, with @file{.txt} (we will also be using the
+text file outputs later). The comments in MakeCatalog's output (in FITS
+headers or lines starting with @code{#} in plain text) contain some
+important information about the dataset that can be useful. So have a look
+at them:
 
 When a clumps image is also address@hidden will look at the
 @code{WCLUMPS} keyword in the objects image to see if it should also use a
-clumps image or not.}, two catalogs will be made. If you asked for a plain
-text file output, two files will be made with a @file{_c.txt} and
address@hidden suffixes. In a FITS output, the two catalos will be
-extensions of each output with the Fits program and then inspect the tables
-with Gnuastro's Table program (see @ref{Table}).
+clumps image or not.}, like this example, two catalogs will be made. If you
+asked for a plain text file output, two files will be made with a
address@hidden and @file{_o.txt} suffixes. If MakeCatalog's output is
+requested to be FITS, the two catalogs will be separate extensions of a
+single file. You can inspect the separate extensions with the Fits program
+like before (as shown below). You can inspect the table in each extension
+with Gnuastro's Table program (see @ref{Table}) as shown below.
 
 @example
-$ astfits catalog/xdf-f105w.fits
-$ asttable catalog/xdf-f105w.fits -h1 --info       # Objects catalog
-$ asttable catalog/xdf-f105w.fits -h2 --info       # Clumps catalog
+$ astfits catalog/xdf-f105w.fits             # Extension information
+$ asttable catalog/xdf-f105w.fits -h1 --info # Objects catalog info.
+$ asttable catalog/xdf-f105w.fits -h1        # Objects catalog columns.
+$ asttable catalog/xdf-f105w.fits -h2 -i     # Clumps catalog info.
+$ asttable catalog/xdf-f105w.fits -h2        # Clumps catalog columns.
 @end example
 
-To see the column contents of each table, you can just remove the
address@hidden option and if you want a special column, you can just
-specify its number (counting from @code{1}, same as output of the command
-above) or its column name (if it has one). For example, if you just want
-the magnitude and magnitude error, you can get it with this command:
+As you see above, to see the column contents of each table, you can just
+remove the @option{--info} (or @option{-i}) option. If you want to print
+the contents of special column(s), just specify the column number(s)
+(counting from @code{1}, same as output of the command above) or the column
+name(s) (if they have one). For example, if you just want the objects and
+clumps magnitude and magnitude error in the F160W filter. You can get it
+with the following commands. Note how we have used the column numbers in
+the F105W catalog and the column names in the F160W catalog. Both commands
+will print the same columns in their respective table.
 
 @example
-$ asttable catalog/xdf-f160w.fits -h1 -cMAGNITUDE -cMAGNITUDE_ERR   \
-           -cUPPERLIMIT_MAG
-$ asttable catalog/xdf-f160w.fits -h2 -cMAGNITUDE -cMAGNITUDE_ERR   \
-           -cUPPERLIMIT_MAG
+$ asttable catalog/xdf-f160w.fits -h1 -cMAGNITUDE -cMAGNITUDE_ERR
+$ asttable catalog/xdf-f160w.fits -h2 -cMAGNITUDE -cMAGNITUDE_ERR
 @end example
 
 Since the clumps catalog has two ID columns (one for the over-all clump ID
 and one for the ID of the clump in its host object), the magnitude column
 numbers differ between the object and clumps catalog. So if you want to
 specify the columns by number, you will need to change the numebers when
-viewing the clump and objects catalogs.
+viewing the clump and objects catalogs. This is a useful advantage of
+having/using column names.
 
 @example
-$ asttable catalog/xdf-f160w.fits -h1 -c4 -c5 -c6
-$ asttable catalog/xdf-f160w.fits -h2 -c5 -c6 -c7
+$ asttable catalog/xdf-f160w.fits -h1 -c4 -c5
+$ asttable catalog/xdf-f160w.fits -h2 -c5 -c6
 @end example
 
-Note how the upper-limit magnitudes differ between the FITS and plain text
-tables while other things are identical. The reason for this is that when
-not specified, the random number generator seed is different every time you
-run MakeCatalog. See @ref{Generating random numbers} for how to fix the
-seed so you get a reproducible result when necessary. You can use the
-upper-limit magnitude to see how reliable the measured magnitude is.
-
 Since we used the same labeled image on both filters, the number of rows in
 both filters are the same. So, let's measure the colors of the objects in
 this image. We'll merge the two catalogs together into one using the
 @code{paste} program on the command-line. The output file will have each
-line of both catalogs merged. AWK will then ignore lines starting with
address@hidden and print the ID, positional columns and the difference between
-the respective magnitude columns.
+line of both catalogs merged. Open @file{xdf-f160w-f105w_o_p.txt}
+after making it to see how @command{paste} has operated. We'll then use AWK
+and ask it to ignore lines starting with @code{#}. For the other lines, AWK
+will print the ID, positional columns and the difference between the
+respective magnitude columns.
 
 @example
 $ paste catalog/xdf-f160w_o.txt catalog/xdf-f105w_o.txt           \
-        > xdf-f160w-f105w_o.txt
-$ awk '!/^#/@{print $1, $2, $3, address@hidden' xdf-f160w-f105w_o.txt     \
-      > f160w-f105w_o_colors.txt
+        > xdf-f160w-f105w_o_p.txt
+$ awk '!/^#/@{print $1, $2, $3, address@hidden' xdf-f160w-f105w_o_p.txt    \
+      > xdf-f160w-f105w_o.txt
 @end example
 
 Gnuastro has a simple program for basic statistical measurements, you can
 see some basic information on the distribution of colors with this
 command. It will print some basic information about the distribution
 (minimum, maximum, median and etc), along with a small ASCII histogram to
-visually help you understand the distribution.
+visually help you understand the distribution on the command-line without
+the need for a graphic user interface. This can be useful when you just
+want some basic and general information and also works on a server (where
+you may not have graphic user interface).
 
 @example
-$ aststatistics f160w-f105w_o_colors.txt -c4
+$ aststatistics xdf-f160w-f105w_o.txt -c4
 @end example
 
-If you just want a specific measure, for example the mean and median and
-standard deviation, you can ask for them specifically:
+Once your processing is complete, you can use Gnuastro's Statistics program
+with @option{--histogram} to build a much more fine-grained histogram to
+feed into a plotting program for a much more accurate/appealing plot.  If
+you just want a specific measure, for example the mean, median and standard
+deviation, you can ask for them specifically:
 
 @example
-$ aststatistics f160w-f105w_o_colors.txt -c4 --mean --median --std
+$ aststatistics xdf-f160w-f105w_o.txt -c4 --mean --median --std
 @end example
 
 Finally, if any of the programs in Gnuastro have been useful for your