[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[gnuastrocommits] master 62ce84b 3/7: Book: extended introduction added
From: 
Mohammad Akhlaghi 
Subject: 
[gnuastrocommits] master 62ce84b 3/7: Book: extended introduction added to bitwise operators 
Date: 
Mon, 4 Oct 2021 18:51:28 0400 (EDT) 
branch: master
commit 62ce84b54429bf793e41dd283b96e8c2254c2248
Author: Mohammad Akhlaghi <mohammad@akhlaghi.org>
Commit: Mohammad Akhlaghi <mohammad@akhlaghi.org>
Book: extended introduction added to bitwise operators
Until now, there was no introductory explanation on the conditional,
bitwise, and building new dataset operators. While the first and third were
pretty easy to understand for a random astronomer, the second (bitwise
operators) could be a little hard to understand, so it was necessary to
give a good introduction in astronomical data analysis scenarios.
With this commit, an introduction has been added for all three sections and
the bitwise operators now have an extended discussion on why they are
useful for a modern astronomer.

doc/gnuastro.texi  64 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 54 insertions(+), 10 deletions()
diff git a/doc/gnuastro.texi b/doc/gnuastro.texi
index 8b51bc4..0686743 100644
 a/doc/gnuastro.texi
+++ b/doc/gnuastro.texi
@@ 12744,11 +12744,12 @@ This is a very powerful notation and is used in
languages like Postscript @footn
@node Arithmetic operators, Invoking astarithmetic, Reverse polish notation,
Arithmetic
@subsection Arithmetic operators
The recognized operators in Arithmetic are listed below.
See @ref{Reverse polish notation} for more on how the operators and operands
should be ordered on the commandline.
The operands to all operators can be a data array (for example a FITS image)
or a number, the output will be an array or number according to the inputs.
+In this section, list of recognized operators in Arithmetic (and the Table
program's @ref{Column arithmetic}) and discussed in detail with examples.
+As mentioned before, to be able to easily do complex operations on the
commandline, the Reverse Polish Notation is used (where you write
`@mymath{4\quad5\quad+}' instead of `@mymath{4 + 5}'), if you aren't already
familiar with it, before continuing, please see @ref{Reverse polish notation}.
+
+The operands to all operators can be a data array (for example a FITS image or
data cube) or a number, the output will be an array or number according to the
inputs.
For example a number multiplied by an array will produce an array.
The conditional operators will return pixel, or numerical values of 0 (false)
or 1 (true) and stored in an @code{unsigned char} data type (see @ref{Numeric
data types}).
+The numerical data type of the output of each operator is described within it.
@cartouche
@noindent
@@ 13209,6 +13210,9 @@ The output will have the same type as the inputs.
This is natural for the @command{min} and @command{max} operators, but for
other similar operators (for example @command{sum}, or @command{average}) the
perpixel operations will be done in double precision floating point and then
stored back in the input type.
Therefore, if the input was an integer, C's internal type conversion will be
used.
+@item
+The operation will be multithreaded, grealy speeding up the process if you
have large and numerous data to stack.
+You can disable multithreaded operations with the @option{numthreads=1}
option (see @ref{Multithreaded operations}).
@end itemize
@item max
@@ 13225,13 +13229,15 @@ $ astarithmetic a.fits b.fits c.fits 3 min omax.fits
For each pixel count the number of nonblank pixels in all given datasets.
The output will be an unsigned 32bit integer datatype (see @ref{Numeric data
types}).
This operator is called similar to the @command{min} operator, please see
there for more.
Note that some datasets may have blank values (which are also ignored in all
similar operators like @command{min}, @command{sum}, @command{mean} or
@command{median}).
Hence final pixel values of this operator will not, in general, be equal to
the number of inputs.
For example
@example
$ astarithmetic a.fits b.fits c.fits 3 number onum.fits
@end example
+Some datasets may have blank values (which are also ignored in all similar
operators like @command{min}, @command{sum}, @command{mean} or
@command{median}).
+Hence, the final pixel values of this operator will not, in general, be equal
to the number of inputs.
+This operator is therefore mostly called in parallel with those operators to
know the ``weight'' of each pixel (in case you want to only keep pixels that
had the full exposure for example).
+
@item sum
For each pixel, calculate the sum in all given datasets.
The output will have the a singleprecision (32bit) floating point type.
@@ 13520,13 +13526,19 @@ $ astarithmetic img1.fits img2.fits img3.fits 3
adddimension
@node Conditional operators, Mathematical morphology operators, Dimensionality
changing operators, Arithmetic operators
@subsubsection Conditional operators
+Conditional operators take two inputs and return a binary output that can only
have two values 0 (for pixels where the condition was false) or 1 (for the
pixels where the condition was true).
+Because of the binary (2valued) nature of their outputs, the output is
therefore stored in an @code{unsigned char} data type (see @ref{Numeric data
types}) to speed up process and take less space in your storage.
+There are two exceptions to the general features above: @code{isblank} only
takes one input, and @code{where} takes three, while not returning a binary
output, see their description for more.
+
@table @command
@item lt
Less than: creates a binary output (values either 0 or 1) where each pixel
will be 1 if the second popped operand is smaller than the first popped operand
and 0 otherwise.
If both operands are images, then all the pixels will be compared with their
counterparts in the other image.
For example:
+
+For example, the pixels in the output of the command below will have a value
of 1 (true) if their value in @file{image1.fits} is less than their value in
@file{image2.fits}.
+Otherwise, their value will be 0 (false).
@example
$ astarithmetic image1.fits image2.fits g1 lt
+$ astarithmetic image1.fits image2.fits lt
@end example
If only one operand is an image, then all the pixels will be compared with the
single value (number) of the other operand.
For example:
@@ 13756,6 +13768,38 @@ $ astarithmetic image.fits invert
@node Bitwise operators, Numerical type conversion operators, Mathematical
morphology operators, Arithmetic operators
@subsubsection Bitwise operators
+@cindex Bitwise operators
+Astronomical images are usually stored as an array multibyte pixels with
different sizes for different precision levels (see @ref{Numeric data types}).
+For example images from CCDs are usually in the unsigned 16bit integer type
(each pixel takes 16 bits, or 2 bytes, of memory) and fully reduced deep images
have a 32bit floating point type (each pixel takes 32 bits or 4 bytes).
+
+On the other hand, during the data reduction, we need to preserve a lot of
metadata about some pixels.
+For example, if a cosmic ray had hit the pixel during the exposure, or if the
pixel was saturated, or is known to have a problem, or if the optical
vignetting is too strong on it, and etc.
+A crude solution is to make a new image when checking for each one of these
things and make a binary image where we flag (set to 1) pixels that satisfy any
of these conditions above, and set the rest to zero.
+However, processing pipelines sometimes need more than 20 flags to store
important perpixel metadata, and recall that the smallest numeric data type
is one byte (or 8 bits, that can store up to 256 different values), while we
only need two values for each flag!
+This is a major waste of storage space!
+
+@cindex Flag (mask) images
+@cindex Mask (flag) images
+A much more optimal solution is to use the bits within each pixel to store
different flags!
+In other words, if you have an 8bit pixel, use each bit as as a flag to mark
if a certain condition has happened on a certain pixel or not.
+For example, let's set the following standard based on the four cases
mentioned above: the first bit will show that a cosmic ray has hit that pixel.
+So if a pixel is only affected by cosmic rays, it will have this sequence of
bits (note that the bitcounting starts from the right): @code{00000001}.
+The second bit shows that the pixel was saturated (@code{00000010}), the third
bit shows that it has known problems (@code{00000100}) and the fourth bit shows
that it was affected by vignetting (@code{00001000}).
+
+Since each bit is independent, we can thus mark multiple metadata about that
pixel in the actual image, within a single ``flag'' or ``mask'' pixel of a flag
or mask image that has the same number of pixels.
+For example a flagpixel with the following bits @code{00001001} shows that it
has been affected by cosmic rays @emph{and} it has been affected by vignetting
at the same time.
+The common data type to store these flagging pixels are unsigned integer types
(see @ref{Numeric data types}).
+Therefore when you open an unsigned 8bit flag image in a viewer like DS9, you
will see a single integer in each pixel that actually has 8 layers of metadata
in it!
+For example the integer you will see for the bit sequences given above will
respectively be: @mymath{2^0=1} (for a pixel that only has cosmic ray),
@mymath{2^1=2} (for a pixel that was only saturated), @mymath{2^2=4} (for a
pixel that only has known problems), @mymath{2^3=8} (for a pixel that is only
affected by vignetting) and @mymath{2^0 + 2^3 = 9} (for a pixel that has a
cosmic ray @emph{and} was affected by vignetting).
+
+You can later use this bit information to mark objects in your final analysis
or to mask certain pixels.
+For example you may want to set all pixels affected by vignetting to NaN, but
can interpolate over cosmic rays.
+You therefore need ways to separate the pixels with a desired flag(s) from the
rest.
+It is possible to treat a flag pixel as a single integer (and try to define
certain ranges in value to select certain flags).
+But a much more easier and robust way is to actually look at each pixel as a
sequence of bits (not as a single integer!) and use the bitwise operators below
for this job.
+For more on the theory behind bitwise operators, see
@url{https://en.wikipedia.org/wiki/Bitwise_operation, Wikipedia}.
+
+
@table @command
@item bitand
Bitwise AND operator: only bits with values of 1 in both popped operands will
get the value of 1, the rest will be set to 0.
@@ 13976,7 +14020,7 @@ $ asttable catalog.fits \
@node Building new dataset, Operand storage in memory or a file, Elliptical
shape operators, Arithmetic operators
@subsubsection Building new dataset

+With the operator here, you can create a new dataset from scratch to start
certain operations without any input data.
@table @command
@item makenew
@@ 14000,7 +14044,7 @@ $ astarithmetic 100 200 2 makenew 5 mknoisesigma
@node Operand storage in memory or a file, , Building new dataset, Arithmetic
operators
@subsubsection Operand storage in memory or a file
In your early days of using Gnuastro, to do multiple operations, it is likely
that you will simply call Arithmetic (or Table, with column crithmetic)
multiple times.
+In your early days of using Gnuastro, to do multiple operations, it is likely
that you will simply call Arithmetic (or Table, with column arithmetic)
multiple times: feed the output file of the first call to the second call.
But as you get more proficient in the reverse polish notation, you will find
yourself combining many operations into one call.
This greatly speeds up your operation, because instead of writing the dataset
to a file in one command, and reading it in the next command, it will just keep
the intermediate dataset in memory!
 [gnuastrocommits] master updated (3d810ae > aac9048), Mohammad Akhlaghi, 2021/10/04
 [gnuastrocommits] master 7743114 2/7: Book: Arithmetic operators now divided into subsections, Mohammad Akhlaghi, 2021/10/04
 [gnuastrocommits] master f8ceb0a 5/7: Library (wcs): free 'imgcrd' memory, Mohammad Akhlaghi, 2021/10/04
 [gnuastrocommits] master 62ce84b 3/7: Book: extended introduction added to bitwise operators,
Mohammad Akhlaghi <=
 [gnuastrocommits] master 327c7d3 6/7: Library (wcs.c): rename array to list converter, Mohammad Akhlaghi, 2021/10/04
 [gnuastrocommits] master bdaa44b 1/7: Library (arithmetic.h): sum operator uses pixels in data with blank, Mohammad Akhlaghi, 2021/10/04
 [gnuastrocommits] master aac9048 7/7: Library (fits.h): better warning message in gal_fits_key_write_filename, Mohammad Akhlaghi, 2021/10/04
 [gnuastrocommits] master 02375f8 4/7: Development: built shell completion files added to .gitignore, Mohammad Akhlaghi, 2021/10/04