gnuastro-commits
[Top][All Lists]

## [gnuastro-commits] master 62ce84b 3/7: Book: extended introduction added

 From: Mohammad Akhlaghi Subject: [gnuastro-commits] master 62ce84b 3/7: Book: extended introduction added to bitwise operators Date: Mon, 4 Oct 2021 18:51:28 -0400 (EDT)

branch: master
commit 62ce84b54429bf793e41dd283b96e8c2254c2248

Book: extended introduction added to bitwise operators

Until now, there was no introductory explanation on the conditional,
bitwise, and building new dataset operators. While the first and third were
pretty easy to understand for a random astronomer, the second (bitwise
operators) could be a little hard to understand, so it was necessary to
give a good introduction in astronomical data analysis scenarios.

With this commit, an introduction has been added for all three sections and
the bitwise operators now have an extended discussion on why they are
useful for a modern astronomer.
---
doc/gnuastro.texi | 64 ++++++++++++++++++++++++++++++++++++++++++++++---------
1 file changed, 54 insertions(+), 10 deletions(-)

diff --git a/doc/gnuastro.texi b/doc/gnuastro.texi
index 8b51bc4..0686743 100644
--- a/doc/gnuastro.texi
+++ b/doc/gnuastro.texi
@@ -12744,11 +12744,12 @@ This is a very powerful notation and is used in
languages like Postscript @footn
@node Arithmetic operators, Invoking astarithmetic, Reverse polish notation,
Arithmetic
@subsection Arithmetic operators

-The recognized operators in Arithmetic are listed below.
-See @ref{Reverse polish notation} for more on how the operators and operands
should be ordered on the command-line.
-The operands to all operators can be a data array (for example a FITS image)
or a number, the output will be an array or number according to the inputs.
+In this section, list of recognized operators in Arithmetic (and the Table
program's @ref{Column arithmetic}) and discussed in detail with examples.
+As mentioned before, to be able to easily do complex operations on the
command-line, the Reverse Polish Notation is used (where you write
@mymath{4\quad5\quad+}' instead of @mymath{4 + 5}'), if you aren't already
familiar with it, before continuing, please see @ref{Reverse polish notation}.
+
+The operands to all operators can be a data array (for example a FITS image or
data cube) or a number, the output will be an array or number according to the
inputs.
For example a number multiplied by an array will produce an array.
-The conditional operators will return pixel, or numerical values of 0 (false)
or 1 (true) and stored in an @code{unsigned char} data type (see @ref{Numeric
data types}).
+The numerical data type of the output of each operator is described within it.

@cartouche
@noindent
@@ -13209,6 +13210,9 @@ The output will have the same type as the inputs.
This is natural for the @command{min} and @command{max} operators, but for
other similar operators (for example @command{sum}, or @command{average}) the
per-pixel operations will be done in double precision floating point and then
stored back in the input type.
Therefore, if the input was an integer, C's internal type conversion will be
used.

+@item
+The operation will be multi-threaded, grealy speeding up the process if you
have large and numerous data to stack.
@end itemize

@item max
@@ -13225,13 +13229,15 @@ $astarithmetic a.fits b.fits c.fits 3 min -omax.fits For each pixel count the number of non-blank pixels in all given datasets. The output will be an unsigned 32-bit integer datatype (see @ref{Numeric data types}). This operator is called similar to the @command{min} operator, please see there for more. -Note that some datasets may have blank values (which are also ignored in all similar operators like @command{min}, @command{sum}, @command{mean} or @command{median}). -Hence final pixel values of this operator will not, in general, be equal to the number of inputs. For example @example$ astarithmetic a.fits b.fits c.fits 3 number -onum.fits
@end example

+Some datasets may have blank values (which are also ignored in all similar
operators like @command{min}, @command{sum}, @command{mean} or
@command{median}).
+Hence, the final pixel values of this operator will not, in general, be equal
to the number of inputs.
+This operator is therefore mostly called in parallel with those operators to
know the weight'' of each pixel (in case you want to only keep pixels that
had the full exposure for example).
+
@item sum
For each pixel, calculate the sum in all given datasets.
The output will have the a single-precision (32-bit) floating point type.
@@ -13520,13 +13526,19 @@ $astarithmetic img1.fits img2.fits img3.fits 3 add-dimension @node Conditional operators, Mathematical morphology operators, Dimensionality changing operators, Arithmetic operators @subsubsection Conditional operators +Conditional operators take two inputs and return a binary output that can only have two values 0 (for pixels where the condition was false) or 1 (for the pixels where the condition was true). +Because of the binary (2-valued) nature of their outputs, the output is therefore stored in an @code{unsigned char} data type (see @ref{Numeric data types}) to speed up process and take less space in your storage. +There are two exceptions to the general features above: @code{isblank} only takes one input, and @code{where} takes three, while not returning a binary output, see their description for more. + @table @command @item lt Less than: creates a binary output (values either 0 or 1) where each pixel will be 1 if the second popped operand is smaller than the first popped operand and 0 otherwise. If both operands are images, then all the pixels will be compared with their counterparts in the other image. -For example: + +For example, the pixels in the output of the command below will have a value of 1 (true) if their value in @file{image1.fits} is less than their value in @file{image2.fits}. +Otherwise, their value will be 0 (false). @example -$ astarithmetic image1.fits image2.fits -g1 lt
+$astarithmetic image1.fits image2.fits lt @end example If only one operand is an image, then all the pixels will be compared with the single value (number) of the other operand. For example: @@ -13756,6 +13768,38 @@$ astarithmetic image.fits invert
@node Bitwise operators, Numerical type conversion operators, Mathematical
morphology operators, Arithmetic operators
@subsubsection Bitwise operators

+@cindex Bitwise operators
+Astronomical images are usually stored as an array multi-byte pixels with
different sizes for different precision levels (see @ref{Numeric data types}).
+For example images from CCDs are usually in the unsigned 16-bit integer type
(each pixel takes 16 bits, or 2 bytes, of memory) and fully reduced deep images
have a 32-bit floating point type (each pixel takes 32 bits or 4 bytes).
+
+On the other hand, during the data reduction, we need to preserve a lot of
+For example, if a cosmic ray had hit the pixel during the exposure, or if the
pixel was saturated, or is known to have a problem, or if the optical
vignetting is too strong on it, and etc.
+A crude solution is to make a new image when checking for each one of these
things and make a binary image where we flag (set to 1) pixels that satisfy any
of these conditions above, and set the rest to zero.
+However, processing pipelines sometimes need more than 20 flags to store
important per-pixel meta-data, and recall that the smallest numeric data type
is one byte (or 8 bits, that can store up to 256 different values), while we
only need two values for each flag!
+This is a major waste of storage space!
+
+A much more optimal solution is to use the bits within each pixel to store
different flags!
+In other words, if you have an 8-bit pixel, use each bit as as a flag to mark
if a certain condition has happened on a certain pixel or not.
+For example, let's set the following standard based on the four cases
mentioned above: the first bit will show that a cosmic ray has hit that pixel.
+So if a pixel is only affected by cosmic rays, it will have this sequence of
bits (note that the bit-counting starts from the right): @code{00000001}.
+The second bit shows that the pixel was saturated (@code{00000010}), the third
bit shows that it has known problems (@code{00000100}) and the fourth bit shows
that it was affected by vignetting (@code{00001000}).
+
+Since each bit is independent, we can thus mark multiple metadata about that
pixel in the actual image, within a single flag'' or mask'' pixel of a flag
or mask image that has the same number of pixels.
+For example a flag-pixel with the following bits @code{00001001} shows that it
has been affected by cosmic rays @emph{and} it has been affected by vignetting
at the same time.
+The common data type to store these flagging pixels are unsigned integer types
(see @ref{Numeric data types}).
+Therefore when you open an unsigned 8-bit flag image in a viewer like DS9, you
will see a single integer in each pixel that actually has 8 layers of metadata
in it!
+For example the integer you will see for the bit sequences given above will
respectively be: @mymath{2^0=1} (for a pixel that only has cosmic ray),
@mymath{2^1=2} (for a pixel that was only saturated), @mymath{2^2=4} (for a
pixel that only has known problems), @mymath{2^3=8} (for a pixel that is only
affected by vignetting) and @mymath{2^0 + 2^3 = 9} (for a pixel that has a
cosmic ray @emph{and} was affected by vignetting).
+
+You can later use this bit information to mark objects in your final analysis
+For example you may want to set all pixels affected by vignetting to NaN, but
can interpolate over cosmic rays.
+You therefore need ways to separate the pixels with a desired flag(s) from the
rest.
+It is possible to treat a flag pixel as a single integer (and try to define
certain ranges in value to select certain flags).
+But a much more easier and robust way is to actually look at each pixel as a
sequence of bits (not as a single integer!) and use the bitwise operators below
for this job.
+For more on the theory behind bitwise operators, see
@url{https://en.wikipedia.org/wiki/Bitwise_operation, Wikipedia}.
+
+
@table @command
@item bitand
Bitwise AND operator: only bits with values of 1 in both popped operands will
get the value of 1, the rest will be set to 0.
@@ -13976,7 +14020,7 @@ $asttable catalog.fits \ @node Building new dataset, Operand storage in memory or a file, Elliptical shape operators, Arithmetic operators @subsubsection Building new dataset - +With the operator here, you can create a new dataset from scratch to start certain operations without any input data. @table @command @item makenew @@ -14000,7 +14044,7 @@$ astarithmetic 100 200 2 makenew 5 mknoise-sigma
@node Operand storage in memory or a file,  , Building new dataset, Arithmetic
operators
@subsubsection Operand storage in memory or a file

-In your early days of using Gnuastro, to do multiple operations, it is likely
that you will simply call Arithmetic (or Table, with column crithmetic)
multiple times.
+In your early days of using Gnuastro, to do multiple operations, it is likely
that you will simply call Arithmetic (or Table, with column arithmetic)
multiple times: feed the output file of the first call to the second call.
But as you get more proficient in the reverse polish notation, you will find
yourself combining many operations into one call.
This greatly speeds up your operation, because instead of writing the dataset
to a file in one command, and reading it in the next command, it will just keep
the intermediate dataset in memory!