[task #16447] Median Absolute Deviation based clipping

[Mohammad Akhlaghi]

URL:
  <https://savannah.gnu.org/task/?16447>

                 Summary: Median Absolute Deviation based clipping
                   Group: GNU Astronomy Utilities
               Submitter: makhlaghi
               Submitted: Wed 04 Oct 2023 02:54:20 AM CEST
         Should Start On: Wed 04 Oct 2023 12:00:00 AM CEST
   Should be Finished on: Wed 04 Oct 2023 12:00:00 AM CEST
                Category: Libraries
                Priority: 5 - Normal
              Item Group: Enhancement
                  Status: None
                 Privacy: Public
        Percent Complete: 0%
             Assigned to: makhlaghi
             Open/Closed: Open
         Discussion Lock: Any
                  Effort: 0.00


    _______________________________________________________

Follow-up Comments:


-------------------------------------------------------
Date: Wed 04 Oct 2023 02:54:20 AM CEST By: Mohammad Akhlaghi <makhlaghi>
Currently, the most commonly used way to reject outliers is through
[https://www.gnu.org/software/gnuastro/manual/html_node/Sigma-clipping.html
sigma-clipping]. However, sigma-clipping is itself based on the standard
deviation (which is strongly affected by outliers!). 

To demonstrate the problem, in the P.S. I am attaching a small script to
generate 9 noisy images, where one of them contains a bright circle in the
middle. Afterwards the script stacks the images by sigma-clipping and opens
DS9 to show the sigma-clipped result. The footprint of the circle is present
in the sigma-clipped mean (very strong) and sigma-clipped median (weaker, but
still present). 

As mentioned above, the cause of this is that the standard deviation is very
easily/strongly affected by outliers. You can see this through the few numbers
of this command (which contain the strong outlier of 15). The third command
doesn't contain the '15':


$ printf "2\n1\n4\n2\n3\n5\n15\n"
2
1
4
2
3
5
15

$ printf "2\n1\n4\n2\n3\n5\n15\n" \
         | aststatistics --sigclip-mean --sigclip-std
4.571429e+00 4.435479e+00

$ printf "2\n1\n4\n2\n3\n5\n" \
         | aststatistics --sigclip-mean --sigclip-std
2.833333e+00 1.343710e+00


As you see here, sigma-clipping failed to clip/exclude the '15' because the
standard deviation itself is so strongly shifted due to it!

In this task, I have started work on generalizing the clipping algorithm to
also do the clipping based on
[https://en.wikipedia.org/wiki/Median_absolute_deviation Median Absolute
Deviation] (MAD). 

P.S. 


# Constants
profn=4
sigma=10
width=201
number=9
radius=30
profsum=1e5

# Initialize internal parameters.
imglist=""

# For the final profile, put a flat circle in the center of the image.
nn=$number-no-noise.fits
center=$(echo $width | awk '{print int($1/2)+1}')
echo "1 $center $center 5 $radius $profn 0 1 $profsum 1" \
    | astmkprof --mode=img --mergedsize=$width,$width --oversample=1 \
                --output=$nn --mcolissum
astarithmetic $nn $sigma mknoise-sigma -o$number.fits
imglist="$imglist $number.fits"

# Build pure noise.
numnoise=$(echo $number | awk '{print $1-1}')
for i in $(seq 1 $numnoise); do
    imglist="$imglist $i.fits"
    astarithmetic $width $width 2 makenew $sigma mknoise-sigma -o$i.fits
done

# Stack the images.
astarithmetic $imglist $number 3 0.2 sigclip-std    -g1 -o stack-std.fits
astarithmetic $imglist $number 3 0.2 sigclip-number -g1 -o stack-num.fits
astarithmetic $imglist $number 3 0.2 sigclip-mean   -g1 -o stack-mean.fits
astarithmetic $imglist $number 3 0.2 sigclip-median -g1 -o stack-median.fits

# Show the images and the stack.
#$imglist
#astscript-fits-view $imglist
#astscript-fits-view $number.fits stack-std.fits stack-median.fits
astscript-fits-view stack-num.fits stack-mean.fits stack-std.fits
stack-median.fits








    _______________________________________________________
File Attachments:


-------------------------------------------------------
Date: Wed 04 Oct 2023 02:54:20 AM CEST  Name: inputs.png  Size: 485KiB   By:
makhlaghi

<http://savannah.gnu.org/task/download.php?file_id=55196>
-------------------------------------------------------
Date: Wed 04 Oct 2023 02:54:20 AM CEST  Name: sigma-clipping.png  Size: 345KiB
  By: makhlaghi

<http://savannah.gnu.org/task/download.php?file_id=55197>

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/task/?16447>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/