[task #15181] Libcurl for optimal download of only the necessary bytes

[Mohammad Akhlaghi]

Update of task #15181 (project gnuastro):

                 Summary: Fits program using network libraries for optimal
download => Libcurl for optimal download of only the necessary bytes

    _______________________________________________________

Follow-up Comment #2:

For a command-line alternative, Ángel López Sainz suggested the following
command that will extract only the first 20 header keywords of the FITS file
in the given URL (note that each keyword is 80 characters/bytes long):


curl -kRL -r "$((0*80))-$((20*80-1))" --output header.txt
$xdfurl/hlsp_xdf_hst_wfc3ir-60mas_hudf_f105w_v1_sci.fits


Besides the 'fopen.c' solution in the commets below, we can try to investigate
we can try to investigate how these command-line options can be called within
the 'curl' library.

The basic idea for FITS files is that the header keywords must be in chunks of
2880 bytes (or 36 keywords); the last chunk is the one that contains the 'END'
characters at the start of the 80-character line. 

So we read chunks of 2880 bytes, and search for the 'END' character at the
start of a line. The moment it is found, we stop the download and have the
full header keyword structure to feed into CFITSIO (or WCSLIB). Once the
necessary pixels to use are found, we can then only read those pixels from the
data component of the HDU (which is just after the chunk containing 'END'). 

Of course, if there are multiple HDUs, from the first HDU we know the byte
number of the next HDU and can go there ;-).


    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/task/?15181>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/