Re: imread on large tiff

[Carnë Draug]

On 15 Jan 2014 20:58:19 +0100,  John Hayes <address@hidden> wrote:
> Hi all,
>
> I have a problem where if I try either imfinfo or imread on a 640x540x1800 
> TIFF file that is ~1.2 GB, Octave hangs seemingly indefinitely. This is 
> occurring with both GraphicsMagick 1.3.18 and 1.3.19. I?m on OS X 10.6.8 with 
> gcc 4.8.2 (I built) and octave 3.8.0 downloaded on Dec. 27, 2013. I?m using 
> the filename as argument to imfinfo and for imread, I use this:
>> video=zeros(numFrames, numRows, numCols, 'single');
>> disp('Reading movie file...')
>> for i=1:numFrames
>>     disp(i);
>>     video(i,:,:)=imread(fileName,i);
>> end
>
> At least for imread, it seems to be hanging somewhere in the __magick_read__ 
> function, but I haven?t dug into it too deeply yet. I?ve extracted the first 
> 10 frames of the TIFF using ImageJ, moved it into the location of the big 
> file, and reran the script on this file. In this case, they seem to work. 
> This suggests the problem may not be that it?s hanging but that it?s just 
> going really slow. But it?s not immediately obvious to me why imfinfo would 
> be so slow (or imread for that matter).
>
> Btw, GraphicsMagick was configured with the following command:
>> ./configure --with-quantum-depth=32 --enable-shared --disable-static 
>> --with-magick-plus-plus=yes
>
> As a final note, when I Ctrl-C to stop Octave, it takes a long time to 
> actually quit. If I hit Ctrl-C multiple times, Octave reports the following:
>> ^C^C^Cpanic: Interrupt -- stopping myself...
>> attempting to save variables to 'octave-workspace'...
>> ^Cpanic: attempted clean up apparently failed -- aborting...
> and dumps a HUGE core file to /cores/ (>6.5 GB!!). In the process of writing 
> the core file, it clearly affects the general I/O on the computer (to the 
> extent I often have to reboot as sudo kill -9 won?t stop it), so there seems 
> to be ?something' gobbling up a lot of memory. :)
>
> Thanks for any insight you can provide on how to solve the main problem 
> (imfinfo/imread usage with a 1.2 GB TIFF). Any clues as I dig deeper would be 
> most helpful...
>
> Best regards,
>
> John

Hi John

I would not recommend using that syntax to read such a large file. Use

video = imread (filename, "Index", "all")

or

video = imread (filename, "Index", 1:numFrames);

The reason it's so slow is on the nature of multipage tiff. If you
want to find where the image #100, you need to go through the previous
99 images. You have 1800 pages so... you get what I mean. By using
this syntax, Octave will read everything in one go.

You mention using imfinfo. I'm assuming you're doing this to get the
number of rows, columns and frames in your image. imfinfo will return
a struct array with a lot of fields for each frame in your file. Note
that it is possible for each page on your TIFF to have different info,
even different size, we can't just deduce all that from the first
page. And that is slow.

Also, because the way GraphicsMagick works, if you built it with
quantum-depth 32, images are read as uint32. Octave then resscales the
values and casts the image to the correct data type. This means that
if your image was uint8, it may temporarily take up 4 times its
original size. And that's considering that the image in your tiff file
is not compressed. So a careful choice of the options used to build
GraphicsMagick should be made if you're dealing with such unusual
cases as 1.2GB images. Maybe a quantum-depth of 16 will be enough for
you.

Carnë