Re: [Openexr-devel] EXR texture memory overhead

[Larry Gritz]

See how there were 1957 unique textures, but you created 144168 ImageInputs? 
That means your file (handle) cache was too small, you were accessing the files 
in an incoherent order and the "working set" was much larger than the file 
cache size (100), so you ended up closing and reopening files a massive number 
of times (73 times each, on average). 

The solution to this is to raise the TextureSystem's (that is, the underlying 
ImageCache's) "max_open_files" significantly. The default of 100 is extremely 
conservative -- trying to keep you out of trouble on operating systems with a 
low handle count per process, or with ImageCache's file operations being 
potentially only a small portion of your process's file needs.

On a modern Linux system, you should be able to have several thousand files 
open simultaneously. I would do that and see what it does to all your figures. 
At the very least, it should substantially lower the "file open time" and maybe 
the file I/O time overall.

But, it may make the memory issues worse -- if libIlmImf is internally holding 
more memory per open file than libtiff does (my tests indicate this as well), 
then having more files open at a time may exacerbate the problem. I haven't 
tested this for a while, though, but I think that Karl is correct that part of 
this is that each of the threads in OpenEXR's pool is holding a bit of working 
memory, and that adds up, so the higher your thread count, the more memory use 
overhead there will be for libIlmImf, and it's very hard for the app to account 
for that or control it.

It is simply a fact that on a per-thread, per-file basis, libIlmImf uses more 
overhead memory per open file than libtiff does. I have not looked into it 
deeply enough to know if that's a good design tradeoff or not (for example, 
maybe that memory is put to good use, and helps speed up some operations).

This is the kind of thing you would never notice reading OpenEXR files 
sequentially, but in something like a texture system where you may have 
thousands of files open simultaneously, an extra 256KB of overhead per file 
adds up fast.


> On Sep 16, 2016, at 3:34 AM, Søren Ragsdale <address@hidden> wrote:
> 
> Hello, OpenEXR devs. I've been doing some comparative rendering tests I've 
> found something a bit surprising. 
> 
> TIFF and EXR texture access *times* seems more or less the same, which is 
> fine because the underlying data is equivalent. (Same data type, compression, 
> tile size, etc.) But the RAM overhead seems much higher for EXRs. We've got a 
> 9GB render using TIFFs and a 13GB render using EXRs.
> 
> Does anyone have some theories why EXR texture access is requiring 4GB more 
> memory?
> 
> 
> Prman-20.11, OSL shaders, OIIO/TIFF textures:
> real 00:21:46
> VmRSS 9,063.45 MB
> OpenImageIO ImageCache statistics (shared) ver 1.7.3dev
>   Options:  max_memory_MB=4000.0 max_open_files=100 autotile=64
>             autoscanline=0 automip=1 forcefloat=0 accept_untiled=1
>             accept_unmipped=1 read_before_insert=0 deduplicate=1
>             unassociatedalpha=0 failure_retries=0 
>   Images : 1957 unique
>     ImageInputs : 136432 created, 100 current, 796 peak
>     Total size of all images referenced : 166.0 GB
>     Read from disk : 55.5 GB
>     File I/O time : 7h 2m 33.9s (16m 54.2s average per thread)
>     File open time only : 27m 44.0s
> 
> 
> Prman-20.11, OSL shaders, OIIO/EXR textures:
> real 00:21:14
> VmRSS 12,938.83 MB
> OpenImageIO ImageCache statistics (shared) ver 1.7.3dev
>   Options:  max_memory_MB=4000.0 max_open_files=100 autotile=64
>             autoscanline=0 automip=1 forcefloat=0 accept_untiled=1
>             accept_unmipped=1 read_before_insert=0 deduplicate=1
>             unassociatedalpha=0 failure_retries=0 
>   Images : 1957 unique
>     ImageInputs : 133168 created, 100 current, 771 peak
>     Total size of all images referenced : 166.0 GB
>     Read from disk : 55.5 GB
>     File I/O time : 6h 15m 42.1s (15m 1.7s average per thread)
>     File open time only : 1m 22.5s
> 
> _______________________________________________
> Openexr-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/openexr-devel

--
Larry Gritz
address@hidden