Hi Ben,
here are some thoughts about OpenEXR as a capture format. I must admit
up front that I don't know much about the internal processing capabilities
of digital cameras, so the following may or may not be particularly
relevant. Of course, if any camera manufacturers are interested, I'd
be more than happy to discuss what it would take to make a camera output
OpenEXR directly.
Is it useful?
The way I see it, OpenEXR's main advantage would be the format's wide
exposure latitude and fine color resolution. A camera's capabilities
are not likely to exceed what the file format can store. This would
allow the camera to save scene referred (or focal-plane referred) images,
where the values stored in the pixels are proportional to the amount of
light emitted by the objects in the original scene (as opposed to storing
the color you want to see when the image is displayed). Color rendering
decisions can be postponed until later. If the camera has knobs to tweak
the look of the images, the settings of those knobs could be saved as
metadata along with the unmodified scene referred images.
This capability to store scene referred images with a wide dynamic range
and to postpone color rendering decisions is similar to the various camera
raw formats that already exist. One advantage of OpenEXR is that the image
representation would not be camera-specific, and that the first stages of
processing the raw sensor data have already been done. This includes Bayer
pattern demosaicing and estimating actual scene colors from the raw sensor
values, operations that are best left to the camera manyfacturer. Unlike
raw files, OpenEXR files are ready to use in standard image processing and
viewing applications.
OpenEXR's multiresolution and tiling capabilities may come in handy for
high-resolution images; I guess that those capabilities are more important
for stills than for moving images.
Is it practical?
An important question for a camera manufacturer would be how difficult it
is to write and read OpenEXR files. Movie cameras must write the files in
real time.
In OpenEXR up to version 1.4 only uncompressed files could possibly be
written in real time at the resolutions required for digital cinema.
The bandwidth required to record high-resolution images with 48 bits or
24 bits per pixel (depending on whether RGB or luminance/chroma encoding
is used) is substantial. OpenEXR's lossless compresssion schemes, PIZ,
ZIP, etc. don't help here. The encoders are too slow to run in real time
on affordable hardware. In addition, since the compression is lossless,
the required recording bandwidth changes based on image content. In the
worst case, compression does not shrink the image data at all, and the
required bandwidth is the same as for uncompressed images. If a recording
system can already handle this worst case, it might as well skip the
compression step alltogether.
In OpenEXR 1.5 we added B44 compression specifically to adress real-time
playback of moving image sequences. B44 achieves a fixed compression rate
of 2.28:1 or 4.57:1, depending on whether the images are stored in RGB or
luminance/chroma format. Compared to uncompressed images, this reliably
reduces the required bandwidth by a factor of 2.28 or 4.57, independent of
image content. B44 compression is simple enough to allow decoding in real
time. Using the open-source playexr program on an HP wx9400 workstation,
we are able to play a 2k version of the STEM movie at 35 frames per second,
directly from disk.
The B44 encoder is more complex than the decoder, but not much. Building
a system that records B44-compressed OpenEXR files in real time should be
possible.
From a camera manufacturer's point of view, the OpenEXR libraries may be
dauntingly complex. However, a camera would most likely output images
only in one fixed format, for example, B44-compressed luminance and chroma.
All the code that deals with arbitrary sets of channels and arbitrary
compression schemes could be thrown out. In addition, knowing the camera's
internal workings, most of the frame buffer code that supports a multitude
of in-memory image layouts could be discarded. Specializing the OpenEXR
libraries for a specific camera and output format would allow very
significant optimizations, affecting both code size and speed. Some of
the code could probably be moved into specialized hardware, for example,
an FPGA.
The on-disk OpenEXR file format may be the source of another objection:
In order to support random access to scan lines and tiles, a scan line or
tile position table is inserted near the beginning of the file. With
lossless compression the position of the scan lines or tiles within the
file depends on image content, and the table entries are not known until
the corresponding scan lines or tiles have been stored in the file. Once
all the pixels have been saved in the file, the OpenEXR library must perform
a seek operation back to the beginning of the table and write the table
entries. If the files are recorded on a streaming medium, for example, a
tape, a backwards seek operation is incompatible with real-time performance.
This means that the entire file would have to be constructed in an in-memory
buffer before it is appended to the output stream. However, for B44-
compressed and uncompressed files, the scan line or tile positions are
known in advance because they do not depend on image content. Seek
operations are not necessary, an the file could be streamed out to the
recording medium while it is being constructed.
Overall I think it should be possible to use OpenEXR as a capture format,
but as I mentioned above, there may be problems that I don't know about.
Florian