openexr-devel
[Top][All Lists]
Advanced
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Openexr-devel] OpenEXR 1.7 not backward compatible?

From: Benoit Leveau
Subject: Re: [Openexr-devel] OpenEXR 1.7 not backward compatible?
Date: Wed, 25 Aug 2010 16:30:50 +0100
User-agent: Thunderbird 2.0.0.12 (X11/20080213) 
I said that my patch would be:
- in ImfName.h, reverting the SIZE to 32 (instead of 256)
- making sure the usesLongName() function in ImfHeader.cpp returns false so the 
flag is never set

Changing the SIZE is actually a bad idea as it has an impact on the size of the 
Imf::Name class because of the member variable '_text'.
It means that you can't select at runtime the patched or non-patched version of 
the library as anything that uses the library had to use the corresponding 
header.
Changing instead the MAX_LENGTH to 31 and leaving the SIZE to 256 doesn't exhibit this problem: you can use the patched library with projects that were compiled against the non-patched library, there is no API change. 
Benoit Leveau wrote:

Is it not possible to avoid using long channel or attribute
names in the first place when backward compatibility is an issue?
Somewhere in your code you are assigning a name longer than 31
characters to an attribute or a channel.  With OpenEXR 1.6 and
earlier that name gets truncated to 31 characters plus a trailing
null.  If you can shorten that name so that it doesn't get
truncated, then the on-disk files will be backward-compatible
without any library changes.
I'm aware that some of our code is using names longer than 31 characters, but this code is possibly spread on a lot of projects/tools, and because this has never been an issue until now we've never tracked all these places. If we were to manually search for these usage, there is no guarantee we'll get all of them and if only one channel has a long name, the exr file won't be readable by any application, breaking the pipeline. It is safer for us to patch the library to make sure this 
never happens.
This is also stated in the ChangeLog, which I would hope is required reading for any developer switching to 1.7.
I read the changelog while doing the upgrade, but dealing with a lot of legacy code for different developers I wasn't aware we were using such long names in a few places. And honestly, it didn't occur to me that the library was behaving differently until I really dig into the issue. Finally, OpenEXR was just one among dozen of third-party libraries we upgraded. Tracking each individual change wasn't an option, especially since we skipped a few versions for each library.
Adding a backwards compatibility mode goes against the whole point of the change.
As I said, OpenEXR was just one of the many libraries we upgraded. Luckily, they didn't apply this motto, and backward compatibility was ensured by most of them. We didn't upgrade the ones which required major changes in our code, and may do the same with OpenEXR (stick with 1.6) if we can't patch 1.7. 

Jonathan Litt wrote:
This is also stated in the ChangeLog, which I would hope is required reading for any developer switching to 1.7.
I'm a bit curious about the fact that this entire time you've been creating exr files with truncated attribute names "prmanExrDriverAddDisplayWindowO" and "prmanExrDriverDisplayWindowOffs", and that has never been a concern? Did those just not get noticed before they became the de facto names instead of the un-truncated ones in all of your facility files? To be fair, channel names rather than attribute names are prime candidates to end up truncated since they often come from user input (e.g., AOV names) rather than, say, a driver developer. (Although the new metadata stuff in Nuke makes creating attributes more accessible to everyone.)
Your situation is a perfect example of why long attribute names are so badly needed in the first place. Adding a backwards compatibility mode goes against the whole point of the change. The only way long name support is going to work is if everyone starts using it, and nothing is stopping apps from doing their own truncation to prevent the flag. IF a 31-char truncation mode is added to the standard distribution it should be a runtime API option that defaults to off, and developers should be strongly encouraged to only enable it upon explicit user request as opposed to compiling it in. If Nuke or prman other apps/drivers wish to pass the runtime truncation option to the user, that's great. On the other hand suggesting that it's ok for developers to turn it off by making it be an official static option is not in the best interest of anyone. Of course I'm not talking about companies that use OpenEXR internally -- I'm talking about companies that ship products with OpenEXR support.
All that said, it couldn't hurt for this change to be publicized more loudly in any further 1.7 announcements. :) 

-Jonathan



On Aug 23, 2010, at 8:11 AM, Benoit Leveau wrote:

I saw that in ImfHeader.cpp, line 117 (version 1.7.0):

// If an OpenEXR file contains any attribute names, attribute type names
// or channel names longer than 31 characters, then the file cannot be
// read by older versions of the IlmImf library (up to OpenEXR 1.6.1).
// Before writing the file header, we check if the header contains
// any names longer than 31 characters; if it does, then we set the
// LONG_NAMES_FLAG in the file version number.  Older versions of the
// IlmImf library will refuse to read files that have the LONG_NAMES_FLAG 
// set.  Without the flag, older versions of the library would mis-
// interpret the file as broken.
This means that if you're writing an image with a channel name longer than 31 characters, - the previous behavior was to truncate the name (readable in all versions), - the new behavior is to set the flag and write the long names (readable in 1.7+ only).
This will require us to either truncate the channel names manually everywhere we use OpenEXr, or edit the OpenEXR library so it never sets the flag and truncates the names manually. 

Benoit

Jeff Clifford wrote:

Ah that would explain it.  Glad you found the difference.
Cheers,
Jeff.
Benoit Leveau wrote:

Hi,

That 0x400 value comes from the following in ImfVersion.h:

const int LONG_NAMES_FLAG       = 0x00000400;   // File contains long
                                              // attribute or channel
                                              // names

which also explains the rest of the diff (i.e., long names).
I guess images written with that flag are not supported by old versions of the library. I'll try to see which option controls that flag when writing images, should be easy now. 

Cheers,
Benoit

Benoit Leveau wrote:

Hi Jeff,

Thanks for the feedback! It's good to know it works on your side.
We used the configure script that comes with the library, and didn't use any specific option.
I just tried to use exrheader to compare files generated with 1.6 and 1.7. 
A diff of the output gives that interesting result:

4c4
< file format version: 2, flags 0x0
---

file format version: 2, flags 0x400

20c20
< prmanExrDriverAddDisplayWindowO (type int): 0
---

prmanExrDriverAddDisplayWindowOffset (type int): 0

22,23c22,23
< prmanExrDriverDisplayWindowOffs (type box2i): (0 0) - (0 0)
< rmanXMLStatisticsFileName (type string): "x+g"
---

prmanExrDriverDisplayWindowOffsetValues (type box2i): (0 0) - (0 0)
rmanXMLStatisticsFileName (type string): "(XXX" <= 'X' for some non-printable characters
There's probably something in our code that doesn't initialize that version/flag. 

Benoit

Jeff Clifford wrote:

Hi Benoit,

It may be worth stating what your configure options are?
The reason I say that is that we too rebuilt our internal apps and PRMan driver against 1.7.0 under Linux64. However we've not hit upon that message. We're able to use older OpenEXR lib apps (like Nuke) to read our 1.7.0 images fine. 

Jeff.


Benoit Leveau wrote:

Hi,
After re-compiling our renderman display driver with OpenEXR 1.7 libraries on linux 64bit, none of the images we generate can be loaded into applications that link with older versions of OpenEXR (e.g. Nuke, in-house tools). 

The error I get with our in-house player is:
OpenEXR input exception:: >> Cannot read image file "xxxxx.exr". The file format version number's flag field contains unrecognized flags. << 

I get the same exact error message in Nuke 6.0.
If I compare a file generated by 1.7 and 1.6.1 (or 1.4.0), the two files indeed differ, even in the first few bytes. It seems that the 1.7 libraries add new header flags, 
and this is not safely ignored by old versions.
IMHO, if files generated by 1.7 can't be loaded by current versions of Nuke, that may greatly affect its widespread usage. 

Thanks for your help,
Benoit


_______________________________________________
Openexr-devel mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/openexr-devel





_______________________________________________
Openexr-devel mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/openexr-devel






_______________________________________________
Openexr-devel mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/openexr-devel






_______________________________________________
Openexr-devel mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/openexr-devel
reply via email to
[Prev in Thread] Current Thread [Next in Thread]