[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: problem loading hdf5 file
|
From: |
E. Joshua Rigler |
|
Subject: |
Re: problem loading hdf5 file |
|
Date: |
Mon, 03 May 2004 09:36:20 -0600 |
The following quote is from the website used to download these hdf data
files:
All SORCE standard data product files are written using the NCSA
HDF5 file format. HDF5 is a heirarchical data format which
consists of two object types: groups and datasets. Groups are
analogous to directories or folders (all files have at least a
root or "/" group), whereas datasets may contain arrays, images
or tables.
I suspect they just lifted the description from the NCSA folks, because
as you (David) pointed out, these data are not actually being saved as
ordinary hdf5 "groups". I work fairly close to at least on of the PIs
on this project, so I'll see what the likelihood of getting them to
clean up their data files is, but somehow I doubt they'll be very
interested.
I don't really know how important it is to "fix" Octave to handle poorly
designed data files. It's probably just easier for me to write some
h5dump scripts for now. Thanks again for your feedback.
-EJR
On Mon, 2004-05-03 at 07:19, David Bateman wrote:
> Ok, the problem is simple to state, but a bit harder to fix... Octave
> only know two compund datatype. Complex and range, with the forms
>
> Type: struct {
> "real" +0 native double
> "imag" +8 native double
> } 16 bytes
>
> Type: struct {
> "base" +0 native double
> "limit" +8 native double
> "increment" +16 native double
> } 24 bytes
>
> Your files have compound datatypes, containing arbitrary values
> representing a structure... Octave assumes that such data will be
> passed to it as an HDF5 group, rather than a compound type of a
> dataset. "h5ls -v <file>" identifies the following compound types
> for your file
>
> Type: struct {
> "instrumentModeId" +16 32-bit big-endian integer
> "julianTimetag" +0 IEEE 64-bit big-endian float
> "version" +12 32-bit big-endian integer
> "minWavelength" +24 IEEE 64-bit big-endian float
> "maxWavelength" +32 IEEE 64-bit big-endian float
> "irradiance" +40 IEEE 64-bit big-endian float
> "irradianceUncertainty" +48 IEEE 64-bit big-endian float
> "quality" +56 IEEE 64-bit big-endian float
> } 64 bytes
>
> Type: struct {
> "julianTimetag" +0 IEEE 64-bit big-endian float
> "parameterName" +32 39-byte null-terminated ASCII
> string
> "value" +16 IEEE 64-bit big-endian float
> "uncertainty" +24 IEEE 64-bit big-endian float
> } 72 bytes
>
>
>
> Type: struct {
> "julianTimetag" +0 IEEE 64-bit big-endian float
> "averageJulianTime" +16 IEEE 64-bit big-endian float
> "averageJulianTimeStdev" +24 IEEE 64-bit big-endian float
> "correctedIrradiance" +32 IEEE 64-bit big-endian float
> "correctedIrradianceUncertainty" +40 IEEE 64-bit
> big-endian float
> "correctedIrradianceStdev" +48 IEEE 64-bit big-endian
> float
> "trueEarthIrradiance" +56 IEEE 64-bit big-endian float
> "trueEarthIrradianceUncertainty" +64 IEEE 64-bit
> big-endian float
> "trueEarthIrradianceStdev" +72 IEEE 64-bit big-endian
> float
> } 80 bytes
> Type: struct {
> "julianTimetag" +0 IEEE 64-bit big-endian float
> "averageJulianTimetag" +8 IEEE 64-bit big-endian float
> "averageJulianTimetagStDev" +16 IEEE 64-bit big-endian
> float
> "timeSpanInHours" +24 8-bit integer
> "diodeNumber" +28 32-bit big-endian integer
> "version" +32 16-bit big-endian integer
> "minWavelengthInBandpass" +36 IEEE 32-bit big-endian
> float
> "maxWavelengthInBandpass" +40 IEEE 32-bit big-endian
> float
> "medianIrradiance" +48 IEEE 64-bit big-endian float
> "averageIrradiance" +56 IEEE 64-bit big-endian float
> "absoluteUncertainty" +64 IEEE 64-bit big-endian float
> "measurementPrecision" +72 IEEE 64-bit big-endian float
> "calculationPrecision" +80 IEEE 64-bit big-endian float
> "degradationModel" +88 8-bit integer
> "numberOfPoints" +92 32-bit big-endian integer
> } 96 bytes
>
> >From which you creat a vector of these compound types. So there are two
> choices, can your software save these as groups? If so then the easiest
> would be to save the data as group, where the names of each subgroup
> takes on the names of the variables in your compound type. Each of
> these could be a normal HDF5 matrix of float or integers. Octave will
> then interpret this as a structure with the array values you'd expect.
> Your HDF5 should have "h5ls -r -v" give something like (description
> written by hand, so its a fake example).
>
> a Group
> Location: 0:1:0:1576
> Links: 1
> a/julianTimetag Dataset {1/1, 10/10}
> Location: 0:1:0:9960
> Links: 1
> Modified: 2004-05-03 15:07:39 CEST
> Storage: 80 logical bytes, 80 allocated bytes, 100.00% utilization
> Type: native double
> a/parameterName Dataset {1/1, 10/10}
> Location: 0:1:0:1952
> Links: 1
> Modified: 2004-05-03 15:07:39 CEST
> Storage: 390 logical bytes, 390 allocated bytes, 100.00% utilization
> Type: 39-byte null-terminated ASCII string
> a/uncertainity Dataset {1/1, 10/10}
> Location: 0:1:0:7072
> Links: 1
> Modified: 2004-05-03 15:07:39 CEST
> Storage: 80 logical bytes, 80 allocated bytes, 100.00% utilization
> Type: native double
> a/value Dataset {1/1, 10/10}
> Location: 0:1:0:11480
> Links: 1
> Modified: 2004-05-03 15:07:39 CEST
> Storage: 80 logical bytes, 80 allocated bytes, 100.00% utilization
> Type: native double
>
> The second option of modifiying octave to read your datatype is not
> that easy, in that how do we create the correct compound datatype,
> since we have to consider the datatype as arbitrary? What I see might
> be possible is to use H5Tget_nmembers to find the number of member
> elements in the compound type, and then H5Tget_member_name and
> H5Tget_member_type, etc to bit by bit reconstruct a compatiable
> compound type to hold your data to be imported as a structure.
>
> This is a non-trivial task, and is probably a few days work.. I'm
> willing to give some pointers if you want to try and write the code
> yourself, but I don't have to time to do this for you....
>
> Regards
> David
>
> According to E. Joshua Rigler <address@hidden> (on 05/01/04):
> > I'm using 2.1.57 and the latest octave-forge (not CVS). The data *is*
> > somewhat complex as it contains solar spectral measurements at many
> > wavelengths, and from different instruments, with corresponding
> > uncertainties, etc. I've attached the one test file I have, but here's
> > the output you requested anyway:
> >
> > h5ls:
> > Solar\ Spectral\ Irradiance Dataset {1083}
> > Solar\ and\ Geophysical\ Parameters Dataset {4}
> > Total\ Solar\ Irradiance Dataset {1}
> > XPS\ EUV\ Solar\ Spectrum Dataset {11}
> >
> > h5ls -r:
> > /Solar\ Spectral\ Irradiance Dataset {1083}
> > /Solar\ and\ Geophysical\ Parameters Dataset {4}
> > /Total\ Solar\ Irradiance Dataset {1}
> > /XPS\ EUV\ Solar\ Spectrum Dataset {11}
> >
> >
> > Also, you might want to read this page for a detailed description of the
> > different data types (there's quite the variety of different floats,
> > integers, and strings in each file):
> >
> > http://daac.gsfc.nasa.gov/upperatm/sorce/level3_ssi.html
> >
> > Thanks for looking into this with me.
> >
> > -EJR
> >
> >
> > On Fri, 2004-04-30 at 17:09, David Bateman wrote:
> > > Can you send me the files, as the ftp site says it can't set guest
> > > privileges. Also what does "h5ls <file>" and "h5ls -r <file>" show?
> > >
> > > It is is relatively simple data, there is no reason the file shouldn't
> > > be able to be loaded in octave. BTW, what version of octave are you
> > > using? There were major changes in the HDF code around 2.1.53, so I
> > > suggest you use a new version of octave.
> > >
> > > Regards
> > > David
> >
--
To announce that there must be no criticism of the president,
or that we are to stand by the president, right or wrong,
is not only unpatriotic and servile, but is morally treasonable
to the American public. -- Theodore Roosevelt, 1918.
-------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.
Octave's home on the web: http://www.octave.org
How to fund new projects: http://www.octave.org/funding.html
Subscription information: http://www.octave.org/archive.html
-------------------------------------------------------------