1 Tools to Improve the Usability of NASA HDF Data Kent Yang and Joe Lee The HDF Group April 17 2012 Tutorial Outline Introduction 5 Tools H4CF Conversion Toolkit h4cf h4toh5 Conversion Tool ID: 574878
Download Presentation The PPT/PDF document "HDF/HDF-EOS Workshop XV" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
HDF/HDF-EOS Workshop XV
1
Tools to Improve the Usability of NASA HDF Data
Kent Yang and Joe LeeThe HDF Group
April 17, 2012Slide2
Tutorial Outline
Introduction
5 Tools
H4CF Conversion Toolkit (h4cf)
h4toh5 Conversion Tool (h4toh5
)HDF-EOS5 to netCDF-4 Converter (eos52nc4
)
HDF-EOS5 Augmentation Tool (
aug_eos5
)
HDF-EOS2 Dumper (eos2dump)Demo
HDF/HDF-EOS Workshop XV
2
April 17, 2012Slide3
Who Said Users Don’t Matter?
April 17, 2012
HDF/HDF-EOS Workshop XV
3Slide4
NASA
HDF/HDF-EOS
Data
Usability of NASA HDF Data
April 17, 2012
HDF/HDF-EOS Workshop XV
4
IDL
MATLAB
Panoply
IDV
A school teacher’s view
Panoply
IDV
A school teacher’s expectationSlide5
The 5 Tools
April 17, 2012
HDF/HDF-EOS Workshop XV
5
Improve the usability of NASA DataSupported by The HDF Group
Funded by NASA / Raytheonhttp://hdfeos.org/software/tool.phpSlide6
hdf4
5 Tools that Improve Usability
April 17, 2012
HDF/HDF-EOS Workshop XV
6
eos2
eos5
ASCII
NETCDF-4
NETCDF-3
Panoply / IDV
MATLAB
IDL
NCLSlide7
H4CF Conversion Toolkit
Follow
CF conventionBoth Library and
Toolshttp://hdfeos.org/software/h4cflibApril 17, 2012HDF/HDF-EOS Workshop XV
7
hdf4
eos2
NETCDF-4
NETCDF-3
h4cf
Panoply / IDVSlide8
H4CF Conversion Toolkit
Two ways to build:
Use
configure script.
Edit Makefile
template.
April 17, 2012
HDF/HDF-EOS Workshop XV
8Slide9
H4CF Conversion Library
Variables (datasets) list
Attributes (metadata) list
All Vars & Attrs follow CF conventionsDevelop your own CF application!
April 17, 2012
HDF/HDF-EOS Workshop XV
9Slide10
H4CF Library APIs Example
// open the example HDF file
h4cf_open("geo.hdf");
// HDF file attributes are obtained in a C++ STL map
object
const
map<string, string>
file_attrs
= h4cf_get_file_attrs();
// HDF file variables are obtained in a
C++ STL
list object
const
list<
var
*>
pvars
= h4cf_get_
vars
();
// close the HDF file
h4cf_close();
April 17, 2012
HDF/HDF-EOS Workshop XV
10Slide11
H4CF Conversion Tools
2 Tools based on H4CF APIs:
cdldumper
h4tonccfApril 17, 2012
HDF/HDF-EOS Workshop XV
11Slide12
cdldumper
Usage:
cdldumper filename.hdfDump file in Common Data Language
.Useful for quick check and debugging purpose.April 17, 2012
HDF/HDF-EOS Workshop XV
12Slide13
cdldumper Demo
April 17, 2012
HDF/HDF-EOS Workshop XV
13
Number of attributes=1
_
FillValue
=255Slide14
cdldumper Demo
April 17, 2012
HDF/HDF-EOS Workshop XV
14
%./
cdldumper -H AMSR_E_L3_5DaySnow_V09_20050126.hdf
netCDF
AMSR_E_L3_5DaySnow_V09_20050126.hdf {
dimensions:
XDim_Northern_Hemisphere
= 721,
XDim_Southern_…
variables:
ubyte
Flags_NorthernPentad
(
YDim_Northern_Hemis
…
Flags_NorthernPentad:_FillValue
= 255;
Flags_NorthernPentad:origname
= "/Northern…
Flags_NorthernPentad:long_name
= "/Northern…
Flags_NorthernPentad:coordinates
= "latitude…
Number of attributes=4
_
FillValue
=255
orig_name
long_name
coordinatesSlide15
h4tonccf
Usage: h4tonccf
filename.hdf [filename.nc]generates either netcdf-3 or netcdf-4 file.follows
netcdf classic model.April 17, 2012HDF/HDF-EOS Workshop XV
15Slide16
h4tonccf Demo
April 17, 2012
HDF/HDF-EOS Workshop XV
16Slide17
Don’t Follow CF Conventions Blindly
April 17, 2012
HDF/HDF-EOS Workshop XV
17[1] CF: scale * value + offset[2] MODIS: scale * (value – offset)
Scale = 0.01Value = 5000
Offset = -15000.0[1] 0.01 * 5000 + (-15000) =
-14950
[2] 0.01 * (5000 – (-15000)) = 200Slide18
-14950 Kelvin?
April 17, 2012
HDF/HDF-EOS Workshop XV
18Slide19
MODIS Scale / Offset Correction
HDF/HDF-EOS Workshop XV
19
April 18, 2012Slide20
Converted File Size Comparison
April 17, 2012
HDF/HDF-EOS Workshop XV
20
HDF-EOS2
Netcdf-4
Netcdf-3
9XSlide21
Limitation – No Support Yet
April 17, 2012
HDF/HDF-EOS Workshop XV
21MISR (SOM/LAMAZ)MERRA
AVHRRLISTO
QuikSCAT/SeaWind
MOPITT L2
etc.Slide22
h4toh5 Conversion Tool
Convert
any
hdf4 to hdf5.http://hdfgroup.org/h4toh5April 17, 2012
HDF/HDF-EOS Workshop XV
22
eos2
NETCDF-4
h4toh5
Usage:
h4toh5
–
eos
–nc4
in
.hdf out.nc
Static Binary
w/ –
eos
& -nc4
:
http://hdfeos.org/software/h4toh5/bin/
Slide23
h4tonccf vs. h4toh5
April 17, 2012
HDF/HDF-EOS Workshop XV
23
h4tonccf
h4toh5
w/ -
eos
–nc4
Handle
non-eos2 hdf4
Yes
No
Follow
CF conventions
Yes
No
Output
netcdf
classic
Yes
No
Output
netcdf
enhanced
No
YesSlide24
eos52nc4
converts hdf-eos5 to netcdf-4.follows
netcdf enhanced model.does not follow CF conventions. Usage:
eos52nc4 test.he5 test.nchttp://hdfeos.org/software/convert_hdfeos5.phpApril 17, 2012
HDF/HDF-EOS Workshop XV
24
eos5
NETCDF-4
eos52nc4Slide25
aug_eos5
augments hdf-eos5 to netcdf-4.Both
HDF-EOS5 and NetCDF4 can access it.
Usage: aug_eos5 [-i|-f map] test.h5modifies the input file directly. does not follow CF conventions.
http://hdfeos.org/software/aug_hdfeos5.php
April 17, 2012
HDF/HDF-EOS Workshop XV
25
eos5
NETCDF-4
aug_eos5Slide26
hdf5
netcdf4
Augmentation Concept
April 17, 2012
HDF/HDF-EOS Workshop XV
26
hdf-eos5
HDF-EOS5
API
NETCDF-4
API
aug_eos5
augmentation
hdf-eos5
HDF5 APISlide27
Why NetCDF-4 Doesn’t Work
April 17, 2012
HDF/HDF-EOS Workshop XV
27
[nLevels][nTimes]
NETCDF-4
API
hdf-eos5Slide28
What aug_eos5 Does
April 17, 2012
HDF/HDF-EOS Workshop XV
28
[nLevels][nTimes]
NETCDF-4
API
hdf-eos5Slide29
Options in aug_eos5
Default option
-i option
-f map_file optionApril 17, 2012
HDF/HDF-EOS Workshop XV
29Slide30
Default option in aug_eos5
Grid
XDim
, YDim: calculated Other dims: default fill valueSwath / ZAAll dims: default fill value
April 17, 2012
HDF/HDF-EOS Workshop XV
30Slide31
Better -i
option in aug_eos5
GridXDim, YDim: calculated
Other dims: 1,2,…,nSwath / ZAAll dims: 1,2,…,n
April 17, 2012
HDF/HDF-EOS Workshop XV
31Slide32
Best -f
option in aug_eos5
Needs an input map file.Input file specifies 3 rules:
What to be filled inHow to fill inWhere to get the valuesMost useful and flexible option
April 17, 2012
HDF/HDF-EOS Workshop XV
32Slide33
aug_eos5 Input File Format
Each line in input file contains
How: 0,1,2
What: dim nameWhere: data file or existing variable in file. (Optional) Multiple grid/swath/
za case only:grid/swath/
za name:<name>
April 17, 2012
HDF/HDF-EOS Workshop XV
33Slide34
aug_eos5 Input File Example
0
nTimes
Time1 nLevels data.txt2 nCandidates
April 17, 2012
HDF/HDF-EOS Workshop XV
34
Copy the data
Read file
data.txt
Fill 1,2,…, n
How
What
Where
95% cases – Copy
the dataSlide35
Multiple Swath Case
swath name:
BrO 0 nTimes
Time1 nLevels data1.txt2 nCandidatesApril 17, 2012
HDF/HDF-EOS Workshop XV
35
How
What
Where
swath name:
BrO
column
2
nTimes
1
nLevels
data2.txt
MultipleSlide36
aug_eos5 vs
eos52nc4April 17, 2012
HDF/HDF-EOS Workshop XV
36
aug_eos5
eos52nc4
hdfeos-5
Yes
No
netcdf-4
Yes
YesSlide37
HIRDLS/MLS Team Use aug_eos5
April 17, 2012
HDF/HDF-EOS Workshop XV
37
So You Can!Slide38
eos2 dump
HDF-EOS2 Dumper Tool
Useful for
tools
that can’t handle
some projections
.
April 17, 2012
HDF/HDF-EOS Workshop XV
38
eos2
ASCII
MATLAB
IDL
NCL
http://hdfeos.org/zooSlide39
Important Tool for Zoo Maintenance
April 17, 2012
HDF/HDF-EOS Workshop XV
39Slide40
Why Tools Need HDF-OES2 Dumper
April 17, 2012
HDF/HDF-EOS Workshop XV
40
Lat / Lon
in
ASCII
MATLAB
IDL
NCL
HDF-EOS2
dumper
LaRC
MISR
SOM Projection
HDF-EOS2
Lat / LonSlide41
HDF-EOS2 Dumper Options
Usage: eos2dump
options filename.hdf What:
1=lat / 2 =lonFormat: m=5 values per line
April 17, 2012
HDF/HDF-EOS Workshop XV
41Slide42
eos2dump Latitude Example
eos2dump –
c1
filename.hdf > latitude.txt What: latFormat: one value per line (default)April 17, 2012
HDF/HDF-EOS Workshop XV
42
90
80
70
60
50
40
30
20
10
0
-10
-20
-30Slide43
eos2dump Longitude Example
eos2dump –
c2m filename.hdf > longitude.txt
What: lonFormat: 5 values per lineMATLAB
cannot read it.
April 17, 2012
HDF/HDF-EOS Workshop XV
43
180 175 170 165 160
155 150 145 140 135
130 125 120 115 110Slide44
Got Tools?
April 17, 2012
HDF/HDF-EOS Workshop XV
44Slide45
hdf4
Enjoy the NASA HDF Data!
April 17, 2012
HDF/HDF-EOS Workshop XV
45
eos2
eos5
ASCII
NETCDF-4
NETCDF-3
Panoply / IDV
MATLAB
IDL
NCL
NASA HDF
data
u
sers Slide46
… and observe
the Earth!April 17, 2012
HDF/HDF-EOS Workshop XV
46Slide47
HDF/HDF-EOS Workshop XV
47
Thank you !
April 17, 2012Slide48
Acknowledgements
HDF/HDF-EOS Workshop XV
48
April 17, 2012
This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA) and by cooperative agreement number NNX08AO77A from the NASA. Any opinions, findings, conclusions, or
recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.Slide49
Questions/comments?
HDF/HDF-EOS Workshop XV
49
April 17, 2012