/
HDF/HDF-EOS Workshop XV HDF/HDF-EOS Workshop XV

HDF/HDF-EOS Workshop XV - PowerPoint Presentation

danika-pritchard
danika-pritchard . @danika-pritchard
Follow
436 views
Uploaded On 2017-08-01

HDF/HDF-EOS Workshop XV - PPT Presentation

1 Tools to Improve the Usability of NASA HDF Data Kent Yang and Joe Lee The HDF Group April 17 2012 Tutorial Outline Introduction 5 Tools H4CF Conversion Toolkit h4cf h4toh5 Conversion Tool ID: 574878

eos hdf workshop april hdf eos april workshop 2012 eos5 netcdf file aug eos2 data h4cf tools nasa h4toh5

Share:

Link:

Embed:

Download Presentation from below link

Download Presentation The PPT/PDF document "HDF/HDF-EOS Workshop XV" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.


Presentation Transcript

Slide1

HDF/HDF-EOS Workshop XV

1

Tools to Improve the Usability of NASA HDF Data

Kent Yang and Joe LeeThe HDF Group

April 17, 2012Slide2

Tutorial Outline

Introduction

5 Tools

H4CF Conversion Toolkit (h4cf)

h4toh5 Conversion Tool (h4toh5

)HDF-EOS5 to netCDF-4 Converter (eos52nc4

)

HDF-EOS5 Augmentation Tool (

aug_eos5

)

HDF-EOS2 Dumper (eos2dump)Demo

HDF/HDF-EOS Workshop XV

2

April 17, 2012Slide3

Who Said Users Don’t Matter?

April 17, 2012

HDF/HDF-EOS Workshop XV

3Slide4

NASA

HDF/HDF-EOS

Data

Usability of NASA HDF Data

April 17, 2012

HDF/HDF-EOS Workshop XV

4

IDL

MATLAB

Panoply

IDV

A school teacher’s view

Panoply

IDV

A school teacher’s expectationSlide5

The 5 Tools

April 17, 2012

HDF/HDF-EOS Workshop XV

5

Improve the usability of NASA DataSupported by The HDF Group

Funded by NASA / Raytheonhttp://hdfeos.org/software/tool.phpSlide6

hdf4

5 Tools that Improve Usability

April 17, 2012

HDF/HDF-EOS Workshop XV

6

eos2

eos5

ASCII

NETCDF-4

NETCDF-3

Panoply / IDV

MATLAB

IDL

NCLSlide7

H4CF Conversion Toolkit

Follow

CF conventionBoth Library and

Toolshttp://hdfeos.org/software/h4cflibApril 17, 2012HDF/HDF-EOS Workshop XV

7

hdf4

eos2

NETCDF-4

NETCDF-3

h4cf

Panoply / IDVSlide8

H4CF Conversion Toolkit

Two ways to build:

Use

configure script.

Edit Makefile

template.

April 17, 2012

HDF/HDF-EOS Workshop XV

8Slide9

H4CF Conversion Library

Variables (datasets) list

Attributes (metadata) list

All Vars & Attrs follow CF conventionsDevelop your own CF application!

April 17, 2012

HDF/HDF-EOS Workshop XV

9Slide10

H4CF Library APIs Example

// open the example HDF file

h4cf_open("geo.hdf");

// HDF file attributes are obtained in a C++ STL map

object

const

map<string, string>

file_attrs

= h4cf_get_file_attrs();

// HDF file variables are obtained in a

C++ STL

list object

const

list<

var

*>

pvars

= h4cf_get_

vars

();

// close the HDF file

h4cf_close();

April 17, 2012

HDF/HDF-EOS Workshop XV

10Slide11

H4CF Conversion Tools

2 Tools based on H4CF APIs:

cdldumper

h4tonccfApril 17, 2012

HDF/HDF-EOS Workshop XV

11Slide12

cdldumper

Usage:

cdldumper filename.hdfDump file in Common Data Language

.Useful for quick check and debugging purpose.April 17, 2012

HDF/HDF-EOS Workshop XV

12Slide13

cdldumper Demo

April 17, 2012

HDF/HDF-EOS Workshop XV

13

Number of attributes=1

_

FillValue

=255Slide14

cdldumper Demo

April 17, 2012

HDF/HDF-EOS Workshop XV

14

%./

cdldumper -H AMSR_E_L3_5DaySnow_V09_20050126.hdf

netCDF

AMSR_E_L3_5DaySnow_V09_20050126.hdf {

dimensions:

XDim_Northern_Hemisphere

= 721,

XDim_Southern_…

variables:

ubyte

Flags_NorthernPentad

(

YDim_Northern_Hemis

Flags_NorthernPentad:_FillValue

= 255;

Flags_NorthernPentad:origname

= "/Northern…

Flags_NorthernPentad:long_name

= "/Northern…

Flags_NorthernPentad:coordinates

= "latitude…

Number of attributes=4

_

FillValue

=255

orig_name

long_name

coordinatesSlide15

h4tonccf

Usage: h4tonccf

filename.hdf [filename.nc]generates either netcdf-3 or netcdf-4 file.follows

netcdf classic model.April 17, 2012HDF/HDF-EOS Workshop XV

15Slide16

h4tonccf Demo

April 17, 2012

HDF/HDF-EOS Workshop XV

16Slide17

Don’t Follow CF Conventions Blindly

April 17, 2012

HDF/HDF-EOS Workshop XV

17[1] CF: scale * value + offset[2] MODIS: scale * (value – offset)

Scale = 0.01Value = 5000

Offset = -15000.0[1] 0.01 * 5000 + (-15000) =

-14950

[2] 0.01 * (5000 – (-15000)) = 200Slide18

-14950 Kelvin?

April 17, 2012

HDF/HDF-EOS Workshop XV

18Slide19

MODIS Scale / Offset Correction

HDF/HDF-EOS Workshop XV

19

April 18, 2012Slide20

Converted File Size Comparison

April 17, 2012

HDF/HDF-EOS Workshop XV

20

HDF-EOS2

Netcdf-4

Netcdf-3

9XSlide21

Limitation – No Support Yet

April 17, 2012

HDF/HDF-EOS Workshop XV

21MISR (SOM/LAMAZ)MERRA

AVHRRLISTO

QuikSCAT/SeaWind

MOPITT L2

etc.Slide22

h4toh5 Conversion Tool

Convert

any

hdf4 to hdf5.http://hdfgroup.org/h4toh5April 17, 2012

HDF/HDF-EOS Workshop XV

22

eos2

NETCDF-4

h4toh5

Usage:

h4toh5

eos

–nc4

in

.hdf out.nc

Static Binary

w/ –

eos

& -nc4

:

http://hdfeos.org/software/h4toh5/bin/

Slide23

h4tonccf vs. h4toh5

April 17, 2012

HDF/HDF-EOS Workshop XV

23

h4tonccf

h4toh5

w/ -

eos

–nc4

Handle

non-eos2 hdf4

Yes

No

Follow

CF conventions

Yes

No

Output

netcdf

classic

Yes

No

Output

netcdf

enhanced

No

YesSlide24

eos52nc4

converts hdf-eos5 to netcdf-4.follows

netcdf enhanced model.does not follow CF conventions. Usage:

eos52nc4 test.he5 test.nchttp://hdfeos.org/software/convert_hdfeos5.phpApril 17, 2012

HDF/HDF-EOS Workshop XV

24

eos5

NETCDF-4

eos52nc4Slide25

aug_eos5

augments hdf-eos5 to netcdf-4.Both

HDF-EOS5 and NetCDF4 can access it.

Usage: aug_eos5 [-i|-f map] test.h5modifies the input file directly. does not follow CF conventions.

http://hdfeos.org/software/aug_hdfeos5.php

April 17, 2012

HDF/HDF-EOS Workshop XV

25

eos5

NETCDF-4

aug_eos5Slide26

hdf5

netcdf4

Augmentation Concept

April 17, 2012

HDF/HDF-EOS Workshop XV

26

hdf-eos5

HDF-EOS5

API

NETCDF-4

API

aug_eos5

augmentation

hdf-eos5

HDF5 APISlide27

Why NetCDF-4 Doesn’t Work

April 17, 2012

HDF/HDF-EOS Workshop XV

27

[nLevels][nTimes]

NETCDF-4

API

hdf-eos5Slide28

What aug_eos5 Does

April 17, 2012

HDF/HDF-EOS Workshop XV

28

[nLevels][nTimes]

NETCDF-4

API

hdf-eos5Slide29

Options in aug_eos5

Default option

-i option

-f map_file optionApril 17, 2012

HDF/HDF-EOS Workshop XV

29Slide30

Default option in aug_eos5

Grid

XDim

, YDim: calculated Other dims: default fill valueSwath / ZAAll dims: default fill value

April 17, 2012

HDF/HDF-EOS Workshop XV

30Slide31

Better -i

option in aug_eos5

GridXDim, YDim: calculated

Other dims: 1,2,…,nSwath / ZAAll dims: 1,2,…,n

April 17, 2012

HDF/HDF-EOS Workshop XV

31Slide32

Best -f

option in aug_eos5

Needs an input map file.Input file specifies 3 rules:

What to be filled inHow to fill inWhere to get the valuesMost useful and flexible option

April 17, 2012

HDF/HDF-EOS Workshop XV

32Slide33

aug_eos5 Input File Format

Each line in input file contains

How: 0,1,2

What: dim nameWhere: data file or existing variable in file. (Optional) Multiple grid/swath/

za case only:grid/swath/

za name:<name>

April 17, 2012

HDF/HDF-EOS Workshop XV

33Slide34

aug_eos5 Input File Example

0

nTimes

Time1 nLevels data.txt2 nCandidates

April 17, 2012

HDF/HDF-EOS Workshop XV

34

Copy the data

Read file

data.txt

Fill 1,2,…, n

How

What

Where

95% cases – Copy

the dataSlide35

Multiple Swath Case

swath name:

BrO 0 nTimes

Time1 nLevels data1.txt2 nCandidatesApril 17, 2012

HDF/HDF-EOS Workshop XV

35

How

What

Where

swath name:

BrO

column

2

nTimes

1

nLevels

data2.txt

MultipleSlide36

aug_eos5 vs

eos52nc4April 17, 2012

HDF/HDF-EOS Workshop XV

36

aug_eos5

eos52nc4

hdfeos-5

Yes

No

netcdf-4

Yes

YesSlide37

HIRDLS/MLS Team Use aug_eos5

April 17, 2012

HDF/HDF-EOS Workshop XV

37

So You Can!Slide38

eos2 dump

HDF-EOS2 Dumper Tool

Useful for

tools

that can’t handle

some projections

.

April 17, 2012

HDF/HDF-EOS Workshop XV

38

eos2

ASCII

MATLAB

IDL

NCL

http://hdfeos.org/zooSlide39

Important Tool for Zoo Maintenance

April 17, 2012

HDF/HDF-EOS Workshop XV

39Slide40

Why Tools Need HDF-OES2 Dumper

April 17, 2012

HDF/HDF-EOS Workshop XV

40

Lat / Lon

in

ASCII

MATLAB

IDL

NCL

HDF-EOS2

dumper

LaRC

MISR

SOM Projection

HDF-EOS2

Lat / LonSlide41

HDF-EOS2 Dumper Options

Usage: eos2dump

options filename.hdf What:

1=lat / 2 =lonFormat: m=5 values per line

April 17, 2012

HDF/HDF-EOS Workshop XV

41Slide42

eos2dump Latitude Example

eos2dump –

c1

filename.hdf > latitude.txt What: latFormat: one value per line (default)April 17, 2012

HDF/HDF-EOS Workshop XV

42

90

80

70

60

50

40

30

20

10

0

-10

-20

-30Slide43

eos2dump Longitude Example

eos2dump –

c2m filename.hdf > longitude.txt

What: lonFormat: 5 values per lineMATLAB

cannot read it.

April 17, 2012

HDF/HDF-EOS Workshop XV

43

180 175 170 165 160

155 150 145 140 135

130 125 120 115 110Slide44

Got Tools?

April 17, 2012

HDF/HDF-EOS Workshop XV

44Slide45

hdf4

Enjoy the NASA HDF Data!

April 17, 2012

HDF/HDF-EOS Workshop XV

45

eos2

eos5

ASCII

NETCDF-4

NETCDF-3

Panoply / IDV

MATLAB

IDL

NCL

NASA HDF

data

u

sers Slide46

… and observe

the Earth!April 17, 2012

HDF/HDF-EOS Workshop XV

46Slide47

HDF/HDF-EOS Workshop XV

47

Thank you !

April 17, 2012Slide48

Acknowledgements

HDF/HDF-EOS Workshop XV

48

April 17, 2012

This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA) and by cooperative agreement number NNX08AO77A from the NASA. Any opinions, findings, conclusions, or

recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.Slide49

Questions/comments?

HDF/HDF-EOS Workshop XV

49

April 17, 2012