Ice Bridge Data Management - PowerPoint Presentation

Slide1

Ice Bridge Data Management

Scope, Challenges, Solutions

Ron Weaver, NSIDC DAAC Manager

National Snow and Ice Data Center

Supporting

Cryospheric

Research Since 1976

Topics

Vision, Guiding PrinciplesScience GoalsData Management Goals In Response to Science GoalsProposed Activities

ScheduleChallengesQuestions

Next StepsIceBridge Data Management. September 2009

2

The Global Vision

Provide timely access to data and productsProvide products useful and understood by the broader data communityCampaign data available before next campaign

Science Goals - IceBridge

Ice Sheet Surface

elevation data after ICESat-1 mission

endsAreas of rapid ice sheet change, Sea ice mass estimates

Computed

measurements critical to ice modelsBed topography, grounding lines, ice and snow thicknessLink ICESat-1 to ICESat-2 through baseline inter-calibration measurementsCampaigns to support sea ice (thickness, type, extent)

Technology incubator and test-bedAirborne and satellite sensors, UAV, photon counting, cross-track measurementsIceBridge Data Management. September 20094

Data System (at NSIDC)

Goals

Data system supports science goals and user needs

Provide appropriate and adaptable data services and tools for these data

Provide (and/or promote) understandable, accessible, and relevant data products to broader science community

Manage data sets from IceBridge, ICESat-1 and WFF ATM (via funded ACCESS project) in a coherent and interoperable mannerPrototype

the ICESat-2 data systemData StewardshipIceBridge Data Management. September 20095

CyberInfrastructure

(and related) Challenges

Understanding the scientific, especially interdisciplinary, needs

Data acquisition—timeliness, consistent formats, easy metadata creation, research vs. operational collectionData system automation and optimization

Data for models and synthesis—matching data to model needsStewardship in a distributed environment (e.g. tracking provenance)

Sustained preservationIceBridge Data Management. September 20096

Conceptual Plan

Work (lots of it) on harmonizing data for science user communities“getting things to play well together”

Building products (precursors to products)

Integrated Tools (and Services)Use

existing NSIDC systems, for data management

Could be distributed architecture that takes advantage of existing instrument data archivesIceBridge Data Management. September 20097

Schematic OIB Data System Architecture

Instrument, Flight Dynamics, Ancillary Data Sources

Instrument

Facility

DM System

Aggregator Services

DM System

NSIDC DAAC DM System

(ECS, non-ECS)

User Interfaces, Tools and Services

Users

ECHO/GCMD Metadata Database

Data flows

Metadata flows

Data Types

L0

L1

L2

L3

Lidars

ATM

GLAS

LVIS

UAF

LiDAR

Ice Radars

PARIS

KU-Ice

Ku-Snow

…

…

Gravity

Magnetics

Profiles/Swath

Points - Waveforms

Point cloud

Waveform

DEM

DEM

surface

Profiles/Swath

Return Energy - Waveforms

Ice in-situ

layers

Snow

Accum

.

DEM

bed

profiles

profiles

Gravity

Potential

surface

Aeroma

g

Potential

surface

Surface maps

Sea ice – ice sheets

Ice sheet roughness

External Products and Services

Aggregation Services

Models

Data Processing Levels

Services

Data Castings

Tools

USERS

ICE

CAP

ICE

CAP

LDEO

Proposed Task Sequencing

ESTABLISH OIB PROJECT TEAM

Coordinate DAAC OIB data mgmt. activities at NSIDC

Coordinate NSIDC activities with:

Science team

Instrument teams

ESDIS and

HQ

CONDUCT DATA ASSESSMENT

Establish OIB initial baseline data list

Characterize existing data formats, metadata processes, services, etc.

Determine initial LOS for all OIB data sets

Assess tools and services needed by OIB Science and Instrument Teams and User Community

Provide NASA HQ with work plan with cost estimates for remaining work

BROKER OIB DATA SETS

Metadata entries in NSIDC data catalog with LOS designated

Collect/extract/reformat documentation and metadata

Metadata entries transferred to ECHO and GCMD

ESTABLISH LIST OF DATA FOR INGEST AT NSIDC

Determine LOS and resources needed for transfer

Negotiate agreements with Instrument

Team(s

)

Determine transfer phasing

EXTEND TOOLS AND SERVICES TO DATA AND PRODUCTS

Determine phasing of development efforts

IMPLEMENT OIB DATA PORTAL

Rapidly implement focused entry point for OIB data and services

From metadata entries in NSIDC and ECHO data catalogs with LOS designated

Reuse/Augment

SearchLight

, CCAWS/SAGE technologies

IMPROVE INSTRUMENT TEAM DATA MANAGEMENT PRACTICES

Consult with Instrument Teams

IMPLEMENT DATA AGGREGATION SERVICES

INGEST DATA INTO NSIDC VO OR ECS SYSTEMS

Phased ingest based on established priority

A

C

D

E

F

H

I

G

B

Task Sequencing – After Preliminary Discussions With NASA HQ

ESTABLISH OIB PROJECT TEAM

Coordinate DAAC OIB data mgmt. activities at NSIDC

Coordinate NSIDC activities with:

Science team

Instrument teams

ESDIS and

HQ

CONDUCT DATA ASSESSMENT

Establish OIB initial baseline data list

Characterize existing data formats, metadata processes, services, etc.

Determine initial LOS for all OIB data sets

Assess tools and services needed by OIB Science and Instrument Teams and User Community

Provide NASA HQ with work plan with cost estimates for remaining work

BROKER OIB DATA SETS

Metadata entries in NSIDC data catalog with LOS designated

Collect/extract/reformat documentation and metadata

Metadata entries transferred to ECHO and GCMD

ESTABLISH LIST OF DATA FOR INGEST AT NSIDC

Determine LOS and resources needed for transfer

Negotiate agreements with Instrument

Team(s

)

Determine transfer phasing

EXTEND TOOLS AND SERVICES TO DATA AND PRODUCTS

Determine phasing of development efforts

IMPLEMENT OIB DATA PORTAL

Rapidly implement focused entry point for OIB data and services

From metadata entries in NSIDC and ECHO data catalogs with LOS designated

Reuse/Augment

SearchLight

, CCAWS/SAGE technologies

IMPROVE INSTRUMENT TEAM DATA MANAGEMENT PRACTICES

Consult with Instrument Teams

IMPLEMENT DATA AGGREGATION SERVICES

INGEST DATA INTO NSIDC VO OR ECS SYSTEMS

Phased ingest based on established priority

A

C

D

E

F

H

I

G

B

PROVIDE APPROPRIATE DATA PRODUCTS DERIVED FROM INGEST DATA STREAMS

Current NSIDC Schematic Architecture

Ingest Paths

Archival Systems

Data & Metadata

Production

Interfaces

& GUIs

Services

EOS Satellite Data

V0 Ingest Paths

PDR

Server

PM

Ingest

Searchlight

Ingest (TBD)

PM

Production

Manual Ingest

WIST

DP

Interface

Machine to machine

Sub setters

DataPool

SAN

ECS

ECHO

V0 SAN

V0 MDDB

(to be replaced)

SearchLight

OIB Portal

Legacy GUIs

SAGE CCAWS GUI

Service Casting

SLight

GUI

Lines indicate data and metadata flows

Dotted outlines are functions under development

Grayed out items are to be replaced

Rose colored items are proposed

Data Sets - a very preliminary list

Instrument

Data System?

NSIDC Ingest?

Data Levels

LVIS

Yes, formats defined, ftp delivery

Yes if needed ( use distributed DM?)

L0 - L3

ATM

Yes, formats defined, ftp delivery

Yes, planned

L0 - L3

PARIS

?

?

L0

Gravity

?

?

?

KU ice penetrating radar

ftp delivery of higher lvl prods.

Yes, KU has suggested L0 in NSIDC syst

L0-L3

KU snow radar

KU ftp site

Maybe

L0?, L2-3

Flight dynamics

?

?

L0?

Our first guesses, based on limited information

.

13

Schedule

HQ decision on NSIDC Proposal (November?)

scope, level of effort, schedule

NSIDC appoint a project leader and identify team members (January?)

(Proj Mgr, scientist, USO, OPS, Engr)

probably aggregate 1-1.5 fte level of effortImplement approved tasksWork on integrated data products?Conduct Data Assessment (prelim report March 2010?)

Iterate with NASA over requirementsIceBridge Data Management. September 200914

Challenges

Timeliness Of Data Delivery

Understanding And Responding To User Community Needs

Complexity Of Data Streams, Especially Experimental InstrumentationChanging Formats Of Experimental DataDataset Co-registration

Appropriate Data Set And System Integration Develop Tools And Services To Fit Complex Data

OrAdapt Complex Data To Known Tools And ServicesDistributed (Virtual?) Data Management SystemsWho Holds What Data? Who Decides? How Does Interoperability Get Accomplished?Note That Archive Volume Is Not On This List

IceBridge Data Management. September 200915

Principles of Data Management

IceBridge Data Management. September 2009

Preservation without access is pointless; access without preservation is impossible

Keep it simple and flexible: it’s about data, not systems!

Think about long-term archiving when planning for data collection

Document uncertainty

Involve scientists and users in data management, and data managers in science

16

BACKUP

It will use NSIDC’s Searchlight

USERS

JAZ

Struts 2 Action Objects

(controller)

WCS Service

External

Database

Metadata

Database

JPA

Hibernate

ECS

File System

ECHO

Metadata

NSIDC File

System

Service Objects

Repository Objects

Domain Objects

Brokered

Database

Workflow Objects

WMS Service

WFS Service

Analysis Objects

CCAWS UI

G-CAP OGC UI

External

Metadata

SearchLight JSP Pages

(view)

OPeNDAP Server

Other Systems

AJAX Libraries (DOJO, YUI)

SearchLight

CCAWS

G-CAP

User Interface Layer

Service Layer

Domain (model) Layer

Infrastructure Layer

Other Systems

Core Software Architecture

IceBridge Data Management. September 2009

18

Searchlight Components

IceBridge Data Management. September 2009

19

Conceptual Flow

Users: cryo

Ice sheets

Sea Ice

Outlet Glaciers

Snow on Ice

Users: Gravity

Users: Vegetation?

Users: synergistic

Modelers

Sea Level

Instruments

ICESat-1 GLAS

LVIS

UAVSAR

ATM

PARIS

Test instruments

Searchlight

ECHO

Primary Data Route

Metadata Route

Instr. Team to User

ECS

Other

Data

Distribute

To

Cryo.

Community

Platforms

and

Instruments

Processing

By

Instr. team

Data

Management

By

Instr. Team or

Facility

Distribute

To

Other

communities

Synergistic

Tools

Between

Communities

IceBridge Data Management. September 2009

20