How does data get from the source to the end user - PowerPoint Presentation

Slide1

How does data get from the source to the end user? Old School vs. New School

Bob Simons

DOC / NOAA / NMFS / SWFSC / ERD

Monterey, CA

bob.simons@noaa.govSlide2

All Data SourcesSlide3

All Users (Diverse)Slide4

From the Server to the UserServer

Sensor

User's

Computer

ArcGIS, R, Matlab, IDL, IDV,

Igor, Ocean Data Viewer, etc.

Dataset specific

?Slide5

Data vs. VisualizationsData

Visualizations

2 GB

21 MB

21 MBSlide6

The Goal: Connect Data and Users

All Data Sources

Diverse UsersSlide7

The Real Picture: Connecting Data and Users

NOAA ...

Scientists

NASA ...

USGS ...

UCAR...

OOI, EC, D1, ...

1000's more

Others...

Surfers

Students

Farmers

FishermanSlide8

Public Access to Research Results (PARR) RequirementsGovernment funded data shall be

publicly

and

freely:

Discoverable via a catalog (data.noaa.gov -> data.gov)Understandable via metadata

Accessible via a web service (e.g., ERDDAP / DAP)(not just downloadable files or a shopping cart)Your

data available within one year (by last year ASAP!)ERDDAP can help with all of these requirements!

https://www.whitehouse.gov/blog/2013/02/22/expanding-public-access-results-federally-funded-researchhttps://www.whitehouse.gov/sites/default/files/microsites/ostp/ostp_public_access_memo_2013.pdfSlide9

Data Distribution Issues

Aggregation

Understanding

Discovery

Requesting dataFile TypesSlide10

Issue #1:Aggregation

Old school: No Aggregation.

1000's of files

(one for each station or for each time point)

1000's of entries in a catalog, often out of order. 1000's of results from Google.

New school: Aggregation1 dataset (with multiple stations, or a time dimension)

1 entry in a catalog1 result from GoogleData servers (THREDDS and ERDDAP) can offer both.

Reusable, free, open-source software. Easy for providers.Slide11

Issue #2: UnderstandingMetadata -

who, what, when, where, why, how

Old school

- minimal, insufficient.

New school - complete metadata, helps Google/catalogsData servers (ERDDAP) make it easy to add/modify metadata.Slide12

Issue #3:Discovery

So many datasets from so many data providers

Old school: Search engines, e.g., Google

New school: Catalogs

and catalogs in the data servers (e.g., ERDDAP)Better

dataset metadata

Better info for search engines

ncISO

ISO 19115 for catalogs

Helpful for UsersSlide13

Issue #4:Requesting Data

Old school -

Files (okay).

Shopping carts (bad - not immediate, not scriptable).New school - Data servers with web services

(aggregated; the user can specify a subset and file type)Slide14

Issue #5:File Types

Old school: one file type

(probably not what you want)

New school:

The data server (eg ERDDAP) offers web services (OPeNDAP, WMS) and generates files on-the-fly, including

web services (DAP, WMS), metadata files (.das, FGDC, ISO 19115), data files (.nc, .csv, ) and image files (.png, .pdf).Slide15

Old School vs. New School

Old school

- some parts still good/useful

(Google, single files)

New school - Is driven by PARR and is helpful

for users.Is made easier for data providers by reusable, free, open-source data servers (ERDDAP and THREDDS).Slide16

My rant:Why is it taking so longto switch to web services?

We had the tools we needed in 2012.

We've had PARR for 2 years.Slide17

My Request

Data providers

:

Please make your datasets

available via ERDDAP and THREDDS.PARR Compliance: Aggregated datasets, Better metadata,ISO 19115 metadata for catalogs, Web services

Happy usersUsers: Please ask data providers to make datasets available via ERDDAP and THREDDS.Data access will be easier for you.Slide18

Thank you!Questions? Comments? Suggestions? email bob.simons@noaa.gov

Give ERDDAP a try!

http://coastwatch.pfeg.noaa.gov/erddap

/