A Collaboration between Geoscience Australia CSIRO and NCI CEOS WGISS 40 Simon Oliver Geoscience Australia Robert Woodcock CSIRO CEOS WGISS40 Overview Brief Review Australian Geoscience Data ID: 461670
Download Presentation The PPT/PDF document "Australian Geoscience Data Cube" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Australian Geoscience Data Cube
A Collaboration between Geoscience Australia, CSIRO and NCICEOS WGISS 40Simon Oliver – Geoscience AustraliaRobert Woodcock - CSIRO
CEOS WGISS40Slide2
Overview
Brief Review – Australian Geoscience Data Cube and CEOSUpdate on progress:Open Source collaboration and wikiVersion 1 APIVersion 2 Roadmapmultidimensional storage unitsingest support for multiple sensorsPC, Cloud, & HPC deploymentAnalysis and production pipeline ProvenanceThe Future: Data Cubes and WGISSAnalysis Ready Data as an input to AGDCDiscrete Global Grid Systems - OGC SWG Current Status and implications for AGDC evolutionDiscussion: Sentinel-2 / SPOT-5/6 prototype processing hub to ARD?...link to CWIC system? CEOS WGISS40Slide3
Data-Intensive Quantitative Science
The Australian Geoscience Data Cube (AGDC) :Supports management and quantitative analysis of massive volumes of Earth observation (EO) and other geoscientific data.Bring users to the dataPixels as observationsThe AGDC to as a sensor-independent system for management, analysis and sharing of EO data CEOS WGISS40Slide4
AGDC Overview
A series of data structures and tools to enable efficient analysis of large earth observation archives in HPC environmentsSimple Data StructuresSpatially regular tilesManaged by a relational databaseCalibrated and Standardised Unique ObservationsSurface Reflectance ObservationsQuality Assured ObservationsFlagged for cloud, cloud shadow, saturation and other quality indicatorsOpen source softwareAnalysis Ready Data
CEOS WGISS40Slide5
Landsat processing pipeline
Analysis Ready Data preparation
DataCubeSlide6
Traditional remote sensing product process
Australian Geoscience Data CubeCEOS WGISS40Slide7
New Data Cube remote sensing paradigm
a common analytical framework for High Performance EO data processingsimple data access and analysisrobust processesquality assured unique observationsProcess once-use many times
CEOS WGISS40Slide8
Update on Progress
Update on progress:Open Source collaboration and wikiVersion 1 APIVersion 2 RoadmapCEOS WGISS40Slide9
Current Status
Partners: GA, CSIRO and the NCIInternational collaborators are increasingly involved including the USGS, NASA and CEOS.In the midst of moving to an updated version reflecting recent advances in technologySupporting the establishment of other international data cubes, initially with Kenya and ColombiaAGDC is currently supporting a range of remote sensing applications across the water, vegetation and mineral domainsProviding valuable information for environmental monitoring and modelling across all Australian jurisdictions
CEOS WGISS40Slide10
WOfS Summary Product Example
Sum the derived temporal water stack: number of water observations per pixelSum the derived “real” observations for every pixel from the Pixel QualityProduce the ratio as a percentage for displayWOfS WMS Menindee Lakes as shown in WOfS, with associated legendCEOS WGISS40Slide11
Using tidal models to map tidal extentsCEOS WGISS40
Tidal Zone Extent
Can be attributed with offsets of LAT to lowest observed tide and HAT to highest observed
Tidal Zone Morphology
Fraction of water observations over the time series. Can we attribute this with depths?
Tidal Range of >10m
Slide12
CSIRO Examples
Vicarious calibration sitesIdentify climatic zones, spatial and temporal variation, and seasonal suitability for calibration activitiesLandsat MODIS blendingBlend Landsat and MODIS scenes to produce Landsat-like data (25m resolution) with MODIS repeat cycle (~ every 4-days)Geoglam RangelandsRemote sensing derived information on rangeland and pasture cover and plant available water content for the globeCEOS WGISS40Slide13
Open Data and Code
AGDC Web: http://www.datacube.org.auAGDC Wiki: http://www.datacube.org.au/wiki/Main_PageCode repositories are available through GitHub:https://github.com/data-cubeData is also available as individual files on the NCI THREDDS catalogue:http://dap.nci.org.au/thredds/catalog.htmlCEOS WGISS40Slide14
AGDC v1HPC DeploymentContinental workflow supportCommand line and python interface
CEOS WGISS40Slide15
AGDC v1 - High
Performance ComputingRaijin @ National Computational Infrastructure57,472 cores (2.6 GHz) in 3592 compute nodes;160 TBytes (approx.) of main memory;10 PBytes (approx.) of usable fast file system (for short-term scratch space).And other CSIROsystems viamanaged replicasCEOS WGISS40Slide16
AGDC v1 API ApplicationsBare SoilLandsat Clean PixelLandsat Median MosaicWetness in the landscape – Tasseled Cap Wetness Index
Big Data for Environmental MonitoringCommand line for non-python usersSlide17
The
growth issues will require fundamental changes to how EO research is done, and create massive opportunity
Community
Supply
Distribution
Coordination
National
International
Quality
Calibration
Validation
Provenance
Versioning
Platforms
Data storage
Data management
Analysis
Operations
PC
Cloud
HPC
Earth Observation Informatics Platforms
CEOS WGISS40Slide18
AGDC v2Multidimensional storage unitsIngest support for multiple sensorsPC, Cloud, & HPC deploymentTowards an Observation and Measurement approach to metadata
Analysis and production pipeline ProvenanceUI Reference ImplementationApache v2 LicenseCEOS WGISS40Slide19
The Future: Data Cubes and WGISSThe Future: Data Cubes and WGISSAnalysis Ready Data as an input to Data CubesDiscrete Global Grid Systems - OGC DGGS SWG Current Status and implications for Data Cube
Discussion: Sentinel-2 / SPOT-5/6 prototype processing hub to ARD?...link to CWIC system?CEOS WGISS40Slide20
Analysis Ready DataAnalysis Ready Data (ARD) is satellite data that have been processed and organized so users are not required to invest time and resources in specialized skills to apply corrections for:instrument calibration (gains, offsets);
geolocation (spatial alignment); andradiometry (solar illumination, incidence angle, topography, atmospheric interference).In addition, ARD products are organized in a defined structure with associated metadata, quality flags and products. CEOS WGISS40Slide21
Analysis Ready DataWhere is ARD processing best performed?What is ARD for different satellite/sensor types? e
.g. SARLimitations in ARD, making choices to early?Grids, resampling, resolutions…?Corrections?CEOS WGISS40Slide22
On Grids - A bit about DGGSA DGGS is a spatial reference system that uses a hierarchical tessellation of cells to partition and address the globe.DGGS are characterized by the properties of their cell structure, geo-encoding, quantization strategy and associated mathematical
algorithmCEOS WGISS40Slide23
OGC Discrete Global Grid Systems (DGGS) SWGSept/Oct 2015 – Candidate DGGS Core Standard Released for 30 day public comment periodNov/Dec 2015 – OGC DGGS v1.0 Core Standard adopted by OGC (assuming no major issues raised during public comment period)
Beginning Jan 2016 – OGC DGGS SWG to begin elaboration of Extension Standards to the DGGS Core StandardAnticipated Extensions include:Interoperability interface protocols for OGC Web Services (e.g. WCS, WCPS, WCTiles, etc…) to facilitate DGGS-to-DGGS communication and processing3D (and higher dimensional) DGGS SpecificationsBest Practice GuideCEOS WGISS40Slide24
Future: Possible WGISS involvement
AWS - S3 object storage studyThe Future: Data Cubes and WGISSAnalysis Ready Data as an input to AGDCDiscrete Global Grid Systems - OGC SWG Current Status and implications for AGDC evolutionDiscussion: Sentinel-2 / SPOT-5/6 prototype processing hub to ARD?...link to CWIC system?CEOS WGISS40