and LGAC Status 21 February 2018 Chris Engebretson USGS engebretusgsgov Doug Daniels The Aerospace Corporation d ouglasdanielsctrusgsgov Topics Landsat Archive Overview Distribution Statistics ID: 775273
Download Presentation The PPT/PDF document " Landsat Archive, Products, Collection " is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Landsat Archive, Products, Collection and LGAC Status
21 February 2018
Chris Engebretson,
USGS
engebret@usgs.gov
Doug Daniels,
The Aerospace Corporation
d
ouglas.daniels.ctr@usgs.gov
Slide2Topics
Landsat Archive Overview / Distribution Statistics
Landsat Collection 1 Status
TM No-PCD and MSS Status
Landsat Collection 2 Definition / Baseline Contents
Level-2 / Global ARD as Collection 2 Standard Product
TM / ETM+ Shutter Intrusion
Overall Development Schedule
Landsat Global Archive Consolidation (LGAC) Status
Slide3U.S. Landsat Archive Overview(Jan 1, 2018)
OLI-TIRS: Landsat 81,143,018 scenes~ 3,952 TB Raw and L0Ra Data average scene size 1813 MB ETM+: Landsat 72,413,896 scenes~ 2,242 TB Raw and L0Ra Data average scene size 487 MB TM: Landsat 4 & Landsat 52,560,626 scenes ~ 1,329 TB Raw and L0Ra Data average scene size 263 MB MSS: Landsat 1 through 51,312,608 scenes ~ 80 TB Raw and L0Ra Data average scene size 32 MB Total:7,430,148 scenes ~ 7,603 TB Raw and L0Ra Data
Slide4Monthly Downloads
Slide5Landsat Collection 1 Status
Collection 1 processing is ongoing for newly acquired downlinksTM No-PCD / MSS processing has now commenced (more details in next section)Pre-Collection data is (mostly) gonePre-Collection data processing was turned off on April 30th, 2017Pre-Collection search and download was disabled for almost all data on October 1st, 2017The only Pre-Collection data which can still be downloaded is Landsat 1-5 MSS, until Collection 1 MSS processing is fully complete
Slide6Landsat Collection 1 Data Types
Slide7TM No-PCD and MSS Processing – Underway!
Slide8TM No-PCD Products – Release / Processing Status
The Image Assessment System (IAS) and Landsat Product Generation System (LPGS) R13.0.1 releases were installed into the operational environment on February 5
th
, 2018
These releases contain the TM No-PCD and MSS processing capabilities
Priority is being given to the TM No-PCD data
Processing of 263,602 No-PCD scenes commenced on February 7
th
As of Tuesday, February 20
th
, ~185,000 scenes are complete
Projecting completion of No-PCD processing on February 28
th
MSS processing will commence following No-PCD processing
Cal / Val finishing up final Calibration Parameter Files (CPFs) for MSS
Brian M. will have more details in his section
Slide9No-PCD Processing Observations
TM No-PCD data generates about the same percentage of L1TP Tier 1 scenes as TM-R until the cloud cover reaches 40%
For cloud cover > 40%, Tier 1 success rate drops off
Unsurprisingly, TM No-PCD has a larger percentage of Tier 2 scenes than TM-R with nominal PCD
There are fewer L1GS No-PCD products since the majority of this data is located over Europe and Northern Africa
L1GS scenes are typically located over islands or other areas with low coverage of GCPs, or are night scenes
Slide10Product Types of No-PCD Data vs. Nominal
Slide11TM No-PCD Products – EE Search Criteria
Slide12TM No-PCD Products – EE Metadata View
Slide13Landsat Collection 2 Definition / Baseline Contents
Definition and initial development activities are underway
Collection 2 is currently defined as the following high-level scope:
Level-2 / Global ARD as standard product (more detail next section)
Phase 4 Ground Control Point (GCP) enhancements to provide consistent registration base with the Sentinel-2 Global Reference Image (GRI)
Refinements to GCP selection, precision modeling process and standard product parameters to allow more products to process to L1TP
Incremental metadata changes to promote consistency across data types and sensors
Miscellaneous enhancements, bug fixes, report changes
Slide14Landsat Collection 2 Change Requests (1 of 2)
Slide15Landsat Collection 2 Change Requests (2 of 2)
Slide16Level-2 / ARD Scope – Introduction
USGS headquarters has directed EROS to make
Level-2
Analysis Ready Data (ARD) as the new standard Landsat distribution product, replacing
Level-1
This scope has been added to Landsat 9 and is being worked through the existing L9 engineering
process
None of this scope existed at Landsat 9 Ground System SRR
The Landsat 9 DPAS team is in the process of getting the DPAS requirements, operations concept, interfaces, and design documents updated with this scope
Slide17Assumptions – Product Generation
For each WRS-2 scene that the DPAS processes, it will perform the highest-possible level of processing:
Level-2 processing will be performed on all scenes for which atmospheric auxiliary data is available and sun angle constraint is met.
If atmospheric auxiliary data is not available but sun angle constraint is met, Level-1 Top of Atmosphere (TOA) reflectance processing involving per-pixel sun angle correction is performed
If atmospheric auxiliary data is not available and sun angle constraint is not met, then standard Level-1 processing is performed
All scenes from all sensors will continue to have a
realtime
Level 1 product generated to meet needs of low-latency users
Slide18Assumptions – Product Format
DPAS / LPGS will continue to produce data in scene-based format – no “hard” tiling scheme will be implemented
Keeping the data scene-based provides more flexibility on the back end
For example, this could enable the ability to do virtual tiling or implement a user-specified tile / ROI “on the fly” as part of data distribution
Scene-based
product format will continue to be
GeoTIFF
May possibly add cloud optimization options to these
products
This could also be done as a post-processing step
Slide19Assumptions - Miscellaneous
Two new Data Format Control Books (DFCBs) will be created:
LSDS-TBD
Level-2
Data Format Control Book (DFCB)
LSDS-TBD Analysis Ready Data (ARD) Data Format Control Book (DFCB
)
Current U.S. ARD product (Albers projection, WELD tiling scheme) will continue to be processed for Collection 2
U.S. scenes will also be processed to the Global ARD specification
No Level-2 browse products will be generated
Level-1 browse offerings are sufficient for data discovery
Slide20Current Activities – Projection Parameters Pilot
A pilot activity is underway to solicit comments from the user community on the map projection specification for Collection 2 Global ARD
Three options are being evaluated:
UTM
– Scene-based or Military
Grid Reference System tiling
(optional)
Includes double-processing of scenes that straddle UTM zones
Azimuthal Equidistant (Equi7)
Sinusoidal
Provisional products have been generated and
a
writeup
has been provided to the Landsat Science Team (LST)
Following LST review, the provisional products will be made available for public comment
A consensus recommendation from the LST would be great to have (hint, hint)!
Currently have TBDs in our Level-2 requirements for this
TBD will be resolved by our Level 2 CDR in May
Slide21Current Activities – Physical Product Format Pilot
A second pilot is
planned to
determine what the “physical format” – which is to say the final output of the LPGS system – should look like
Current assumption is that we will continue to generate our standard
GeoTIFF
-based product
Physical format pilot could end up recommending something like Cloud-Optimized
GeoTIFF
(COG), which has been getting some traction lately and would enable some more flexible dynamic access to the imagery
Slide22Level-2 / ARD Scope – Overall Schedule
December 2017:
Level-2 ARD Requirements Engineering Peer
Review (COMPLETED)
Review requirements and high-level operations concept for L2 / ARD
February 2018: Landsat 9 DPAS Preliminary Design Review
No design details related to L2 / ARD will be presented at
this
PDR, but requirements and overall status and strategy will be briefed
May 2018: Level-2 ARD Critical Design Engineering Peer Review
Review detailed design deltas to the DPAS element and subsystems related to L2 / ARD scope
additions
Similar in concept to the requirements EPR that was held in December
June 2018: Landsat 9 DPAS Critical Design Review
L2 / ARD scope is all “caught up” to the rest of the DPAS
!
Slide23When the Landsat archive was processed for Collection 1, there were a lot of TM and even ETM+ scenes that had not been processed to Level 1 beforeAn unintended consequence of this was unearthing a fair number of scenes that have IC shutter synchronization anomalies / intrusion artifactsScan Mirror Anomaly (SMA)Late Start Anomaly (LSA)Caterpillar Tracks
TM / ETM+ Shutter Intrusion – Intro
Slide24TM / ETM+ Shutter Intrusion – Examples
Scan Mirror Anomaly
Late Start Anomaly
Caterpillar Tracks
Slide25TM / ETM+ Shutter Intrustion – Caterpillar Tracks Detail
Slide26USGS is proposing a two-phase approach related to this data:Phase I: Scene-based detection of IC shutter intrusion eventsFields exist in the TM / ETM+ PCD to help detect issues with the shutter synchronization, but they are not reliable and not sampled often enough to detect all artifacts even if they wereWe will develop an algorithm to detect the shutter intrusion pattern in the imagery and flag it in the new DATA_ANOMALY metadata fieldImplementation Target: Landsat Collection 2Phase II: Per-pixel flagging of IC shutter intrusions in the QA bandThis will involve a far more detailed algorithm to analyze the contents of each image band and individually flag pixels impacted by intrusion in the scene’s QA bandImplementation Target: Landsat Collection 3
TM / ETM+ Shutter Intrusion –
Plan Forward
Slide27The Level-2 / ARD integration and additional Collection 2 enhancement work are being performed by the Landsat 9 projectGlobal ARD subsystem releases are scheduled to be available to the Landsat 1-8 System Test environment in May 2019Integrated System Test activities will take place in this environmentAn extended period of operational verification will follow (Collection 2 flows will be activated in the ops environment, but not be visible to the public)Projected public availability date for Collection 2: August 2019This date is TENTATIVE and could be impacted by a few things:Availability of Sentinel-2 GRI for GCP harmonizationAddition of new scope to Landsat Collection 2Significant proposed changes to metadata files and / or formats
Overall Development Schedule
Slide28Landsat 9 DPAS Schedule
Slide29Landsat Global Archive Consolidation Status
29
Slide30LGAC Status
For the period July 2017 through February 2018 just over 500,000 TM and ETM+ scenes were processed (~490,000 TM and ~10,000 ETM+)TM Scenes:Network Attached Storage (NAS) drives from European Space Agency (ESA) = ~448,000Between the KIS and FUI stations an additional 260,000 TM (NO-PCD) scenes to process through Collection-1 - anticipate end of March 2018 for completion of the NO-PCD dataRemaining TM scenes recovered from High Density Tapes (HDT) and Digital Linear Tape (DLT) from Saudi Arabia (RSA)1,773 HDT tapes have been recovered during this period in ~53,418 scenes being recovered or which 29,767 (55.7%) are new and unique to archive98% success rate with RSA HDT tapes (averaging approximately 250 tapes per month)ETM+ Scenes:10,000 ETM+ scenes recovered from 318 DLT media from China682 DLT media remains to work
“Woodcock Metric”: 4,959,866 scenes
(February 2018)
Slide31LGAC Status
Argentina – LTOsTM and ETM+ data delivery continuesAustralia – Electronic data delivery complete~40,000 additional unique TM scenes were identified and delivered to USGSBrazil – High Density Digital Tapes (HDDTs)USGS to investigating setting up Wideband Video Drive to read tapes (capability to read tapes not available)~875 tapes to be sent by Brazil upon sample tape successPrimarily consist of MSS data, with very small number of TM intervals also included
Country (Organization)
Ground Station
%
LGAC Delivered
% LGAC Ingested
Argentina (CONAE)
COA
TM
TM
ETM+
ETM+
Australia (GA-NEO)
ASA
MSS
MSS
TM
TM
ETM+
ETM+
Australia (GA-NEO)
HOA
TM
TM
ETM+
ETM+
Brazil (INPE)
CUB
MSS
MSS
TM
TM
ETM+
ETM+
Canada (CCMEO)
GNC
MSS
MSS
TM
TM
ETM+
ETM+
Canada (CCMEO)
PAC
MSS
MSS
TM
TM
ETM+
ETM+
Slide32LGAC Status
Country (Organization)Ground Station% LGAC Delivered% LGAC IngestedChina (RADI)BJCTMTMETM+ETM+China (RADI)KHCTMTMEcuador (IEE)CPETMTMEurope (ESA)FUIMSSMSSTMTMETM+ETM+Europe (ESA)KISMSSMSSTMTMETM+ETM+Europe (ESA)MTITMTMETM+ETM+Europe (ESA)MPSMSSMSSTMTMETM+ETM+Europe (ESA)NSGETM+ETM+
China
– Electronic data delivery
Currently
reading DLTs containing ETM+ data;
RADI
identifying if any outstanding TM data
available
Processing completion anticipated mid-2018
Ecuador
– Hard Drives (HDs)
IEE transcription process halted due to equipment challenges
Europe
– NAS HDs
MSS
data delivery pending
Separately working with DLR to obtain 568 DLTs containing all of NSG's historical L7 data
Slide33LGAC Status
India – All data has been receivedMSS data was only available in partially processed Level 1 formatUSGS to assess options for data archival and distributionNo timeline for completionIndonesia – All data has been receivedDCRSi drive parts needed for remaining tapesNo timeline for completionPakistan – Linear Tape-open (LTOs)Delivery of additional TM data on LTOs pendingNo timeline for completion
Country (Organization)
Ground Station
%
LGAC Delivered
% LGAC Ingested
India (ISRO)
SGI
MSS
MSS
TM
TM
ETM+
ETM+
Indonesia (LAPAN)
DKI
TM
TM
ETM+
ETM+
Japan (HIT/HEEIC)
HIJ
ETM+
ETM+
Japan (JAXA/RESTEC)
HAJ
MSS
MSS
TM
TM
ETM+
ETM+
Kyrgyzstan (DLR)
BIK
TM
TM
Mongolia
(DLR)
ULM
TM
TM
Pakistan (SUPARCO)
ISP
MSS
MSS
TM
TM
Russia (ScanEx)
IKR
TM
TM
Russia (ScanEx)
MGR
TM
TM
Russia (ScanEx)
MOR
TM
TM
Slide34LGAC Status
Saudi Arabia – All HDDTs and Digital Linear Tapes (DLTs) have been delivered HDDT tape baking, cleaning, reading, and ingest currently in progressDLT ingest pending software updatesData from multiple sensors identified on mediaApproval has been received for reading the non-Landsat data South Africa – Electronic data deliveryDelivery of MSS data pendingThailand – All data has been deliveredTM HDDTs completedMSS HDDT tape baking, cleaning, reading, and ingest currently on holdNo timeline for completion
Country (Organization)
Ground Station
%
LGAC Delivered
% LGAC Ingested
Saudi
Arabia (KACST)
RSA
MSS
MSS
TM
TM
South Africa (SANSA)
JSA
MSS
MSS
TM
TM
ETM+
ETM+
Taiwan (CSRSR-NCU)
CLT
TM
TM
Thailand (GISTDA)
BKT
MSS
MSS
TM
TM
ETM+
ETM+
US (U
of
Puerto
Rico)
UPR
ETM+
ETM+
Slide35LGAC Notional Timeline – TM
Slide36Questions?