OOI-CI Data Team Tasks 07 May 2015 - PowerPoint Presentation

1. OOI-CI Data Team Tasks07 May 2015

2. EPE Construction through August 31Core EPE software and integration will be complete on June 30 (Data Investigation Builder, Concept Mapping, Vocab Navigator, Custom Visualization tools)EPE expects to host a conditional acceptance of software in late JuneEPE will need Raytheon Omaha and ASA to complete web services and asset management construction before several EPE visualization tools can be completed and integrated with uFrame. Visualization tools that will be completed include single time series, single time series with map, double time series, glider profiler explorer, glider profiler comparer and the profiler explorer/comparerRemaining funds of $XX,XXX will be used by Sage Lichtenwalner and John Kerfoot to complete the EV tool construction/integration (X hours), and for the University of Maine to test and document the tools (X hours).1

3. Science Program ManagerCoordinating management of the entire OOI CI project with the System Engineering PMManaging the Data Management team and the User Support and Outreach positionEnsure continuous quality data delivery from the entire OOIStrategic planning, scheduling and budgetary responsibilityWorking with the Marine IOs, Science Advisory Teams and COL on data oversightManaging subcontractors (e.g. ASA)Authoring quarterly reports2

4. User Support and OutreachServe as the primary point of contact for software inquiries and general OOI questions from end usersincluding researchers, scientists, marine operators, educators and studentsProvide technical support on the software and user interfaceCoordinate with the Data Manager to identify and resolve user issuesWork with software developers to resolve user issues with the system and user interfaceDevelop training and outreach materials Oversee the organization of four technical training and outreach programs per year to support observatory users, early career scientists, students and educators 3

5. Help Desk Process4help@oceanobservatories.org (planned)Redmine used to track trouble tickets. Help desk is responsible for triaging initial inquiries and ticket closing.Help desk will be covered by DM and PM in Help Desk absence.Policy & ProposalsMike KellyExternal Users QuestionsInstrument AdditionsMike KellyMarine HWScience PM & Marine IOsDataMike VardaroSoftware AssistanceSage LichtenwalnerSystem & ITIvan RoderoHelp DeskSage LichtenwalnerGeneral OOI InformationSage Lichtenwalner

6. OOI FAQsThe existing OOI FAQ page will be significantly updated this month and include the following four categories:Background: Questions on the general OOI, components, science themes, partnerships, funding and managementOOI Related Proposals: General discussion on the NSF plan for acceptance of proposals and future announcement.Access to OOI Data: Questions on who has access, data policy, ownership, new sensors, publishingOOI Data Content and QC: Questions regarding responsibilities, sampling strategies, calibration, shipboard data, QA/QC process and algorithms5

7. Data Manager TasksManages (and assist in hiring of) team of four data evaluators, directing data evaluation efforts Provides technical expertise for evaluations of data and OOI GUI capabilities Works with domain scientists and Marine IOs to translate requirements into QA/QC, validation, and sampling strategy best practicesWork with the CI systems and software team on the design, operation, and optimization of the data management infrastructureResponsible for managing the data life cycle and archiving methodsCommunicate OOI data management strategies to community and collect feedback for incorporation into OOI processes and procedures6

8. Data Evaluator TasksEvaluate OOI data daily and report evaluations weeklyDevelop appropriate scripts and plotting tools to analyze streaming & telemetered dataMaintain database of evaluated data Outward facing towards Global, Regional, Endurance, and Pioneer ArraysRemain up to date on data processing tools, computer languages, and instrumentation changesAssist with writing of technical reports and preparation of conference presentations 7

9. Rutgers Data Evaluation TeamScience Leadership Team:Scott Glenn Physics, Model-Observation Fusion, Fluid DynamicsOscar Schofield Biology, Optics, Genetics, EcologyProgram Manager:Mike Crowley (M.S. 1993)Physics, coastal oceanography, remote sensing, Gulf StreamData Manager: Mike Vardaro (M.S. 2004, Ph.D. 2008)Marine Biologist, OOI Project Scientist, expertise in benthic biogeochemistry, photography, carbon fluxAssistant Data Manager: Leila Belabbassi (M.S. 2001, Ph.D. 2006)Physical expertise, experience in water column dynamics, acoustics, oxygen/hypoxiaData Evaluators:Lori Garzio (M.S., 2012)Biological expertise, focus on plankton biology, high-latitude dynamics, bioacousticsFriedrich Knuth (M.S., 2014) – June 8 start dateGeology expertise, focus on geomorphology, geochemistry, data visualization, acoustic surveyingMike Smith (B.S. 2009)Physical expertise, focus on atmospheric and coastal physics and circulation, some biology8

10. Responsibility Matrix9

11. OOI Science Prospectus10Instruments

12. Data Evaluation Schedule - ConstructionDaily:9:00am – Daily integration callDaily team meeting to discuss progress, issues, and tasks for the dayEvaluate data (see Evaluation Procedures slide)Check in and re-assess statusWeekly:Meet to conduct team evaluation of data plots, write up summary report11

13. OOI Data Reviewed to DatePioneer (13 out of 24 instrument classes):Gliders (5): multiple deployments, telemetered and recovered – CTD, Fluorometer, PAR, O2, ADCP (partial), EngineeringProfilers (5): multiple deployments, telemetered and recovered – CTD, Fluorometer, PAR, 3D Point Velocity, ADCP (partial), EngineeringSurface Moorings (2): one deployment, telemetered – Seafloor pressure, CTD (salinity only), Point Velocity (incorrect values), Air-Sea pCO2 (partial pressure only)Global (8 out of 21 instrument classes):Station Papa:Gliders (3): telemetered and recovered - CTD, O2Surface Moorings (2): DO, Fluorometer, pHIrminger:Surface Moorings (1): CTD, DO, Fluorometer, pH, Point Velocity, ADCP12NOTESReviewed does not mean approvedNov. – Jan. (Drops 1-2) evaluations focused on every instrument on every platform. Drops 3-5 focus on a single example of each to prove that drivers, calibration tables, ingest, L1 & L2 processing all produce scientifically reasonable output

14. OOI Data Reviewed to Date, cont.Endurance (16 out of 25 instrument classes): Gliders (8): multiple deployments, telemetered and recovered – CTD (no density), Fluorometer, PAR, O2, ADCP (partial), EngineeringWire-Following Profilers (1): CTD only (no density), EngineeringSurface-Piercing Profilers (1): CTD, Nitrate, Optical Attenuation/Absorption (bad data)Surface Moorings (1): ADCP, pH, Spectral Irradiance, Point Velocity, DO, Fluorometer, Surface WavesCabled Array (20 out of 34 instrument classes): Group 1B (7): seafloor test data – CTD (pumped and profiling), O2, Fluorometer, PAR, Temperature-Resistivity (broken instrument), Point Velocity, ADCPGroup 2A (3): seafloor test data – CTD, O2, Nitrate (bad data), Temperature-Resistivity (broken instrument)Group 3 (5): seafloor test data – CTD, O2, Seafloor pressure, Spectral Irradiance, 3D Thermistor ArrayGroup 4 (3): seafloor test data – Seawater pCO2, pH, 3D point velocity, VADCP (5-beam), D1000 (thermistors), Bottom Pressure-Tilt13NOTESReviewed does not mean approvedNov. – Jan. (Drops 1-2) evaluations focused on every instrument on every platform. Drops 3-5 focus on a single example of each to prove that drivers, calibration tables, ingest, L1 & L2 processing all produce scientifically reasonable output

15. Data Evaluation Schedule – O&MDaily:Investigate overnight issuesDaily team meeting to discuss progress, identify QC alerts and alarms, assign tasks for the daySplit into small groups to evaluate and conduct QC checks on data (see Evaluation Procedures)Check in to re-assess statusContinue QC workWeekly:Evaluator contact with designated IO repsGroup evaluation of data plots and science discussionBi-Weekly/Monthly:Draft group report on data quality and metrics14

16. Current Data Evaluation Procedures & ToolsScript scrapes uFrame server for active links, creates list of available dataPython script downloads NetCDF files of available data (as long of a time period as possible, ideally 1 week to 1 month)Plot all data, 1 week, 1 day, and 1 hour for each data product using PythonSpecialized programs (such as Panoply) used to create plots for instruments like ADCPs with binned or other non-standard data formatsFinal Python tool imports plots into a Word documentData team reviews for completeness, data gaps, unit errors, out of order time stamps, scientific validity, etc.Team annotates plots using highlight tool, text, Google documentGoogle document summary combined with plots to create final data evaluation report document, submitted to Integration Team LeadData issues entered into Redmine as trouble tickets by Data Manager15

17. O&M Data Evaluation Procedures & ToolsCheck GUI for active platforms, list of available data, QC alertsPython script downloads NetCDF files of available data (as long of a time period as possible, ideally 1 week to 1 month) or uses GUI to create plotsPlot all data, 1 week, 1 day, and 1 hour for targeted data productsData team conducts quick reviews for completeness, data gaps, unit errors, out of order time stamps, scientific validity, etc.Consult Science Oversight Committee and outside experts for deep divesTeam annotates plots and datasets using GUI annotation and event toolData issues entered into Redmine as trouble tickets by Evaluators and Data Manager; QA issues referred to IO’s via SOCMonthly data evaluation report document (including metrics) submitted to CI Project Manager, Ocean Leadership, and NSF16