Developing Community Guidelines for AI-Ready Data
Description: Developing Community Guidelines for AI-Ready Data NCEI Science Council Meeting, 2022-04-14 Stacie Robinson, Douglas Rao, Rob Redmon, Eric Kihn Motivation Executive Order 2019: Maintaining American Leadership in Artificial Intelligence: ...
Related Topics
Download Presentation
"Developing Community Guidelines for AI-Ready Data" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. Developing Community Guidelines
for AI-Ready Data NCEI Science Council Meeting, 2022-04-14 Stacie Robinson, Douglas Rao, Rob Redmon, Eric Kihn<br>
slide2. Motivation Executive Order 2019: Maintaining American Leadership in Artificial Intelligence:
“... agencies shall improve data and model inventory documentation to enable discovery and usability, and shall prioritize improvements to access and quality of AI data and models based on the AI research community's user feedback.” NOAA’s strategy on AI plans to:
“Define the data quality, documentation, and dissemination requirements for NOAA’s AI-ready open data to ensure consistent standards across federal agencies.” https://sciencecouncil.noaa.gov/NOAA-Science-Technology-Focus-Areas ‹#›<br>
slide3. ‹#› NCAI’s Approach for AI-Ready Data ‹#› How can data users find data that is easy to use in AI/ML?
How can data providers assess and improve usability?
What is needed:
A definition of AI-readiness
Specific, community-defined user guidelines
Assessment tools for data providers
Way to represent readiness level so providers can report data readiness and users can compare
Feedback and iteration to improve the guidelines
Ideally, a formally published standard (or set of standards)
NOAA is leading a collaboration under the Earth Science Information Partners (ESIP) working to develop the community guidelines
US Federal agencies, universities, NGOs, private sector, and international partnership
https://wiki.esipfed.org/Data_Readiness NOAA acoustic data used in deep learning to identify whale songs. https://www.fisheries.noaa.gov/science-blog/ok-google-find-humpback-whales<br>
slide4. Engaging the AI / ML community to define the requirements.
Researching related efforts (e.g. Datasheets for Datasets, Analysis-ready data, Data Stewardship Maturity Matrix)
Promoting community guidelines, assessment tools, and standards NCAI’s Approach for AI-Ready Data ‹#›<br>
slide5. Engaging AI/ML Community ‹#› Targeted Audience: AI practitioners
Objective: information gathering on data factors that enable / prohibit AI development
Expected Outcome: community-driven data requirements to inform guidelines and improvements
End goal: enable efficient AI development with AI-ready data Survey time period: 2021-11 – 2022-01
# of Responses: 104
Government: 40
Academic: 41
Private Sector: 23
Survey results will be published as a public report via ESIP Data Readiness Cluster.<br>
slide6. Key AI-Ready Data Attributes ‹#›<br>
slide7. Researching Related Efforts ‹#› Analysis
Ready
Data Data
Stewardship
Maturity FAIR
Data Principles AI – Ready
Data<br>
slide8. Promoting Standards & Tools ‹#› Assessment Checklist NOW Available!
https://doi.org/10.6084/m9.figshare.19983722.v1
AI-Ready Data Guidelines
End goal is a formal standard
Applications
evaluate key datasets
identify gaps & priorities
improvements over time<br>
slide9. Promoting Standards & Tools User focus groups & reporting Draft community guidelines Pilot readiness assessment Use case driven AI-ready data collection Jan Feb Mar Apr May Jun Jul Aug Sep 2022 AI-ready data for Climate / Blue Economy / Wildfire / … ESIP Meeting
AI-Ready
Data-thon ‹#›<br>
slide10. Opportunities ‹#› Enabling AI Application for Climate:
Developing A Collection of AI-ready Open Climate Data – Data-A-Thon
Join the AI-ready Data-a-Thon July 20 at ESIP
We need your help to pilot AI-ready data assessments
Evaluate your dataset with the AI-ready matrix
Help us improve and finalize the matrix<br>
slide11. Thank you! ESIP Data Readiness Cluster: https://wiki.esipfed.org/Data_Readiness
Rob Redmon (NCAI Lead) – rob.redmon@noaa.gov
Douglas Rao (AI-ready data co-lead) – douglas.rao@noaa.gov
Stacie Robinson (AI-ready data co-lead, detail) – stacie.robinson@noaa.gov<br>
for AI-Ready Data NCEI Science Council Meeting, 2022-04-14 Stacie Robinson, Douglas Rao, Rob Redmon, Eric Kihn<br>
slide2. Motivation Executive Order 2019: Maintaining American Leadership in Artificial Intelligence:
“... agencies shall improve data and model inventory documentation to enable discovery and usability, and shall prioritize improvements to access and quality of AI data and models based on the AI research community's user feedback.” NOAA’s strategy on AI plans to:
“Define the data quality, documentation, and dissemination requirements for NOAA’s AI-ready open data to ensure consistent standards across federal agencies.” https://sciencecouncil.noaa.gov/NOAA-Science-Technology-Focus-Areas ‹#›<br>
slide3. ‹#› NCAI’s Approach for AI-Ready Data ‹#› How can data users find data that is easy to use in AI/ML?
How can data providers assess and improve usability?
What is needed:
A definition of AI-readiness
Specific, community-defined user guidelines
Assessment tools for data providers
Way to represent readiness level so providers can report data readiness and users can compare
Feedback and iteration to improve the guidelines
Ideally, a formally published standard (or set of standards)
NOAA is leading a collaboration under the Earth Science Information Partners (ESIP) working to develop the community guidelines
US Federal agencies, universities, NGOs, private sector, and international partnership
https://wiki.esipfed.org/Data_Readiness NOAA acoustic data used in deep learning to identify whale songs. https://www.fisheries.noaa.gov/science-blog/ok-google-find-humpback-whales<br>
slide4. Engaging the AI / ML community to define the requirements.
Researching related efforts (e.g. Datasheets for Datasets, Analysis-ready data, Data Stewardship Maturity Matrix)
Promoting community guidelines, assessment tools, and standards NCAI’s Approach for AI-Ready Data ‹#›<br>
slide5. Engaging AI/ML Community ‹#› Targeted Audience: AI practitioners
Objective: information gathering on data factors that enable / prohibit AI development
Expected Outcome: community-driven data requirements to inform guidelines and improvements
End goal: enable efficient AI development with AI-ready data Survey time period: 2021-11 – 2022-01
# of Responses: 104
Government: 40
Academic: 41
Private Sector: 23
Survey results will be published as a public report via ESIP Data Readiness Cluster.<br>
slide6. Key AI-Ready Data Attributes ‹#›<br>
slide7. Researching Related Efforts ‹#› Analysis
Ready
Data Data
Stewardship
Maturity FAIR
Data Principles AI – Ready
Data<br>
slide8. Promoting Standards & Tools ‹#› Assessment Checklist NOW Available!
https://doi.org/10.6084/m9.figshare.19983722.v1
AI-Ready Data Guidelines
End goal is a formal standard
Applications
evaluate key datasets
identify gaps & priorities
improvements over time<br>
slide9. Promoting Standards & Tools User focus groups & reporting Draft community guidelines Pilot readiness assessment Use case driven AI-ready data collection Jan Feb Mar Apr May Jun Jul Aug Sep 2022 AI-ready data for Climate / Blue Economy / Wildfire / … ESIP Meeting
AI-Ready
Data-thon ‹#›<br>
slide10. Opportunities ‹#› Enabling AI Application for Climate:
Developing A Collection of AI-ready Open Climate Data – Data-A-Thon
Join the AI-ready Data-a-Thon July 20 at ESIP
We need your help to pilot AI-ready data assessments
Evaluate your dataset with the AI-ready matrix
Help us improve and finalize the matrix<br>
slide11. Thank you! ESIP Data Readiness Cluster: https://wiki.esipfed.org/Data_Readiness
Rob Redmon (NCAI Lead) – rob.redmon@noaa.gov
Douglas Rao (AI-ready data co-lead) – douglas.rao@noaa.gov
Stacie Robinson (AI-ready data co-lead, detail) – stacie.robinson@noaa.gov<br>