Curt Tilmes NASA Version 10 February 2013 Section Local Data Management Copyright 2013 Curt Tilmes Overview Some guidelines for choosing and adopting community accepted standards Background ID: 781347
Download The PPT/PDF document "Data Formats: Choosing and Adopting Com..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Data Formats: Choosing and Adopting Community Accepted Standards
Curt TilmesNASA
Version 1.0February 2013
Section:
Local Data Management
Copyright
2013 Curt Tilmes
Slide2OverviewSome guidelines for choosing and adopting community accepted standards
Slide3BackgroundMost projects (rightly so) focus on the content of their data files, you need to consider the format as well.
Since you captured or created the data, and stored them in your own files, you knowhow the data are organized,how to
read them,how to use them,characteristics of the data that could constrain their use.The goal of a good data format is to make it easier for others to read the data too.Many hours have gone into developing standards for formats – try to learn from them.
Slide4Why use community standards?If you try to develop your data format from scratch, you will
forget something.Build on the experience and improvements built into the community standards over years of use.Tools and analysis software natively support reading community standard data.Reduce development effort and support reuse.Positive feedback – they are more likely to be adopted by others.
Slide5Why use community standards?
http://xkcd.com/927/
Slide6A few guidelinesConsider your archive:
Do they have any recommendations?Consider your users:Who wants this data? Why do they want it?What do they want to do with it?
Will they be using your data in concert with other data?Consider heritage:What worked well for similar data in the past?What could be done better for newly created data?Consider tools:Try to use data formats supported by the software you intend to use it with.
Slide7Some examplesHDF – Hierarchical Data FormatHDF4 and HDF5 versions are in use todayA NASA variant called HDF-EOS is used within the Earth Observing System program.
The Aura project developed a common approach across their instruments and released guidelines as a Technical Note.NetCDF – Network Common Data FormWidely used by agencies including NASA and NOAA Climate and forecast (CF) metadata conventions help standardize some things into NetCDF in a common manner.
Slide8Adopting standardsThe standard gives you a starting point, not a complete solution.
Communicate early with a broad range of data users: archivists, software engineers, scientists.Consider how you will be writing the data and how you will be
reading the data.Get feedback before making final decisions.Start sharing sample data in proposed format to nail down specifics and work out ambiguities.Document your use and application of the standard completely.
Slide9ResourcesHDF:
http://www.hdfgroup.orgHDF-EOS: http://hdfeos.org
HDF-EOS Aura File Format Guidelines: http://disc.sci.gsfc.nasa.gov/Aura/additional/documentation/HDFEOS_Aura_File_Format_Guidelines.pdfhttp://www.esdswg.org/spg/spgfolder/events/esdswg-meeting-october-25-27-2005/auraasabestpracticerev2.pdfNetCDF: http://www.unidata.ucar.edu/software/netcdfCF: http://cf-pcmdi.llnl.gov/
Slide10Other Relevant ModulesLocal Data Management – Data Formats: Using Self-describing Data Formats
Learn more about the advantages of using formats for your data that have important metadata and other information embedded within them
Slide11Recommended Citations
Copyright
2013 Curt Tilmes
.Tilmes, C. 2013. “Local Data Management – Data Formats: Choosing and Adopting Community Accepted Standards.” In Data Management for Scientists Short Course, edited by Ruth Duerr and Nancy J. Hoebelheinrich, Federation of Earth Science Information Partners: ESIP Commons. doi:10.7269/P33N21B6