From Innovation to Impact: Research and Development in Data - PowerPoint Presentation

Slide1

From Innovation to Impact: Research and Development in Data Intensive High Performance Computing

Indiana University9/14/2011Scott A. Klaskyklasky@ornl.gov

Special Thanks to:

H. Abbasi

2, Q. Liu2, J. Logan1, M. Parashar6, N. Podhorszki1, K. Schwan4,A. Shoshani3, M. Wolf4, S. Ahern1, I.Altintas9, W.Bethel3, V. Bhat6, S. Canon3,L. Chacon1, CS Chang10, J. Chen5, H. Childs3, T. Critchlow16, J. Cummings13,C. Docan6, G. Eisenhauer4, S. Ethier10, B. Geveci8, G. Gibson17, R. Grout7, R. Kendall1,J. Kim3, T. Kordenbrock5, Z. Lin10, J. Lofstead5, B. Ludaescher18, X. Ma15,K. Moreland5, R. Oldfield5, V. Pascucci12, M. Polte17, J. Saltz18, N. Samatova15,W. Schroeder8, R. Tchoua1, Y. Tian14, R. Vatsavai1, M. Vouk15, K. Wu3, W. Yu14,F. Zhang6, F. Zheng41ORNL, 2 U.T. Knoxville, 3LBNL, 4Georgia Tech, 5Sandia Labs, 6 Rutgers, 7NREL, 8Kitware, 9UCSD, 10PPPL, 11UC Irvine, 12U. Utah, 13 Caltech, 14Auburn University, 15NCSU, 16PNNL, 17CMU, 18EmorySlide2

OutlineThe ADIOS storyStep 1: Identification of the problem

Step 2: Understand the application use casesStep 3: Investigate related researchStep 4: Solve the problemStep 5: InnovateStep 6: ImpactStep 7: FutureWork supported under:ASCR : CPES, SDM, Runtime Staging, SAP, OLCF, Co-designOFES : GPSC, GSEPNSF : EAGER, RDAVNASA : ROSESSlide3

We want our system to be so easy, even a chimp can use it

Even I can use it!SustainableFastScalablePortableSlide4

The ADIOS storyInitial partnership

with GTC (PPPL)Study the codes I/OPosix , netcdf, HDF5 usedRequirement: visually monitoringData Streaming APIs createdCatamount forces to use MPI-IO for checkpoint-restarts (SDM team, but 30% overhead.Assemble a team (ORNL, Georgia Tech, Rutgers): Create a new I/O abstractionNew file format introduced to reduce the I/O impact + increase resiliencyIntroduce SOA to answer a new challenges from the CPES teamNew questions arise: reduce I/O variability, (Sandia)Faster reads needed, (Auburn)S3D team wants the reads even faster!

Staging techniques introduced for reading, NREL.Slide5

The ADIOS storyCouple

multi-scale, multi-physics codes as separate executables, (NYU, CalTech)Data reduction necessary, NCSUIndex the data, FastBit LBNLOptimize the I/O for vis., (Kitware, Sandia, LBNL)Introduce workflow automation for in situ processing, SDSCIBM BGP requirements NCSUIntegrate U.Q. into ADIOS staging for co-design, U. TexasAbstract data monitoring with I/O abstraction, ORNLImage processing, spatio-temporal queries (ORNL, Emory)Cloud analysis requirements, (Wisconsin )Slide6

Identify the problemStep

1Slide7

Extreme scale computing

TrendsMore FLOPSLimited number of users at the extreme scaleProblemsPerformanceResiliencyDebuggingGetting Science doneProblems will get worseNeed a “revolutionary” way to store, access, debug to get the science done!ASCI purple (49 TB/140 GB/s) – JaguarPF (300 TB/200 GB/s)

From J. Dongarra

, “Impact of Architecture and Technology for Extreme Scale on Software and Algorithm Design,” Cross-cutting Technologies for Computing at the Exascale, February 2-5, 2010.

Most people get < 5 GB/s at scaleSlide8

I/O challenges for the extreme scaleApplication ComplexityAs the complexity of physics increases, data grows

Checkpoint-restart (write)Analysis and visualization (read/write)Coupled application formulations (read/write)System/Hardware ConstraintsOur systems are growing by 2x FLOPS/yearDisk Bandwidth is growing ~20%/yearAs the systems grow, the MTTF fallsFurther stresses due to shared file systemsNeed new and innovative approaches to cope with these problem!JaguarPF224 K

sith

1K

lens512MDS1 nodeSANSlide9

Next generation I/O and file system challengesAt the architecture

or node levelUse increasingly deep memory hierarchies coupled with new memory propertiesAt the system levelCope with I/O rates and volumes that stress the interconnect and can severely limit application performance Can consume unsustainable levels of powerAt the exascaleImmense aggregate I/O needs with potentially uneven loads placed on underlying resourceCan result in data hotspots, interconnect congestion and similar issuesSlide10

Understand the application use casesStep

2Slide11

Our team works directly with Fusion: XGC1, GTC, GTS, GTC-P, Pixie3D, M3D, GEMCombustion: S3D

Relativity: PAMRNuclear Physics: Nuclear PhysicsAstrophysics: Chimera, Flashmany more application teamsRequirements (in order)Fast I/O with low overheadSimplicity in using I/OSelf Describing file formatsQoSExtras, if possibleAbility to visualize data from simulationCouple multiple codesPerform in situ workflows

Applications are the main driversSlide12

Investigate related researchStep

3Slide13

Parallel netCDFNew file format to allow for large array supportNew optimizations for non-blocking callsIdea is to allow

netcdf to work in parallel and for large files and large arrays2011 paper: A Case Study for Scientic I/O: Improving the FLASH Astrophysics Code http://www.mcs.anl.gov/uploads/cels/papers/P1819.pdf

5% peak

0.8% peak Slide14

Parallel netCDF-4/ HDF5Use HDF5 for the file formatKeep backward compatibility in tools to read netCDF 3 filesHDF5 optimized chunking

New journaling techniques to handle resiliencyMany other optimizations5 GB/shttp://www.hdfgroup.org/pubs/papers/howison_hdf5_lustre_iasds2010.pdfVorpal, 40^3: 1.4 MB/procSlide15

Solve the problem

Step 4Slide16

The “early days”: 2001, Reduce I/O overhead for 1 TB data from the GTC code and “real-time” monitoringS. Klasky

, S. Ethier, Z. Lin, K. Martins, D. McCune, R. Samtaney, “Grid -Based Parallel Data Streaming implemented for the Gyrokinetic Toroidal Code,” SC 2003 Conference. V. Bhat, S. Klasky, S. Atchley, M. Beck, D. McCune, and M. Parashar, “High Performance Threaded Data Streaming for Large Scale Simulations,” 5th IEEE/ACM International Workshop on Grid Computing (Grid 2004)Key IDEAS:Focus on I/O and WAN for an applicationdriven approachBuffer Data, and combine all I/O requestsfrom all variables into 1 write call

Thread the I/OWrite data out on

the receiving side

Visualize the data near-real-timeFocus on the 5% ruleSlide17

Adaptable I/O System

Provides portable, fast, scalable, easy-to-use,metadata rich output with a simple APIChange I/O method by changing XML input fileLayered software architecture:Allows plug-ins for different I/O implementationsAbstracts the API from the method used for I/OOpen source:http://www.nccs.gov/user-support/center-projects/adios/High Writing PerformanceS3D: 32 GB/s with 96K cores, 1.9MB/core: 0.6% I/O overhead with ADIOSXGC1 code  40 GB/s, SCEC code 30 GB/sGTC code

 40 GB/s, GTS

code: 35 GB/sSlide18

Performance Optimizations for WriteNew technologies are usually constrained by the

lack of usability in extracting performanceNext generation I/O frameworks must addressthis concernPartitioning the task of optimizationsfrom the actual description of the I/OInnovate to deal with high I/O variability, and increase the “average” I/O performancestd dev. timeSlide19

Understand common read access patterns

Restarts: arbitrary number of processors reading

All of a few variables on a arbitrary number of processors.

All of 1 variable

Full PlanesSub PlanesSub VolumeConclusion: File formats that chunk the data routinely achieved betterread performance than logically contiguous file formatsSlide20

Understand why: (Examine reading a 2D plane from a 3D dataset)Use Hilbert curve to place chunks on lustre file system with an Elastic Data Organization

Theoretical concurrencyObserved PerformanceSlide21

InnovateStep

5Slide22

SOA philosophyThe overarching design philosophy of our framework is based on the

Service-Oriented Architecture Used to deal with system/application complexity, rapidly changing requirements, evolving target platforms, and diverse teamsApplications constructed by assembling services based on a universal view of their functionality using a well-defined APIService implementations can be changed easilyIntegrated simulation can be assembled using these servicesManage complexity while maintaining performance/scalabilityComplexity from the problem (complex physics)Complexity from the codes and how they are Complexity of underlying disruptive infrastructureComplexity from coordination across codes and research teamsSlide23

SOA Scales (Yahoo data challenges)Data diversityRich set of processing – not just database queries (SQL), but analytics (transformation, aggregation, …)

Scalable solutionsLeverage file system’s high bandwidth Multiple ways to represent dataDeal with reliability Make it easy to use: self-management, self-tuning Make it easy to change: adaptabilityThey do this for $$$$$$If Yahoo and Google can do it, so can we!24PB/day and growingSlide24

Designing the system

Why staging?Decouple file system performance variations and limitations from application run timeEnables optimizations based on dynamic number of writersHigh bandwidth data extraction from application: RDMA

Scalable data movement with shared resources requires us to

manage the transfersSlide25

Our discovery of data scheduling for asynchronous I/O

Scheduling techniqueNaïve scheduling (Continuous Drain), can be slower than synchronous I/OSmart scheduling can be

scalable and much better than synchronous I/OSlide26

Monolithic staging applications

Multiple pipelines are separate staging areasData movement is between each staging codeHigh level of fault toleranceStaging Revolution: Phase 1Application

Staging Process 1

Staging Process 2

StorageStorageStaging AreaStaging AreaSlide27

A

pproach allows for in-memory code couplingSemantically-specialized virtual shared spaceConstructed on-the-fly on the cloud of staging nodesIndexes data for quick access and retrievalComplements existing interaction/coordination mechanisms In-memory code coupling becomes part of the I/O pipelineSlide28

Comparison of performance against MCT

M=512 , N=2048 M=1024 , N=4096 Slide29

Phase 2: Introducing Plugins

Disruptive stepStaging codes broken down into multiple pluginsStaging area is launched as a framework to launch pluginsData movement occurs between all pluginsScheduler decides resource allocationsApplication

Plugin A

Plugin

BPlugin EPlugin CPlugin DStaging AreaSlide30

Resources are co-allocated

to plugins Plugins are executed consecutively, co-located plugins for no data movementScheduler attempts to minimize data movementPhase 3: Managing the staging areaApplication

Plugin A

Plugin

BPlugin EPlugin CPlugin DManaged Staging AreaCo-located pluginsSlide31

Hybrid staging

Plugins may be executed in the application spaceScheduler decides which plugin is migrated upstream to the applicationMaintain resiliency of the staging architectureApplication

Plugin A

Plugin

BPlugin EPlugin CPlugin DManaged Staging AreaPhase 4: Hybrid StagingApplication partition

Plugins executed within the application nodeSlide32

EnStage

Runtime placement decisionsDynamic code generationFilter specialization Move work to data (I/II)Staging functions are implemented in C-on-Demand (CoD)

CoD can be transparently moved from staging nodes to application nodes or vice versaSlide33

Move work to data (II/II)

ActiveSpaces – Dynamic binary code deployment and executionProgramming support for defining custom data kernels using native programming language Operates only on data of interestRuntime system for binary code transfer and execution in the staging areaSlide34

Data Management (Data Reduction/in situ)

ImpactObjectives Develop a lossy compression methodology to reduce “incompressible” spatio-temporal double precision scientific data

Ensure high compression rates with ISABELA, while adhering to user-controlled accuracy bounds

Provide

fast and accurate approximate query processing technique directly over ISABELA-compressed scientific dataUp to 85% reduction in storage on data from petascale simulation applications: S3D, GTS, XGC, …Guaranteed bounds in per-point error, and overall 0.99 correlation, and 0.01 NRMSE with original dataLow storage requirementCommunication-free, minimal overhead technique easily pluggable into any simulation applicationISABELA Compression Accuracy at 75% reduction in sizeAnalysis of XGC1 Temperature DataSlide35

Impact for the futureData movement is a deterrent to effective use of the dataThe output costs increase the runtime

and force the user to reduce the output dataInput costs can make up to 90% of the total running time for scientific workflowsOur goal is to create an infrastructure to make it easy for you to forget the technology and focus on your science!Slide36

eSiMon collaborative monitoringCritical for multiple scientists to monitor their simulation when they run at scale

These scientists generally look for different pieces of the output and require different levels of simplicity in the technologyProvenance used for post-processingSAVE $$’s and timeSlide37

From research to innovation to impact principlesUnderstand the important problems in the field

Partner with the “best” researchers in the fieldForm teams based on each members ability to deliverAvoid overlap as much as possibleUnderstand the field, and the GAPS as we move forward in terms of applications involvement, technology changes, and software issues, and grow the softwareFurther develop the software so that it can be sustainableSlide38

ImpactStep

6Slide39

Community highlightsFusion2006

Battelle Annual report, cover image http://www.battelle.org/annualreports/ar2006/default.htm2/16/2007 Supercomputing Keys Fusion Researchhttp://www.hpcwire.com/hpcwire/2007-02-16/supercomputing_keys_fusion_research-1.html7/14/2008 Researchers Conduct Breakthrough Fusion Simulationhttp://www.hpcwire.com/hpcwire/2008-07-14/researchers_conduct_breakthrough_fusion_simulation.html8/27/2009 Fusion Gets Fasterhttp://www.hpcwire.com/hpcwire/2009-07-27/fusion_gets_faster.html4/21/2011 Jaguar Supercomputing Harnesses Heat for Fusion Energyhttp://www.hpcwire.com/hpcwire/2011-04-19/jaguar_supercomputer_harnesses_heat_for_fusion_energy.htmlCombustion9/29/2009 ADIOS Ignites Combustion Simulationshttp://www.hpcwire.com/hpcwire/2009-10-29/adios_ignites_combustion_simulations.htmlComputer science2/12/2009 Breaking the bottleneck in computer data flowhttp://ascr-discovery.science.doe.gov/bigiron/io3.shtml9/12/2010 Say Hello to the NEW ADIOShttp://www.hpcwire.com/hpcwire/2010-09-12/say_hello_to_the_new_adios.html2/28/2011 OLCF, Partners Release eSiMon Dashboard Simulation Toolhttp://www.hpcwire.com/hpcwire/2011-02-28/olcf_partners_release_esimon_dashboard_simulation_tool.html

7/21/2011 ORNL ADIOS Team Releases Version 1.3 of Adaptive Input/Output System

http://www.hpcwire.com/hpcwire/2011-07-21/ornl_adios_team_releases_version_1.3_of_adaptive_input_output_system.htmlSlide40

2008 – 2011 I/O Pipeline Publications

SOAS. Klasky, et al., “In Situ data processing for extreme-scale computing”, appear SciDAC 2011.C. Docan, J. Cummings, S. Klasky, M. Parashar, “Moving the Code to the Data – Dynamic Code Deployment using ActiveSpaces”, IPDPS 2011.F. Zhang, C. Docan, M. Parashar, S. Klasky, “Enabling Multi-Physics Coupled Simulations within the PGAS Programming Framework”, CCgrid 2011.C. Docan, S. Klasky, M. Parashar

, “DataSpaces: An Interaction and Coordination Framework for Coupled Simulation Workflows”, HPDC’10. ACM, Chicago Ill.

Cummings, Klasky, et al., “EFFIS: and End-to-end Framework for Fusion Integrated Simulation”, PDP 2010, http://www.pdp2010.org/.

C. Docan, J. Cummings, S. Klasky, M. Parashar, N. Podhorszki, F. Zhang, “Experiments with Memory-to-Memory Coupling for End-to-End fusion Simulation Workflows”, ccGrid2010, IEEE Computer Society Press 2010.N. Podhorszki, S. Klasky,  et al.: Plasma fusion code coupling using scalable I/O services and scientific workflows. SC-WORKS 2009C. Docan, M. Parashar and S. Klasky. “DataSpaces: An Interaction and Coordination Framework for Coupled Simulation Workflows“. Journal of Cluster Computing, 2011Data FormatsY. Tian, S. Klasky, H. Abbasi, J. Lofstead, R. Grout, N. Podhorszki, Q. Liu, Y. Wang, W. Yu, “EDO: Improving Read Performance for Scientific Applications Through Elastic Data Organization”, to appear Cluster 2011.Y. Tian, “SRC: Enabling Petascale Data Analysis for Scientific Applications Through Data Reorganization”, ICS 2011, “First Place: Student Research Competition”M. Polte, J. Lofstead, J. Bent, G. Gibson, S. Klasky, "...And Eat it Too: High Read Performance in Write-Optimized HPC I/O Middleware File Formats," in In Proceedings of Petascale Data Storage Workshop 2009 at Supercomputing 2009, ed, 2009.S. Klasky, et al., “Adaptive IO System (ADIOS)”, Cray User Group Meeting 2008.Slide41

2008 – 2011 I/O Pipeline Publications

SOAS. Klasky, et al., “In Situ data processing for extreme-scale computing”, appear SciDAC 2011.C. Docan, J. Cummings, S. Klasky, M. Parashar, “Moving the Code to the Data – Dynamic Code Deployment using ActiveSpaces”, IPDPS 2011.F. Zhang, C. Docan, M. Parashar, S. Klasky, “Enabling Multi-Physics Coupled Simulations within the PGAS Programming Framework”, CCgrid 2011.C. Docan, S. Klasky, M. Parashar

, “DataSpaces: An Interaction and Coordination Framework for Coupled Simulation Workflows”, HPDC’10. ACM, Chicago Ill.

Cummings, Klasky, et al., “EFFIS: and End-to-end Framework for Fusion Integrated Simulation”, PDP 2010, http://www.pdp2010.org/.

C. Docan, J. Cummings, S. Klasky, M. Parashar, N. Podhorszki, F. Zhang, “Experiments with Memory-to-Memory Coupling for End-to-End fusion Simulation Workflows”, ccGrid2010, IEEE Computer Society Press 2010.N. Podhorszki, S. Klasky,  et al.: Plasma fusion code coupling using scalable I/O services and scientific workflows. SC-WORKS 2009Data FormatsY. Tian, S. Klasky, H. Abbasi, J. Lofstead, R. Grout, N. Podhorszki, Q. Liu, Y. Wang, W. Yu, “EDO: Improving Read Performance for Scientific Applications Through Elastic Data Organization”, to appear Cluster 2011.Y. Tian, “SRC: Enabling Petascale Data Analysis for Scientific Applications Through Data Reorganization”, ICS 2011, “First Place: Student Research Competition”M. Polte, J. Lofstead, J. Bent, G. Gibson, S. Klasky, "...And Eat it Too: High Read Performance in Write-Optimized HPC I/O Middleware File Formats," in In Proceedings of Petascale Data Storage Workshop 2009 at Supercomputing 2009, ed, 2009.S. Klasky, et al., “Adaptive IO System (ADIOS)”, Cray User Group Meeting 2008.Data StagingH. Abbasi, G. Eisenhauer, S. Klasky, K. Schwan, M. Wolf. “Just In Time: Adding Value to IO Pipelines of High Performance Applications with JITStaging, HPDC 2011.H. Abbasi, M. Wolf, G. Eisenhauer, S. Klasky, K. Schwan, F. Zheng, “DataStager: scalable data staging services for petascale applications”, Cluster Computer, Springer 1386-7857, pp. 277-290, Vol 13, Issue 3, 2010.C. Docan, M. Parashar, S. Klasky: Enabling high-speed asynchronous data extraction and transfer using DART. Concurrency and Computation: Practice and Experience 22(9): 1181-1204 (2010)H. Abbasi, Wolf, M., Eisenhauer, G., Klasky, S., Schwan, K., , Zheng, F. 2009. DataStager: scalable data staging services for petascale applications. In Proceedings of the 18th ACM international Symposium on High Performance Distributed Computing (Garching, Germany, June 11 - 13, 2009). HPDC '09. ACM, New York, NY, 39-48.H. Abbasi, J. Lofstead, F. Zheng, S. Klasky, K. Schwan, M. Wolf, “Extending I/O through High Performance Data Services”, Cluster Computing 2009, New Orleans, LA, August 2009.C. Docan, M. Parashar, S. Klasky, “Enabling High Speed Asynchronous Data Extraction and Transfer Using DART,” Proceedings of the 17th International Symposium on High-Performance Distributed Computing (HPDC), Boston, MA, USA, IEEE Computer Society Press, June 2008.H. Abbasi, M. Wolf, K. Schwan, S. Klasky, “Managed Streams: Scalable I/O”, HPDC 2008.Slide42

2008 – 2011 I/O Pipeline Publications

Usability of OptimizationsJ. Lofstead, F. Zheng, Q. Liu, S. Klasky, R. Oldfield, T. Kordenbrock, Karsten Schwan, Matthew Wolf. "Managing Variability in the IO Performance of Petascale Storage Systems". In Proceedings of SC 10. New Orleans, LA. November 2010.

J. Lofstead, M.

Polte, G. Gibson, S. Klasky, K. Schwan, R. Oldfield, M. Wolf, “Six Degrees of Scientific Data: Reading Patterns for extreme scale science IO”, HPDC 2011.

Y. Xiao, I. Holod, W. L. Zhang, S. Klasky, Z. H. Lin, “Fluctuation characteristics and transport properties of collisionless trapped electron mode turbulence”, Physics of Plasmas, 17, 2010.J. Lofstead, F. Zheng, S. Klasky, K. Schwan, “Adaptable Metadata Rich IO Methods for Portable High Performance IO”, IPDPS 2009, IEEE Computer Society Press 2009.Lofstead, Zheng, Klasky, Schwan, “Input/Output APIs and Data Organization for High Performance Scientific Computing”, PDSW SC2008. J. Lofstead, S. Klasky, K. Schwan, N. Podhorszki, C. Jin, “Flexible IO and Integration for Scientific Codes Through the adaptable IO System”, Challenges of Large Applications in Distributed Environments (CLADE), June 2008.Slide43

2008 – 2011 I/O Pipeline Publications

Data ManagementJ. Kim, H. Abbasi, C.

Docan, S. Klasky, Q. Liu, N. Podhorszki, A. Shoshani, K. Wu, “Parallel In Situ Indexing for Data-Intensive Computing”, accepted 2011 LDAV

T. Critchlow

, et al., “Working with Workflows: Highlights from 5 years Building Scientific Workflows”, SciDAC 2011.S. Lakshminarasimhan, N. Shah, Stephane Ethier, Scott Klasky, Rob Latham, Rob Ross, Nagiza F. Samatova, “Compressing the Incompressible with ISABELA: In Situ Reduction of Spatio-Temporal Data”, Europar 2011.S. Lakshminarasimhan, J. Jenkins, I. Arkatkar, Z. Gong, H. Kolla, S. H. Ku, S. Ethier, J. Chen, C.S. Chang, S. Klasky, R. Latham, R. Ross, , N. F. Samatova, “ISABELA-QA: Query-driven Analytics with ISABELA-compressed Extreme-Scale Scientific Data”, to appear SC 2011.K. Wu, R. Sinha, C. Jones, S. Ethier, S. Klasky, K. L. Ma, A. Shoshani, M. Winslett, “Finding Regions of Interest on Toroidal Meshes”, to appear in Journal of Computational Science and Discovery, 2011J. Logan, S. Klasky, H. Abbasi, et al., “Skel: generative software for producing I/O skeletal applications”, submitted to Workshop on D3Science, 2011.E. Schendel, Y. Jin, N. Shah, J. Chen, C.S. Chang, S.-H. Ku, S. Ethier, S. Klasky, R. Latham, R. Ross, N. Samatova, “ISOBAR Preconditioner for Effective and High-throughput Lossless Data Compression”, submitted to ICDE 2012.I. Arkatkar, J. Jenkins, S. Lakshminarasimhan, N. Shah, E. Schendel, S. Ethier, CS Chang, J. Chen, H. Kolla, S. Klasky, R. Ross, N. Samatova, “ALACRI2TY: Analytics-driven lossless data compression for rapid in-situ indexing, storing, and querying”, submitted to ICDE 2012.Z. Gong, S. Lakshminarasimhan, J. Jenkins, H. Kolla, S. Ethier, J. Chen, R. Ross, S. Klasky, N. Samatova, “Multi-level layout optimizations for efficient spatio-temporal queries of ISABELA-compressed data”, submitted to ICDE 2012. Y. Jin, S. Lakshminarasimhan, N. Shah, Z. Gong, C Chang, J. Chen, S. Ethier, H. Kolla, S. Ku, S. Klasky, R. Latham, R. Ross, K. Schuchardt, N. Samatova, “S-preconditioner for Multi-fold Data Reduction with Guaranteed user-controlled accuracy”, ICDM 2011.Slide44

2008 – 2011 I/O Pipeline PublicationsSimulation Monitoring

R. Tchoua, S. Klasky, N. Podhorszki, B. Grimm, A. Khan, E. Santos, C. Silva, P. Mouallem, M. Vouk, "Collaborative Monitoring and Analysis for Simulation Scientist", in Proceedings of CTS 2010.P. Mouallem, R. Barreto, S. Klasky, N. Podhorszki, M. Vouk. 2009. Tracking Files in the Kepler Provenance Framework. In Proceedings of the 21st International Conference on Scientific and Statistical Database Management (SSDBM 2009), Marianne Winslett (Ed.). Springer-Verla, 273-282.E. Santos, J. Tierny, A. Khan, B. Grimm, L.

Lins, J. Freire, V.

Pascucci, C. Silva, S. Klasky, R. Barreto, N.

Podhorszki, “Enabling Advanced Visualization Tools in a Web-Based Simulation Monitoring System”, escience 2009.R. Barreto, S. Klasky, N. Podhorszki, P. Mouallem, M. Vouk: “Collaboration Portal for Petascale Simulations”, International Symposium on Collaborative Technologies and Systems, pp. 384-393, Baltimore, Maryland, May 2009. R. Barreto, S. Klasky, C. Jin, N. Podhorszki, “Framework for Integrating End to End SDM Technologies and Applications (FIESTA).”, Chi 09 workshop, The Changing Face of Digital Science, 2009.Klasky, et al., “Collaborative Visualization Spaces for Petascale Simulations”, to appear in the 2008 International Symposium on Collaborative Technologies and Systems (CTS 2008).Slide45

2008 – 2011 I/O Pipeline Publications

In situ processingA. Shoshani, et al., “The Scientific Data Management Center: Available Technologies and Highlights”,

SciDAC 2011.

A. Shoshani, S. Klasky, R. Ross, “Scientific Data Management: Challenges and Approaches in the Extreme Scale Era”, SciDAC 2010.

F. Zheng, H. Abbasi, C. Docan, J. Lofstead, Q. Liu, S. Klasky, M. Parashar, N. Podhorszki, K. Schwan, M. Wolf, “PreDatA - Preparatory Data Analytics on Peta-Scale Machines”, IPDPS 2010, IEEE Computer Society Press 2010.S. Klasky, et al., “High Throughput Data Movement,”, “Scientific Data Management: Challenges, Existing Technologies, and Deployment,” Editors: A. Shoshani and D. Rotem, Chapman and Hall, 2009G. Eisenhauer, M. Wolf, H. Abbasi, S. Klasky, K. Schwan, “A Type System for High Performance Communication and Computation”, submitted to Workshop on D3Science, 2011.Misc.C. S. Chang, S. Ku, et al., “Whole-volume integrated gyrokinetic simulation of plasma turbulence in realistic diverted-tokamak geometry”, SciDAC 2009.C. S. Chang, S. Klasky, et al., “Towards a first-principles integrated simulation of tokamak edge plasmas”, SciDAC 2008.Z Lin, Y. Xiao, W. Deng, I. Holod, C. Kamath, S. Klasky, Z. Wang, H. Zhang, “Size scaling and nondiffusive features of electron heat transport in multi-scale turbulence”, IAEA 2010.Z. Lin, Y. Xiao, I. Holod, W. Zhang, W. Deng, S. Klasky, J. Lofstead, C. Kamath, N. Wichmann, “Advanced simulation of electron heat transport in fusion plasmas”, SciDAC 2009.Slide46

FutureStep

7Slide47

Integrating new technologiesNew collaborations blended with current collaborators to identify the gaps, and who can deliver solutionsNew techniques for I/O pipeline optimizations on HPC Utilizing new memory technologies more effectively

Hybrid Staging areaPGAS-like programming models for in situ processingIntegrating with cloudsExtend ADIOS to stream ‘analysis’ data from staging area to ‘cloud’ resourcesAllow researchers to build components in the open frameworkExplore cloud file systems to match programming models used for HPC analyticsAllow for extra levels of fault toleranceExplore different programming models for HPC + CloudsSlide48

Possible integration of ADIOS + TwisterADIOS componentizes data movement for I/O pipelinesReplace communication mechanism with ADIOS APIsWhat changes in ADIOS will be necessary to support Twisters requirements?

Allows for RDMA, socket, and file data movementHow well will this work with a cloud file system?Disk/Stream Map Disk/Stream Reduce

ADIOS

: methods for low latency go to RDMA/memory references, high latency/persistence file system

iterationsin memory Map ReduceTwister (Pub/Sub/Collectives)service service Cloud/HPC file systemCloud/HPC file systemCloud/HPC file systemCloud/HPC file systemCloud/HPC file systemCloud/HPC file systemSlide49

Conclusions and Future DirectionStrong leadership and a complete understanding of the domain allows a community to lead from innovation

ADIOS, developed by many institutions, is a proven DOE framework which blends togetherSelf-describing file formats for high levels of I/O performanceStaging methods with visualization and analysis “plugins”Advanced data management plugins, for data reduction, multi-resolution data reduction, and advanced workflow schedulingMultiplexing using a novel SOA approachAdded levels of resiliency for next generation knowledge discover in HPCFuture work should build on these successes to integrate HPC + cloud technologyOur goal is to deliver breakthrough science and get the job done no matter what it takes