PhD defense: Marta Garc í - PowerPoint Presentation

1. PhD defense: Marta GarcíaDevelopment and validation of the Euler-Lagrange formulation on a parallel and unstructured solver for large-eddy simulationDirector: T. Poinsot & Co-director: V. Moureau

2. THE CONTEXTHuman nature: try to understand phenomena, comprehension effort, power of fire, energy conversion … Observation Experimentation Numerical simulation Gas turbineenginesfirelocomotiveThis thesis is focused on the improvement of current tools to the comprehension of multiphase flows by using numerical simulation.

3. THE CONTEXT: EXPERIMENTS vs NUMERICAL SIMULATIONSEXPERIMENTSNUMERICAL SIMULATION expensive destructives difficult to reproduce exactly less expensive not destructives reproductibilityHam et al. Annual Research Briefs 2003 CTR Stanford Univ.Spray evolution from a realistic gas-turbine injector.

4. THE CONTEXT: INCREASE OF COMPUTER POWER(1) Flops: floating point operations per second1.65x1012 Flops(1)280x1012 Flops1012 = 1.000.000.000.0001105x1012 Flops12.64x1012 FlopsIn the last 3 years CPU time divided by 8 (approx.)8 weeks  1 week

5. THE CONTEXT: TWO-PHASE FLOW NUMERICAL SIMULATIONEulerian formulationPhDcerfacs2 21PhDimft1PhDimft1up~ TPF TeamCurrent treatment of the dispersed phase in AVBP(TPF)PhDPhDPhDPhDPhDPhDPhDPhDPhDPhDPhDPhDPhDPhDPhDLagrangian formulation ? in 2005 …PhD … considering the increasing computer power and my experience accumulated in the past 3 years … ???Kaufmann 2004PhD’s: CERFACS & IMFTMossa 2005Pascaud 2006Boileau 2007Riber 2007Lavedrine 2008 Lamarque 2007 …PhDcerfacs2

6. THE CONTEXT: TWO-PHASE FLOW NUMERICAL SIMULATIONEuler-Euler vs Euler-LagrangeEuler-LagrangeIndividual particle trajectories are computed Easy modeling of particle movements and interactions. Robust and accurate if enough particles are used. Size distributions easy to describe. Easy to implement physical phenomena (e.g. heat and mass transfer, wall-particle interaction). Delicate coupling with combustion. Difficult to run in parallel. Each ‘particle’ actually represents an ensemble of particles.Euler-Euler Easy treatment of dense zones. Similarity with gaseous equations. Direct transport of Eulerian quantities. Similarity with gaseous parallelisme. Difficult description of polydispersion. Difficulty of crossing sprays treatment. Limitation of the method in very dilute zones.Particle ensemble viewed as a continuous field

7. THE OBJECTIVES OF THE WORK Develop a Lagrangian formulation for two-phase flow treatment within a parallel, unstructured and hybrid solver AVBP. Perform the first simulations on academic and complex geometries. Verify the efficient parallel implementation to maintain good performance on massively parallel machines.

8. THE PLAN OF THE PRESENTATIONPresentation of the simulation tool: AVBP solverDescription of the Lagrangian moduleApplication test cases:Conclusions and perspectives Decaying homogenous isotropic turbulence Polydisperse two-phase flow of a confined bluff body Particle equations of motion Particle tracking algorithm Quick introduction Domain partitioning Rounding errors and repetitivity of LES

9. THE AVBP SOLVER Parallel solver started in 1993. Unstructured solver capable of handling hybrid grids of different cell types. Computational Fluid Dynamics (CFD) code to solve laminar and turbulent compressible Navier-Stokes equations in 2 and 3 space dimensions. Built upon a modular software library that includes integrated parallel domain partition and data reordering tools, message passing (MPI) and includes supporting routines for dynamic memory allocation, routines for parallel I/O and iterative methods. Written in standard Fortran 77 and C, but it is being upgraded to Fortran 90 in a gradual fashion. Highly portable to different parallel machines.

10. THE AVBP SOLVER: PARTITIONING ALGORITHMSRCBRIBRGBR = recursiveB = bisectionC = coordinateI = inertialG = graph… currenly available in AVBPDUAL MESHNODAL MESH

11. THE AVBP SOLVER: PARTITIONING ALGORITHMSMESHTotal No of nodes495,232503,230530,852367,313(before partitioning)(after partitioning)RCBRIBRGB+ 35%+ 37%+ 45%CPU time of 1000 it. (s)361.5 366.96405.64+ 1.5%+ 12%The choice of partitioning algorithm has an effect on the CPU time of your simulation. Need of a new partitioning algorithm: Faster partitioning Lower number of total nodes after partitioning With parallel version With multi-constraint partitioning optionsChoice done: METIS package implemented during this thesis

12. THE AVBP SOLVER: PARTITIONING ALGORITHMSSome results obtained with METIS multilevel partitioning algorithm …ARRIUS2_10MARRIUS2_44MCOMPARISON OF ALGORITHMSNo of nodes after partitioningNo of nodes after partitioningNo of subdomainsNo of subdomainsNo of subdomains29 minutes21 minutes4096 procsMETIS algorithm is fasterIt produces a lower number of nodes after partitioning

13. THE AVBP SOLVER: ROUNDING ERRORS… and repetitivity of LESMESHFinite precision computation: lack of associativity property !! AVBP: Parallel solver, highly portable to solve laminar and turbulent compressible Navier-Stokes equations.What that means …zoomCMRCM ABCDABCDWork published in the AIAA Journal publication:AIAA Journal Vol. 46, No 7, July 2008“Growth of Rounding Errors and Repetitivity of Large-Eddy Simulations”J.-M. Senoner, M. García, S. Mendez, G. Staffelbach, O. Vermorel and T. Poinsot

14. THE AVBP SOLVER: ROUNDING ERRORSAxial velocity fields of a turbulent channel (TC) at different instants(t1)4 procs8 procsAxial velocity (m/s)(t2)Axial velocity (m/s)(t3)Axial velocity (m/s)4 procs8 procs4 procs8 procsInstantaneous solutions in unsteady simulations.Same initial conditions.Different number of processors.DIFFERENCES OBSERVED BETWEEN TWO SNAPSHOTSTWO NORMS ARE USED TO COMPARE RESULTS BETWEEN TWO SOLUTIONS

15. THE AVBP SOLVER: ROUNDING ERRORSDifferent effects observed on repetitivity of LESEffect of node reorderingEffect of initial conditionsReprinted by permission of the American Institute of Aeronautics and Astronautics.Effect of machine precisionquadrupledoublesimpleEffect of turbulenceturbulentlaminarMachine precision differencesNorm saturationAny sufficiently turbulent flow computed in LES exhibits significant sensitivity to small perturbations, leading to instantaneous solutions which can be totally different.The divergence of solutions is due to 2 combined facts: The exponential separation of trajectories in turbulent flows. The different propagation of rounding errors induced by domain partitioning and scheduling operations.The validation of an LES code after modifications may only be based on statistical fields.

16. THE PLAN OF THE PRESENTATIONPresentation of the simulation tool: AVBP solverDescription of the Lagrangian moduleApplication test cases:Conclusions and perspectives Decaying homogenous isotropic turbulence Polydisperse two-phase flow of a confined bluff body Particle equations of motion Particle tracking algorithm Quick introduction Domain partitioning Rounding errors and repetitivity of LES

17. THE LAGRANGIAN MODULE: PARTICLE EQUATIONS… of motionIndividual particle trajectories are computed with a Lagrangian solver coupled to the LES code for the gas phase.N droplets to track (order of a few millions) [ Schiller & Nauman. 1935 ]Assumptions:spheres Particles equation of motion Need to know the gas velocity at each particle location (linear interpolation) The effect of the subgrid fluid velocity is not considered in this thesis.[ Fede & Simonin 2006 ]drag + gravity

18. THE LAGRANGIAN MODULE: KEY POINTS X3X1X2n1n2n3 Xp ( Xp-Xi ) ni 0Locating particles in cellsKnowing particles positions at time n: exchange particles between processors cell iparticleSubdomain 1Subdomain 2influence nodeTwo-way couplingParticles load-balancingInterpolation algorithm gas velocity ug,i at each particle locationInjection…Particle-wall treatment… for Lagrangian schemes in unstructured meshes

19. THE LAGRANGIAN MODULE: LOCATING PARTICLES Shape functions: Calculation of partial volumes:… in elements of arbitrary shape2D:3D:To decide if the particle is in the cell or not, the scalar product between the vector starting from the vertex of the cell to the particle and the inward normal vector of the corresponding edge is taken. The particle is inside the cell if all the scalar products of each edge are positive. X1X2n1n2n3 XpXp X3n3-+n3 ( Xp-Xi ) ni 0 Face-normals:

20. THE LAGRANGIAN MODULE: SEARCH ALGORITHMS… for different situationsSearch particles for the first timeSearch injected particlesSearch particles during simulationSearch particles crossing boundaries between processors Cells of the interface (type 2) Initial particle location New particle location Interface between processors Old cell containing the particle Cells surrounding the old cell Initial particle location Nodes of the containing cell New particle location Cells of the injection areaParticles injectedInterface between processorsInjection areaNodes of the injection areaQuad/OctreeF. Collino(CERFACS)Use of different search algorithms depending on the situation to reduce memory and CPU time requirements.

21. THE LAGRANGIAN MODULE: INTERPOLATIONEx. 3D with hexa: n=2 (trilinear interpolation) Lagrange interpolation (only for coordinate grids with quads or hexahedras)1222 Linear Least Squares (LAPACK subroutine DGELS) 1st order Taylor SerieEx. 1D and 1st order… of gaseous-phase properties at particle position

22. THE LAGRANGIAN MODULE: TWO-WAY COUPLING= Coupling force12346789= constant of proportionalityExist an analytical solutionValidation test of two-way couplingMomentum eq. = cte[ Boivin et al. 1998, Boivin et al. 2000 ][ Ph.D. O. Vermorel 2003 ]xyIn the framework of PIC methods… source terms and validation

23. THE LAGRANGIAN MODULE: PARTICLE INJECTION INJECTION GEOMETRY: simple injection options available Point injection: all droplets are injected at the same point. Disk injection: droplets are injected over a disk.Example of input parameters: Coordinates of the injection point, disk diameter, normal to define disk direction, tolerance … PARTICLE SIZE DISTRIBUTION: Monodisperse: all particles have the same diameter. Polydisperse: different particle diameters.Gaussian distributionLog-normal distributionExample of input parameters: Type of distribution, maximum and minimum diameters, mean and standard deviation …

24. THE LAGRANGIAN MODULE: PARTICLE INJECTION- Particle mass flow rate- Particle diameter(s), density, mean/rms velocity ...ZOOMInjection tubez=-3mmInject # particles by timestep… example of a disk injection

25. THE PLAN OF THE PRESENTATIONPresentation of the simulation tool: AVBP solverDescription of the Lagrangian moduleApplication test cases:Conclusions and perspectives Decaying homogenous isotropic turbulence Polydisperse two-phase flow of a confined bluff body Particle equations of motion Particle tracking algorithm Quick introduction Domain partitioning Rounding errors and repetitivity of LES

26. THE APPLICATION TEST CASESDecaying Homogeneous Isotropic turbulence (HIT)Polydisperse two-phase flow of a confined bluff body Academic test case Well documented Complex recirculating flow Large amount of data available

27. THE APPLICATION TEST CASES: DECAYING HIT Simple configuration to: Validate first developments of the Lagrangian version. Localisation algorithms, interpolation, processor exchanges, etc.

28. THE APPLICATION TEST CASES: DECAYING HIT Illustration of preferential concentrationValidation of particle kinetic energy resultsPerformance analysis of particle location322416815105020x103

29. THE APPLICATION TEST CASES: CONFINED BLUFF BODYBorée, J., Ishima, T. and Flour, I. 2001. The effect of mass loading and inter-particle collisions on the development of the polydispersed two-phase flow downstream of a confined bluff body. J. Fluid Mech., 443, 129-165.z (m)r (m)EDF - R&DDescription of the configurationWork published in Journal of Computational Physics:J. Comput. Phys. Vol. 228, No 2, pp. 539-564 2009“Evaluation of numerical strategies for large eddy simulation of particulate two-phase recirculating flows”E. Riber, V. Moureau, M. García, T. Poinsot and O. Simonin

30. THE APPLICATION TEST CASES: CONFINED BLUFF BODYNumerical parameters GridTetrahedraHexahedraNb cells2 058 8833 207 960Nb nodes 367 3133 437 576Nb particles~ 560 000 ~ 370 000Time step (ms)3,24,22LES modelSmagorinskyWALETurbulence injection on gasNoYesTurbulence injection on particlesYesYesWall treatmentLaw of the wall (Schmitt et al. JFM 2006)No slipScheme3rd order TTGC scheme compressibleTwo-way couplingYesIn the following: results of velocity profiles of the polydisperse simulationAt the end of the presentation: study of particle load imbalance

31. THE APPLICATION TEST CASES: CONFINED BLUFF BODYAnimation with AVBP-EL: gas velocity modulus with particles

32. THE APPLICATION TEST CASES: CONFINED BLUFF BODYParticle trajectories for the polydisperse case20 microns40 microns60 microns80 micronsParticle trajectories of polydisperse case give expected results, behavior is different depending on the particle size.Lighter particles respond to the flow faster. Their trajectories are deviated and more influenced by turbulence. Heavier particles penetrate more into the recirculation bubble.

33. THE APPLICATION TEST CASES: CONFINED BLUFF BODYz (m)`r (m)20 microns40 microns60 microns80 microns4 int12 int20 int3 mm8 int16 int10 int4 int12 int20 int3 mm8 int16 int10 int4 int12 int20 int3 mm8 int16 int10 int4 int12 int20 int3 mm8 int16 int10 int0.1300.0750.0103 days12 days45 daysCross-section velocity profiles (effect of the No of samples)

34. THE APPLICATION TEST CASES: CONFINED BLUFF BODY20 microns40 microns60 microns80 micronsAxial mean particle velocity profiles: [-2, 6] (m/s)EXP:AVBP_EL: 0.25, 1, 4 (s)Location of recirculation zone is shifted by a few mm.

35. THE APPLICATION TEST CASES: CONFINED BLUFF BODY20 microns40 microns60 microns80 micronsEXP:AVBP_EL: 0.25, 1, 4 (s)Axial RMS particle velocity profiles: [0.0, 1.5] (m/s)Minor problem to capture the first stagnation point

36. THE APPLICATION TEST CASES: CONFINED BLUFF BODY20 microns40 microns60 microns80 micronsEXP:AVBP_EL: 0.25, 1, 4 (s)Radial mean particle velocity profiles: [-1, 1] (m/s)

37. THE APPLICATION TEST CASES: CONFINED BLUFF BODY20 microns40 microns60 microns80 micronsEXP:AVBP_EL: 0.25, 1, 4 (s)Radial RMS particle velocity profiles: [0.0, 1.5] (m/s)

38. THE APPLICATION TEST CASES: CONFINED BLUFF BODYImportant point to retain of a two-phase Lagrangian parallel simulation Gaseous phaseNOT A GOOD PARALLEL SIMULATION !!Gaseous phaseDispersed phaseNo cellsGaseous phaseDispersed phaseA GOOD PARALLEL LAGRANGIAN SIMULATION !!No particlesNo cells(B)(A)NOT A GOOD PARALLEL LAGRANGIAN SIMULATION !!

39. THE APPLICATION TEST CASES: CONFINED BLUFF BODY(A)(B)(A)(B)Imbalanced simulationBalanced simulationSingle-constraint (RIB) vs two-constraints (METIS) partitioning algorithm Meshedge-cut(A)(B)RIBMETISLoad-balancing the disperse phase with a two-constraint partitioning algorithm improves the performance of the two-phase Lagrangian simulation

40. CONCLUSIONS AND PERSPECTIVES The effects of rounding errors on the repetitivity of LES was demonstrated and analysed. An efficient implementation of a Lagrangian formulation is related to the study of partitioning algorithms, data structure, load-balancing capabilities and parallel facilities, between others. The increase of computer power opens a new way for two-phase Lagrangian simulations that were considered prohibitive years ago. Validation of the Lagrangian module in an Homogeneous Isotropic Turbulence (HIT) which allows a simple analysis of several aspects of performances and particle behavior. A more complete study and validation has been done in a particle-laden bluff-body configuration. Results are in good agreement with experiments. Feasibility demonstrated of load-balancing capabilities.CONCLUSIONS

41. CONCLUSIONS AND PERSPECTIVESModeling Evaporation model (Ph.D. F. Jaegle). Treatment of particle-wall interactions (Ph.D. F. Jaegle). Improvement of particle injection (Ph.D. J.M. Senoner + C. Habchi IFP). Introduction of collision and coalescence models. Introduction of subgrid-scale fluid velocity on particle components.PERSPECTIVESNumerics Improvement of searching algorithms and data structure. Improve analysis of current performances: communications, algorithms, memory requirements, etc.

42. Thank you for your attention !Any question ?

43. ARRIUS2_44M with RCB: 0.5 [hours] * 4096 [processors] * 0.2 [euros/processor/hour] = 409.6 euros !!Need of a new partitioning algorithm: Faster partitioning Less number of total nodes after partitioning With parallel version With multi-constraint partitioning optionsChoice done: METIS packageTHE AVBP SOLVER: PARTITIONING ALGORITHMSEffect of different partitioning algorithms on CPU timeSAME MESH, DIFFERENT ALGORITHMSSAME ALGORITHM, DIFFERENT MESHESRIB4.7 hours; 4096 procs

44. THE AVBP SOLVER: ROUNDING ERRORSThe representation of numbers BCB+B-C+C-A…A+A-zoom(71)10 = 7 x 101 + 1 x 100(1000111)2 = 1 x 26 + 0 x 25 + 0 x 24 + 0 x 23 + 1 x 22 + 1 x 21 + 1 x 20 decimalbinary(5.5)10 = (101.1)2 = 1 x 22 + 0 x 21 + 1 x 20 + 1 x 2-1binary (real)-1-20121/2-1/20.1Numbers represented in a lineA+D=CA+D’=B

45. THE APPLICATION TEST CASES: CONFINED BLUFF BODY(A)(B)(A)(B)Bad speedupGood speedupSingle-constraint (RIB) vs two-constraints (METIS) partitioning algorithm Meshedge-cut(A)(B)RIBMETISLoad-balancing the disperse phase with a two-constraint partitioning algorithm improves the performance of the two-phase Lagrangian simulation