High Performance Computing for MHD - PowerPoint Presentation

Slide1

High Performance Computing for MHD

Ramakanth Munipalli, P.-Y. HuangHyPerComp Inc., Westlake Village, CA 91361In collaboration withS.Smolentsev, N.B.Morley, A.Ying, G.Pulugundla, M.Abdou,UCLA2nd EU-US DCLL Workshop, UCLA, November 15, 2014Research supported by current and prior SBIR funding from DOESlide2

Outline

1. HyPerComp Inc. – who are we? Small company, located 30 miles NW of LA, estd. 1998, Lengthy, fruitful collaboration with UCLA in numerical MHD2. MHD – what have we done so far in fusion related MHD modeling?3. Multiphysical extensions Mass transport, corrosion, integrated multiphysical modeling4. What are we working on now?5. High performance computing – some tools and techniquesSome prospects and future directionsSlide3

1. HyPerComp Inc. – Who are we?Slide4

HyPerComp develops high performance computing software in fluid mechanics and electromagnetics

3 Recent Examples:Very large scale CPU/GPU parallel electromagnetics modeling Liquid rocket combustion dynamicsCoupled fluid-structure-acoustics modeling for rotorcraftSlide5

Computational Electromagnetics

RCS, SAR Imagery and range profiles,Penetrable materials, Antennas, Optics Magnetohydrodynamics& PlasmadynamicsFlow Control,Liquid metal MHD – nuclear fusion,Combustion controlComputational Fluid DynamicsIncompressible to Hypersonic flow,Turbulent Combustion, Rotorcraft, Acoustics, Morphing structures, Optimization

High Performance Multiphysical ComputationsVery high order accurate solutions, CPU/GPU based solvers, Reduced Basis Models

A Technology Portfolio

TEMPUS

HYCESlide6

2. MHD – what have we done so far?

MHD Top-level concern in fusion blanket studies (pressure drop, corrosion, tritium transport, etc.)Difficult problem to solve No analytical solutions in all but the most elementary flows Numerical solutions riddled with difficulties and uncertaintiesSlide7

H

yPerComp Incompressible MHD solver for Arbitrary GeometryHIMAG was developed by HyPerComp in a research partnership with UCLA to model the flow of liquid metals in nuclear fusion reactor design. These flows are characterized by very high magnetic field strength, complex geometry and fluid-solid coupling via electric current and heat. HIMAG is a pioneer in the reliable computation of such flows among both commercial as well as research simulation software.Slide8

HIMAG is a parallel, unstructured mesh based MHD solver.

High accuracy at high Hartmann numbers is maintained even on non- orthogonal meshes HIMAG can model single-phase as well as two-phase (free surface) flows Multiple conducting solid walls may be present in the computational domain Graphical User Interfaces are provided for the full execution of HIMAG Heat transfer, natural convection, temperature dependent properties can be modeledExtensive validation and benchmarking has been performed for canonical problems. Cases involving Ha > 1000 have never been

demonstrated on non-rectangular meshes prior to HIMAG

HIMAG: Technical SummarySlide9

Governing Equations

Two phase fluid flow without MHD

Two phase fluid flow with MHD, heat transferSlide10

Treatment of electromagnetic fields in liquid metal MHD

When the magnetic Reynolds number is low (Rem = μ0σLU << 1), induced magneticfield is negligible. Applied magnetic field is the total magnetic field, and an electric potentialmay be derived:

The potential is obtained from the condition that the current density is

solenoidal

Ohm’s law:

A more general treatment solves for the induced magnetic field:

solid

fluid

Electric current can flow across interfaces between media

Appropriate simplifications (J=0, etc.), or,

Boundary conditions (normal current = 0) can be usedSlide11

Numerical difficulties in computing high Ha MHD flows

The need to resolve and minimize numerical errors in Hartmann layers(Thickness about 1/Ha) The pace of convergence of Poisson equation solvers(for Pressure, Electric potential, divergence of B)Computationally intensive corrections for non-orthogonal meshesLong periods of integration needed to account for flow development, unsteady effectsTime taken to develop CAD to CFD mesh for high Ha problemsThe high Hartmann number problemAs Ha = increases, regions of sharp variations in velocity and electric current density appear in the flow.Numerically, small inaccuracies in the Hartmann layer are greatly amplified

Flow enters a strong B-fieldSlide12

1 / Ha

1 / sqrt(Ha)

U(z)

j(y,z)

d

Ha

≈Ha

-1

B

d

//

≈Ha

-1/2

2h

Hartmann layer

Side layer

FLOW

The exacting needs of numerical MHDSlide13

Fully developed flow at Ha = 10,000 in a duct

with square cross section – compared with the exact analytical solution

Verification against canonical benchmark problems

Fully developed flow at Ha = 1,000 in a duct

with circular cross sectionSlide14

Ra = 10

3Ra = 104

Ra = 10

5

Ra = 10

6

Ra

Mesh

Nu

HIMAG

Nu

comp

10

3

21x21

41x41

81x81

1.1227

1.1191

1.1181

1.118

10

4

21x21

41x41

81x81

2.3084

2.2611

2.2488

2.245

10

5

21x21

41x41

81x81

4.9370

4.6312

4.5490

4.522

10

6

21x21

41x41

81x81

161x161

10.661

9.4598

8.9885

8.8659

8.829

Natural Convection

A good match of

Nusselt

number with published data

has been observed. Second order accuracy has

been ascertained.Slide15

Ha=5800

Validation against experimental data: ALEX duct experiment (1987, ANL)

Circular duct

Square ductSlide16

Wall functions

Contact resistanceCoarse mesh in Ha layers

Jump in potential at arbitrary material interfaces can now be captured

HIMAG:

Some MHD specific

numericsSlide17

Robust modeling of strong natural convection

Newton-Krylov based schemes are used to perform matrix inversion in a upwind semi-implicit procedure to stabilize the simulation of flow with strong natural convection (Gr = 109). 2a

=20 cm

B

2 m

Y

X

40 cm

Gr = 10

9

in a square cavity

Streamlines (left) and isotherms (right)

Flow in a 3-D channel

with MHD and heat transfer

(results in next chart)Slide18

Upward flow

# cells

Speedup

Full solution iwall=0

299,440

1

Wall function iwall=10

70,080

19.18

Wall function iwall=11

162,336

6.2

Flow Re = 10,000, Ha = 400, Gr = 10

7

Wall functions are used to model MHD flow in an insulating channel with dimensions similar to DCLL. These functions are applied at all walls (iwall=10) or only at Hartmann walls (iwall=11)

Temperature

Velocity contours

Full MHD solution (left) and solution with wall functions (right)

g

A case study in the use of wall functions in solution acceleration

Sample speedup results on 16 CPUs

Comparison of full and wall function

solutions along centerlineSlide19

DCLL concept and crucial MHD featuresSlide20

z

x

y

B

g

Full 3-D MHD (with natural convection) model of the DCLL

Flow enters from lower right, exits above

A low conductivity flow channel insert is present

Velocity profiles shown on right and pressure on leftSlide21

3. Multiphysical studiesSlide22

Fusion relevant liquid metal MHD includes a variety of multiphysical

phenomena Fluid, heat and mass transport Natural convection Steady and unsteady electromagnetic phenomena Contact resistance (thermal and electric) Ferromagnetic effects Two phase flow, surface tension Phase change (due to high heat flux, pressure variation, etc.) Corrosion, electrochemistry at walls Fluid-structure interactionSlide23

HIMAG: Free surface flow simulations

MTOR, experiment at UCLALiquid metal jet in a magnetic fieldSlide24

Plasma

Liquid metal

Plasma-liquid interaction

Plasma-liquid metal interaction in a magnetic field: DiMES

HIMAG uses

the level set method in unstructured meshes.

Method permits large deformations of the free surface.

We are presently working

on massively parallel free

surface capture

simulations with scalable adaptive meshing

(w. RPI)

HIMAG: Free surface flow simulations – contd.Slide25

Mass Transport

Tritium transportWe built an independent software named CATRIS from the basic data structure of HIMAG, to focus on specific issues in mass transport. CorrosionSome capabilities to simulate corrosion were added to CATRIS, together with Lagrangian models for particulate transport Slide26

CATRIS (Corrosion And TRItium transport Solver)

Written as a new stand-alone system, CATRIS focuses on the following: the transport of tritium and its permeation through walls corrosion and deposition of iron contained within structural materials of the system. Slide27

A sample study to inject particles in an MHD flow, which migrate in response to the gradient in an applied magnetic field. These particles will be produced at walls using a corrosion BC and deposited likewise.

Tritium concentration in module computed for B=4T (Ha = 15 000), u=0.065m/s.Tritium transport and particle transport models in CATRIS Slide28

4. What are we working on now?Slide29

Current Research Objectives

Brand new software implementation of the induced magnetic field formulation – adds ability to model high magnetic Reynolds number, strongly unsteady flows Improvement in time accuracy Enhanced robustness and speed – better parallelization Dramatically reduce simulation time for MHD flows in geometries of practical interest – DCLL, etc. Transition to general EM applications in materials processingSlide30

Induced magnetic field formulationSlide31

b) Velocity profile along x=0 and z=0 sections

Case 1 – 3-D Lid-driven cavity with conducting walls except top lid (Re=100, Ha=10, Cw=0.4)a) Schematic view of 3-D lid-driven cavity with conducting walls (Blue area), Red – Liquid.

U

BSlide32

f) Convergence history.

Case 1 – Cavity with conducting walls - Continuede) Comparison of velocity curves by B-formulation with those by potential solver (HIMAG).Slide33

Case 2 – 3-D Lid-driven cavity with insulating walls (Re=100, Ha=45)

b) Velocity profile along x=0.5 and y=0.5 sectionsa) Schematic view of 3-D lid-driven cavity with insulating walls: Red – Liquid.

U

BSlide34

Comparison of velocity u(z) by B-formulation with published data

Case 2 – Cavity with insulating walls - ContinuedComparison of velocity w(x) by B-formulation with published dataSlide35

5. High Performance ComputingSlide36

Mathematical methodsMultigrid methods:

Agglomeration, unnested pre-meshed scheme, algebraic multigridHybrid meshesImplicit schemes – time stepping, accuracyCN, AB, SIRK schemesSIMPLE scheme for steady flowLocal time steppingFull matrix solvers – BICGSTAB on CPU/GPUInterpolation proceduresData storage: Non-orthogonal correctionGPU-based programmingRapid prototypingCanonical decomposition of DCLL geometryUser interfaces for inlet, fci, bend, etc.Template for multigrid (3-D) – dir. agglomerationTemplate for hybrid mesh generation blockingApplications of analysis EM coupling between neighboring channelsFringing fields, self-consistent field formulationsWall functions – insulating, perfectly conductingWall functions for fringing fieldsPatched analytical – numerical solutionCombination of duct flow solutions

Summary of our approachSlide37

Poisson solver with Neumann BCs

Hunt’s fully developed channel flow3-D circular duct Ha = 10, Re = 10, cw = 0.13-D rectangular ductHa = 0, Re = 15,250

3-D rectangular duct

Ha = 100, Re = 10,

c

w

= 0.1

Poisson solver acceleration by sparse matrix storage and inversionSlide38

The use of non-orthogonal meshes

Among the many benefits of the HIMAG approach has been the preservation ofaccuracy on arbitrary mesh systems – making the calculation more efficient

Hybrid/unstructured meshes reduce number of meshpoints needed, by transitioning between regions of different resolution requirements Slide39

“Template” driven mesh generationSlide40

HIMAG:

A nested multigrid method

A sequence of

unnested

grids for

multigrid

Poisson solution, showing reduced convergence time (above

), automatic

generation of a sequence of meshes from CAD input (below) Slide41

Some prospects and future directionsSlide42

Summary

We have a number of ongoing software development activities which can servesimulation needs in fusion: Robust, comprehensive, self-consistent MHD physics modeling CPU/GPU parallel computing Highly scalable parallel mesh adaptation Strengths in coupled MHD/heat/mass transfer analysis (numerous existing physical and numerical models) Integrated and customizable software solutionsSlide43

Future Directions

EM toolkit for flow and materials processingCombined calculation of fluid flow, heat transfer (radiative, convective), mass transfer (including corrosion) and electromagnetics - applications in aerospace propulsion, materials processing2. Localized plasma models – thermal as well as weak ionization, extended to large length scales as source terms (external flow control, plasma assisted processing, radar analysis)3. Two phase flow - models of free surfaces such as melt layers4. Phase change - evaporation, melting, solidification, crystallization, etc.