SIAM R EVIEW Society for Industrial and Applied Mathematics Vol PDF document - DocSlides

SIAM R EVIEW  Society for Industrial and Applied Mathematics Vol PDF document - DocSlides

2014-12-12 121K 121 0 0


45No 2pp 259266 TheTortoise and the Hare Restart GMRES Mark Embree Abstract When solving large nonsymmetric systems of linear equations with the restarted GMRES algorithm one is inclined to select a relatively large restart parameter in the hope of ID: 22362

Direct Link: Embed code:

Download this pdf

DownloadNote - The PPT/PDF document "SIAM R EVIEW Society for Industrial and..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.

Presentations text content in SIAM R EVIEW Society for Industrial and Applied Mathematics Vol

Page 1
SIAM R EVIEW 2003 Society for Industrial and Applied Mathematics Vol. 45,No. 2,pp. 259–266 TheTortoise and the Hare Restart GMRES Mark Embree Abstract. When solving large nonsymmetric systems of linear equations with the restarted GMRES algorithm, one is inclined to select a relatively large restart parameter in the hope of mim- icking the full GMRES process. Surprisingly, cases exist where small values of the restart parameter yield convergence in fewer iterations than larger values. Here, two simple exam- plesarepresentedwhereGMRES(1)convergesexactlyinthreeiterations,whileGMRES(2) stagnates. One of these examples reveals that GMRES(1) convergence can be extremely sensitive to small changes in the initial residual. Key words. restarted GMRES convergence, (rylov subspace methods AMS subject classifications. )5F1,, -./-, PII. S,,-)1005,1-11)15 1. Introduction. GMRES is an iterative method for solving large nonsymmetric systems of linear equations, Ax [8]. Throughout science and engineering, this algorithmanditsvariantsroutinelysolveproblemswithmillionsofdegreesoffreedom. Its popularity is rooted in an optimality condition: at the th iteration, GMRES computes the solution estimate that minimizes the Euclidean norm of the residual Ax over a subspace of dimension = min (0)=1 ((.() where denotesthosepolynomialswithdegreenotexceeding , and Ax is the initial residual. As each iteration enlarges the minimizing subspace, the residual norm decreases monotonically. GMRES optimality comes at a cost, however, since each new iteration demands both more arithmetic and memory than the one before it. A standard work,around is to restart the process after some -xed number of iterations, . The resulting algorithm, GMRES( ), uses the approximate solution as the initial guess for a new run of GMRES, continuing this process until convergence. The global optimality of the original algorithm is lost, so although the residual norms remain monotonic, therestartedprocesscanstagnatewithanonzeroresidual, failingtoeverconverge[8]. Since GMRES( ) enforces local optimality on ,dimensional spaces, one anticipates Received by the editors 2ecember 12, 2,,13 accepted for publication (in revised form) /ovember ., 2,,23 published electronically May 2, 2,,-. 4his work was supported by 5( Engineering and 6hysical Sciences Research 7ouncil grant GR8M12010. http988www.siam.org8:ournals8sirev805-28-11)1.html Oxford 5niversity 7omputing ;aboratory, Wolfson Building, 6arks Road, Oxford O=1 -Q2, 5(. 7urrent address9 2epartment of 7omputational and Applied Mathematics, Rice 5niversity, )1,, Main Street—MS 1-0, Houston, 4= ..,,5-1A12 ( 259
Page 2
260 MARK EMBREE that increasing will yield convergence in fewer iterations. Many practical examples con-rm this intuition. .e denote the th residual of GMRES( )by . To be precise, one cycle betweenrestartsofGMRES( )iscountedas individualiterations. Conventionally, then, one expects for . Indeed, this must be true when Surprisingly, increasing the restart parameter sometimes leads to slower conver, gence: for . The author encountered this phenomenon while solving a discretized convection,di0usion equation described in [1]. In unpub, lished experiments, de Sturler [(] and .alker and .atson [((] observed similar be, havior arising in practical applications. 2ne wonders, how much smaller than might be3 The smallest possible cases compare GMRES(() to GMRES(4) for 5,by,5 matrices. Eiermann, Ernst, and Schneider present such an example for which (1) (2) =6 4(71 ... [4, pp. 481–487]. 2therwise, the phenomenon we describe has apparently received little attention in the literature. The purpose of this article is twofold. 9irst, we describe a pair of extreme ex, amples where GMRES(() converges exactly at the third iteration, while GMRES(4) seems to never converge. The second example leads to our second point: small per, turbations to the initial residual can dramatically alter the convergence behavior of GMRES((). These observations are based on careful calculations: the underlying mechanism behind these results is yet to be rigorously understood. ;opefully this note will spark further investigation of these curious issues. 2. First Example. Consider using restarted GMRES to solve Ax for ((( 6(5 66( (4.() Taking yields the initial residual . Using the fact that and are real, we can derive explicit formulas for GMRES(() and GMRES(4) directly from the GMRES optimality condition ((.(). The recurrence for GMRES((), (1) +1 (1) (1)T Ar (1) (1)T Ar (1) Ar (1) (4.4) was studied as early as the (=76s [5, sect. 7(], [7]. 9or the and de-ned in (4.(), this iteration converges exactly at the third step: (1) (1) (1) Expressions for one GMRES(4) cycle can likewise be derived using elementary calcu, lus. The updated residual takes the form (2) +2 (2) , where )=(? αz βz is a quadratic whose coefficients (2) ) and (2) ) are given by (2)T AAr (2) )( (2)T AAr (2) (2)T Ar (2) )( (2)T AAr (2) (2)T Ar (2) )( (2)T AAr (2) (2)T AAr (2) )( (2)T AAr (2) (2)T Ar (2) )( (2)T AAr (2) (2)T AAr (2) )( (2)T Ar (2) (2)T Ar (2) )( (2)T AAr (2) (2)T AAr (2) )( (2)T AAr (2)
Page 3
RESTARTED GMRES 26, Fig.1 Convergence curves for GMRES (1) and GMRES (2) applied to (2.1) with Executing GMRES(4) on the matrix and right,hand side (4.() reveals (2) (2) (2) 48 41 47 55 (2) (44 8( (68 (A4 TheinferiorityofGMRES(4)continueswellbeyondthefourthiteration. 9orexample, (2) 5 ,.-.)AAA21,,255-2... 1, ,.-.)5,20AAA5A11,... 15 ,.-.)01)12.1-)5--... 2, ,.-.)01),55100A).... 25 ,.-.)0151152A5)2)... -, ,.-.)0151A01,1,A.... The entire convergence curve for the -rst 56 iterations is shown in 9igure (, based on performing GMRES(4) in exact arithmetic using Mathematica. The particular value of (and thus ) studied above is exceptional, as it is unusual for GMRES(() to converge exactly in three iterations. Remarkably, though, GMRES(()maintainssuperiorityoverGMRES(4)forawiderangeofinitialresiduals. 9or this matrix , GMRES(4) converges exactly in one cycle for any initial residual with zero in the third component, so we restrict attention to residuals normalized to the form =( ξ,η, () . 9igure 4 indicates that GMRES(4) makes little progress for most such residuals, while GMRES(() converges to high accuracy for the vast maBor, ity of these values. The color in each plot reCects the magnitude of 100 blue indicates satisfactory convergence, while red signals little progress in one hun, dred iterations. (To ensure this data’s -delity, we performed these computations in both double and quadruple precision arithmetic: di0erences between the two were negligible.)
Page 4
262 MARK EMBREE Fig.2 Convergence of GMRES (1) ( left and GMRES (2) ( right for the matrix in (2.1) over a range of initial residuals of the form C( ξ,η, 1) . The color indicates 100 on a loga- rithmic scale: blue regions correspond to initial residuals that converge satisfactorily, while the red regions show residuals that stagnate or converge veryslowly. To gain an appreciation for the dynamics behind 9igure 4, we -rst examine the action of a single GMRES(() step. 9rom (4.4) it is clear that GMRES(() will com, pletely stagnate only when Ar = 6. 9or the matrix speci-ed in (4.() and =( ξ,η, () , this condition reduces to ?5 ?(=6 (4.5) the equation for an oblique ellipse in the ( ξ, ) plane. Eow writing (1) =( ξ,η, () , consider the map (1) (1) +1 that proBects (1) +1 into the ( ξ, ) plane, (1) +1 =( (1) +1 (1) +1 (1) +1 where ( (1) +1 denotes the th entry of (1) +1 , which itself is derived from (1) via (4.4). 9or the present example, we have (1) +1 ?5 ?= ?7 ?(6 ?4 ?7 ?(6 (4.1) .e can classify the -xed points ( ξ, ) satisfying (4.5) by investigating the Facobian of(4.1). 2neofitseigenvaluesisalways(,whiletheothereigenvaluevariesaboveand below ( in magnitude. In the left,hand plot of 9igure 4, we show the stable portion of the ellipse (4.5) in black and the unstable part in white. .e can similarly analyze GMRES(4). This iteration will never progress when, in addition to the stagnation condition for GMRES((), also satis-es AAr =6. 9or the present example, this requirement implies ?4 ?7 ?A ?(=6
Page 5
RESTARTED GMRES 263 the equation for an oblique parabola. This curve intersects the ellipse (4.5) at two points, drawn as dots in the right,hand plot of 9igure 4, the only stagnating residuals ξ,η, () forGMRES(4). .ecananalyzetheirstabilityasdoneaboveforGMRES((). The proBected map for this iteration, (2) (2) +2 , takes the form (2) +2 ?1 ?= ?1 ?= (4.7) Analyzing the Facobian for this GMRES(4) mapat the pair of -xed points, we -nd one to be unstable (shown in black in the right,hand plot of 9igure 4) while the other is stable (shown in white). This stable -xed point is an attractor for stagnating residuals. .e return brieCy to the initial residual =(4 () . After the -rst few itera, tions, the angle between (2) and the -xed vector steadily converges to zero at the rate 6 A174 ... suggested by the Facobian’s dominant eigenvalue. .e conclude with high con-dence that GMRES(4) never converges for this initial residual. (If one cycle of GMRES( ) produces a residual parallel to , then either or Thus a residual can’t remain -xed in the -nite ( ξ, ) plane, but still converge to .) 3. SecondExample. The matrix in (4.() is nondiagonalizable, and one might betemptedtoblameitssurprisingconvergencebehavioronthisfact. Todemonstrate that nondiagonalizablity is not an essential requirement, we exhibit a diagonalizable matrix with eigenvalues for which restarted GMRES also produces extreme behavior. Take (4 641 665 (5.() with . Again, we construct the -rst few residuals. 9or GMRES((), (1) (1) (1) while GMRES(4) yields (2) (2) (2) (7 (4 (2) A7 (4 (4 48 9igure 5 illustrates the convergence curve for 56 iterations, again computed using exact arithmetic. As with the -rst example, we investigate the performance of restarted GMRES for a range of =( ξ,η, () , shown in 9igure 1. GMRES(4) performs as before, mak, ing little progress for virtually all the residuals shown: there are two -xed points, one stable and the other not. The GMRES(() phase plane, on the other hand, contains fascinatingstructure. .hethertheiterationconvergesorstagnatesappearssensitively dependent on the initial residual, highlighted in 9igure 7. Red regions—indicating
Page 6
264 MARK EMBREE Fig.3 Convergence curves for GMRES (1) and GMRES (2) applied to (-.1) with Fig.4 Comparison of GMRES (1) ( left and GMRES (2) ( right as in Figure , but for the matrix in (-.1) Double and quadruple precision computations differ notablyonlyat the boundaries between convergence and stagnation. stagnation—are drawn towards the arc of stable -xed points (shown in black in 9ig, ure 1). The boundary between stagnating and converging residuals exhibits hallmark fractal qualities, as one might establish by analyzing the proBected GMRES(() map (1) +1 ?4 (( ?4 (6 ?8 ?7 (4 ?4 ?4 ?46
Page 7
RESTARTED GMRES 265 Fig.5 Close-up of the left-hand plot of Figure . The white curve denotes unstable fixed points of the map (1) +1 4. ClosingRemarks. Examples like (4.() and (5.() abound. .e discovered these byvaryingthethreeupperrightentriesof andthe-rsttwocomponentsof among the integers from 7 to 7, while -xing all other entries. Even among such a restricted set, numerous other examples exhibit similar behavior. 2ur contrived examples are extreme models of a phenomenon experienced in practical computations. 9or the convection,di0usion discretization described in [1], GMRES(() or GMRES(7) can outperform GMRES(46) on moderately re-ned grids. The optimal choice of restart parameter depends on the problem. Since, on average, one GMRES( ) iteration is cheaper than one GMRES( ) iteration when [7, A], the potential advantage of smaller restarts is especially acute. There is much still to understand about this unusual restarting behavior. ;ow common is sensitive dependence on , especially for larger values of the restart pa, rameter3 .hat characterizes susceptible matrices3 2ne hopes an improved GMRES convergence theory will identify better practical guidelines for choosing the restart parameter. 2ne also wonders if related algorithms, including GMRES restarted with an augmented subspace [4] and HiCGSTAH( ) [=], exhibit similarly unusual behavior. Such e0ects might also arise from automatic shift,selection strategies in the restarted Arnoldi algorithm for calculating eigenvalues [(6]. NoteAdded in Proof. Since submitting this work for publication, several other interesting examples have emerged. The simplest one, which Fohn Sabino and I have studied, involves the matrix 6(
Page 8
266 MARK EMBREE Consider the two sequences of vectors =0 and =0 , where ?() /j (4 ?() (4 () These sequences interlace one another, yet if , GMRES(() converges exactly in ?( iterations, while one can prove that if , GMRES(() must stagnate. Even for this remarkably simple problem, one obtains a convergence phase plane with striping roughly akin to 9igure 7. GMRES stagnation for 4,by,4 matrices has been studied in detail by Iavorin, 2’Jeary, and Elman [(4]. Acknowledgments. I am indebted to Eick Trefethen and Andy .athen for ad, vice that both inCuenced the course of this research and improved its presentation. I also thank ;enk van der Vorst for his helpful comments. REFEREN.ES D1E E.deSturler All good things come to an end, or the convergence of GMRES ), talk at SFAMGs 05th Anniversary Meeting, Stanford 5niversity, 111.. D2E M. Eiermann, O. G. Ernst, and O. Schneider Analysis of acceleration strategies for restarted minimal residual methods , H. 7omp. Appl. Math., 12- (2,,,), pp. 2)1I212. D-E D.K.FaddeevandV.N.Faddeeva Computational Methods of Linear Algebra , Freeman, San Francisco, 11)-. 4ranslated by Robert 7. Williams. D0E B.Fischer,A.Ramage,D.Silvester,andA.J.Wathen On parameter choice and itera- tive convergence of stabilised advection-diffusion problems , 7omput. Methods Appl. Mech. Engrg., 1.1 (1111), pp. 1A5I2,2. D5E Y. Huang and H. van der Vorst Some Observations on the Convergence Behavior of GMRES , 4ech. Rep. A1I,1, Faculty of 4echnical Mathematics and Fnformatics, 2elft 5ni- versity of 4echnology, 4he /etherlands, 11A1. D)E W.A.Joubert On the convergence behavior of the restarted GMRES algorithm for solving nonsymmetric linear systems , /umer. ;inear Algebra Appl., 1 (1110), pp. 02.I00.. D.E M.A.Krasnosel’ski ıandS.G.Kre ın An iteration p rocess with minimal residuals , Mat. Sb. (/.S.), -1 (1152), pp. -15I--0 (in Russian). DAE Y.SaadandM.H.Schultz GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems , SFAM H. Sci. Statist. 7omput., . (11A)), pp. A5)IA)1. D1E G.L.G.Sleij,enandD.R.Fokkema BiCGSTAB for linear equations involving unsym- metric matrices with complex spectrum ,Electron.4rans./umer.Anal.,1(111-),pp.11I-2. D1,E D.C.Sorensen ,mplicit application of polynomial filters in a --step Arnoldi method , SFAM H. MatrixAnal. Appl., 1- (1112), pp. -5.I-A5. D11E L...Watson personal communication , 2,,1. D12E I.Zavorin,D.P.O’Leary,andH.Elman Stagnation of GMRES , 4ech. Rep. 7S-4R-001), 5niversity of Maryland 7omputer Science 2epartment, 2,,1.

About DocSlides
DocSlides allows users to easily upload and share presentations, PDF documents, and images.Share your documents with the world , watch,share and upload any time you want. How can you benefit from using DocSlides? DocSlides consists documents from individuals and organizations on topics ranging from technology and business to travel, health, and education. Find and search for what interests you, and learn from people and more. You can also download DocSlides to read or reference later.