The developmentof highly paralleldirect solversfor large, sparse linear systemsof equations (e.g. for finite elementor finite differencemodels)is laggingbehind progressin paralleldirect solversfor dense matricesand iterativemethodsfor sparse matrices. We describea massivelyparallel(MP) multifrontal solverfor the directsolutionof large sparse linear systems,such as those routinelyencounteredin finite element structuralanalysis, in an effort to address concerns about the viabilityof scalable, MP

more »

... irect methodsfor sparse systemeand enhance the softwarebase for MP applications.Performanceresultsare presentedand futuredirectionsare outlinedfor researchand developmenteffortsin parallelmultifrontal and related solvers. In particular,parallelefficlenciesof 25% on 1024 nCUBE 2 nodes and 36% on 64 Intel iPSC860 nodes have been demonstrated,and parallel efficienciesof 60-85% are expected when a severe load imbalanceis overcome by staticmapping and dynamic load balance techniquespreviously developedforotherparallelsolvers and applicationcodes. Fortranproductioncode for generatingsimulatedradar images, again on the first-generationnCUBE/ten hypercube (Gustafsonet al. 1989). As dozens of major applicationshave moved to MP systemsthe questionhasalways remained: How willwe approachthe next application,howmuch effortwill be required,and what will the resultingparallel algorithms,code and performance look like? Of particularconcern to the author: the development of highly paralleldirect solvers for large, sparse linear systems of equations (e.g. for finite element or finite differencemodels),whichhas been laggingbehind progressin paralleldirectsolvers fordense matrices * Thisworksupfx>,'ted bytheAppliedMathematical SciencesProgram, U.S. Department of Energy, Officeof Energy Research.lt wasdoneat SandiaNational Laboratories, operatedfortheU.S. Department of Energyundercontract number DE-AC04-76DP00789. _, T_ _,_ (e.g. Heath et al. 1991 and references therein, Sears 1990) and parallel iterative methodsfor sparse matrices. This paperdescribesan approachto this issueand somepreliminaryresultsfor a multifrontal solver,a direct solver routinelyused in structuralanalysisand other finite element applicationcodes (e.g., in fluid mechanics,combustion, etc.). A massivelyparallel multifrontal solver is beingdeveloped in an effortto addressconcems aboutthe viabilityof MP direct methodsfor sparse systems,as well as to improvethe softwarebase for MP applications. The present investigationuses two MIMD (Multiple Instruction Multiple Data) parallel computing systems at Sandia. One is an nCUBE 2 hypercubewith 1024 processor nodes and 4 MBytes of local memory per node. This system is of particular interest becauselt enables us to investigatescalabilityof parallelalgorithmsintothe thousand-processorregime. Aggregatesustained performanceof this system is 1 to 2 GFLOPS for many parallelapplications,even thoughthe individualprocessorsare proprietary, 20 MHz scalar processorsand, therefore,are relativelyslow. The system is well balanced in terms of its computationand communicationcapabilitiesand it featuresa mature,robustand relativelyflexibleparallel debugger. Insofaras we have had severalparallelapplications at Sandia give performanceon an Inteli860 processor node in the range of five to six timesthatof an nCUBE 2 processor,the latteralso helps us to establisha realisticexpectation of what sustainedperformancewe might hope for the multifrontal solver to achieve on an iPSC 860 system. The second system is an Intel iPSC860 hypercubewith 64 nodes. Sixteen of the nodes have 32 MBytes of local memory apiece, while the remaining nodes have 8 Mbytes apiece. Although not massively parallel, the superscalar i860 processor nodes provide a testbed for parallel algorithm performanceon superscalarMP architecturesof the near future. The availabilityof 32 MBytes nodes is noteworthy,because lt enables greater study of the scalabiiityof the parallel solver as a function of nroblemsize than is otherwisepossibleon thetwo hypercubes. The following sections discuss frontal and multifrontal solvers, their history as parallel solvers, a massivelyparallelimplementation,results,and futuredirections. FRONTAL AND MULTIFRONTAL METHODS A fundamental issue is the merit of direct methods relativeto iterativemethods. Direct methodsare memory intensive,compute intensive, and, in general, have more challengingserial bottlenecks than Iterative methods, so it is not clear that they are worth the effort requiredto parallelizethem. lt has not been establishedthat parallel direct methodsfor large sparse matricescan even achieve high parallel efficiencyon thousandsof processors.That is,it is unclearthat thisclassof numericalmethodis scalable when appliedto sparse linearsystems;in fact, someimplementationsare knownto scalepoorly. Conversely,directmethodsare stillused in many applicationsbecause: (1) they are often more robust than preconditioned iterative methods, (2) suitable preconditioners have not emerged for iterative methods applied to some applications,and (3) they are applicableas a coarse mesh solverfor multigrid methods. In addition,scalable parallelizationtechniquesdeveloped for sparsedirect methodscould be used, in part, in some parallelpreconditionersand in hybriddirect-iterativemethods. Frontalsolversand theiroffspring,multifrontal solvers(Duffand Reid 1982), are the methodsof choice for finite element analysis in several application areas, includingsolid mechanics,thermal analysis,and fluid mechanics. An excellent review of these methods is provided by Liu (1992), so only a brief