This paper sketches the main research developments in the area of iterative methods for solving linear systems during the 20th century. Although iterative methods for solving linear systems nd their origin in the early nineteenth century work by Gauss, the eld has seen an explosion of activity spurred by demand due to extraordinary technological advances in engineering and sciences. The past ve decades have been particularly rich in new developments, ending with the availability of large

more »

... of specialized algorithms for solving the very large problems which arise in scienti c and industrial computational models. As in any other scienti c area, research in iterative methods has been a journey characterized by a c hain of contributions building on each other. It is the aim of this paper not only to sketch the most signi cant of these contributions during the past century, b u t a l s o to relate them to one another. 1 elimination approach. In this paper, we w i l l s k etch the developments and progress that has taken place in the twentieth century with respect to iterative methods alone. As will be clear, this sub eld could not evolve in isolation, and the distinction between iterative methods and Gaussian elimination methods is sometimes arti cial -and overlap between the two methodologies is signi cant i n m a n y instances. Nevertheless, each of the two has its own dynamics and it may b e o f i n terest to follow one of them more closely. It is likely that future researchers in numerical methods will regard the decade just passed as the beginning of an era in which iterative methods for solving large linear systems of equations started gaining considerable acceptance in real-life industrial applications. In looking at past literature, it is interesting to observe that iterative and direct methods have often been in competition for solving large systems that arise in applications. A particular discovery will promote a given method from one camp only to see another discovery promote a competing method from the other camp. For example, the 50s and 60s saw an enormous interest in relaxation-type methods -prompted by the studies on optimal relaxation and the work by Y oung, Varga, Southwell, Frankel and others. A little later, sparse direct methods appeared that were very competitive -both from the point of view of robustness and computational cost. To this day, there are still applications dominated by direct solvers and others dominated by iterative solvers. Because of the high memory requirement of direct solvers, it was sometimes thought that these would eventually be replaced by iterative solvers, in all applications. However, the superior robustness of direct solvers prevented this. As computers have become faster, very large problems are routinely solved by methods from both camps. Iterative methods were, even halfway in the twentieth century, not always viewed as promising. For instance, Bodewig 23, p.153 , in 1956, mentioned the following drawbacks of iterative methods: nearly always too slow except when the matrix approaches a diagonal matrix, for most problems they do not converge at all, they cannot easily be m e chanised 1 and so they are more appropriate for computing by hand than with machines, a n d do not take advantage of the situation when the equations are symmetric. The only potential advantage seen was the observation that Rounding errors do not accumulate, they are restricted to the last operation. It is noteworthy that Lanczos' method was classi ed as a direct method in 1956. The penetration of iterative solvers into applications has been a slow process that is still ongoing. At the time of this writing for example, there are applications in structural engineering as well as in circuit simulation, which are dominated by direct solvers. This review will attempt to highlight the main developments in iterative methods over the past century. It is clear that a few pages cannot cover an exhaustive s u r v ey of 100 years of rich d e v elopments. Therefore, we will emphasize the ideas that were successful and had a signi cant impact. Among the sources we used for our short survey, w e w ould like t o m e n tion just a few that are notable for their completeness or for representing the thinking of a particular era. The books by V arga 188 and Young 205 g i v e a complete treatise of iterative methods as they were used in the 60s and 70s. Varga's book has several excellent historical references. These two masterpieces remained the handbooks used by academics and practitioners alike for three decades. Householder's book 102 c o n tains a fairly good overview of iterative methods -speci cally oriented towards projection methods. Among the surveys we note the outstanding booklet published by the National Bureau of Standards in 1959 which c o n tains articles by Rutishauser 150 , Engeli 68 and Stiefel 170 . Later Birkho 21 , who supervised David Young's PhD thesis in the late 1940s, wrote an excellent historical perspective on the use of iterative methods as he experienced them himself from 1930 to 1980. The more recent literature includes the books by Axelsson 7 , Brezinski 29 , Greenbaum 88 , Hackbusch 97 , and Saad 157 , each o f w h i c h h a s a slightly di erent perspective and emphasis. 1 This remark was removed from the second edition in 1959; instead Bodewig included a small section on methods for automatic machines 24, Ch.9 . The earlier remark was not as puzzling as it may seem now, in view of the very small memories of the available electronic computers at the time. This made it necessary to store intermediate data on punched cards. It required a regular ow of the computational process, making it cumbersome to include techniques with row interchanging. 2