Scientific Computing

1 Preface

These notes began as a study guide for CS 111 (Scientific Computing) at UC Santa Barbara and expanded as topics arose in research and graduate coursework: eigenvalue algorithms, iterative solvers, spectral methods, Koopman operator theory. The guiding principle throughout is Feynman’s criterion for understanding: you have not truly grasped a concept until you can reconstruct it from first principles. Each section was written when that test failed.

For a rigorous treatment of this material, use the lecture notes of Professor David Bindel (Cornell, CS 6210: Matrix Computations), publicly available on his course website.

Computers were built to do numerical computation. The word ``computer’’ originally meant a person whose job was to evaluate functions by hand. The U.S. Army employed hundreds of them during the Second World War to compute artillery firing tables, a task so large it directly motivated the construction of ENIAC (1945). The machine’s first classified application was thermonuclear weapon simulation, directed by John von Neumann, who also contributed early work on numerical stability. In 1950, Jule Charney used ENIAC to run the first machine weather forecast — Lewis Fry Richardson had estimated the same computation would require 64,000 human computers working in parallel. The Apollo guidance computer navigated to the Moon in real time with 4 kilobytes of memory. The Cooley-Tukey FFT (1965) reduced a core signal processing computation from \(O(n^2)\) to \(O(n \log n)\). The demand for numerical computation at scale was the original motivation for programmable machines.

Almost every numerical algorithm reduces a nonlinear problem to a sequence of linear ones, because linear systems are the only class of problems we understand completely: their solution structure, their sensitivity to perturbations, their reliable algorithms. Newton’s method, finite element discretization, PCA, gradient-based optimization — each is, at its core, a repeated linearization. Linear algebra is the substrate of scientific computing.

Much of the perspective here came from CS 210A: Matrix Analysis, taught by Professor Shivkumar Chandrasekaran in the ECE department at UC Santa Barbara. If you have the chance to take that course, take it.

Aditya Dendukuri \ UC Santa Barbara, 2025