Solution and Optimization methods for Large Scale Problems
Overview on iterative solution methods techniques for large scale systems. Parallel solution of large linear and non-linear system. Hierarchical methods for linear systems and for minimization problems. Decomposition methods. Beyond first order methods in machine learning
Large scale systems and large scale optimization problems are of central importance in computational science, optimization, and machine learning. Since standard solution and minimization methods in general do not scale optimally, alternative solution strategies have been developed during the last decades. In particular hierarchical solution strategies and parallel strategies have been developed. Prominent examples are multilevel or domain decomposition methods, originally developed for linear elliptic problems. We start from basic iterative methods, and then consider Krylov-space methods and eventually subspace correction methods for linear and non-linear problems,. We will discuss multilevel optimization methods such as MG/OPT, (recursive) trust-region methods (RMTR) and hierarchical minimization methods for machine learning, including variance reduction methods.
Lecture, reading, self study, hands-on implementation, discussion, tutorial, written weekly assignments
There will be a midterm, either as larger project-like assignment or as an written exam. The final exam will be written. The written weekly assignments will also count for the final grade.
- A Multigrid Tutorial; William L. Briggs, Van Emden Henson, and Steve F. McCormick; Second Edition, SIAM, 2000 (book home page), ISBN 0-89871-4621.
- Multigrid Methods and Applications; Wolfgang Hackbusch; Springer, 1985.
- An Introduction to Multigrid Methods; Pieter Wesseling; Corrected Reprint. Philadelphia: R.T. Edwards, Inc., 2004. ISBN 1-930217-08-0.
- Matrix computations; Gene H. Golub and Charles F. Van Loan.
- Domain Decomposition Methods; Toselli, Widlund
- Nocedal Wright, Numerical Optimisation;
- Trust-Region Methods; Conn Gould Toint.
- Practical Methods of Optimisation; R. Fletcher.
- Numerical Optimisation, Series: Springer Series in Operations Research and Financial Engineering, Nocedal, Jorge, Wright, Stephen 2nd ed., 2006, XXII, 664 p. 85 illus., www.springer.com/mathematics/book/978-0-387-30303-1
- Deep Learning, Ian Goodfellow and Yoshua Bengio and Aaron Courville, MIT Press, http://www.deeplearningbook.org2016