Albert Berahas, Northwestern University 

Título: Fast, Adaptive and Scalable Nonlinear Optimization Algorithms: From Machine Learning to Derivative-Free Optimization

Resumen:[+]

The need for solving even larger nonlinear optimization problems continues unabated. Such problems are ubiquitous, arising in areas such as machine learning, engineering design, and statistics. The focus of my research has been on designing and analyzing scalable nonlinear optimization algorithms that have sound theoretical properties. Specifically, I have explored four subfields of nonlinear optimization: (i) general nonlinear optimization algorithms, (ii) optimization for machine learning, (iii) derivative-free optimization, and (iv) distributed optimization. In this talk, I will present a few topics that I have worked on in the last couple of years. In the first part of the talk, I will describe a new class of quasi-Newton methods developed for training deep neural networks. The key idea is to leverage the fact that quasi-Newton methods can incorporate second-order information at a reasonable cost, and to enhance this information using more recent and local information via sampling. As a result, these methods outperform their classical quasi-Newton variants and are competitive with the state-of-the-art first-order methods on deep learning tasks. In the second part of the talk, I will present an adaptive finite difference quasi-Newton method for the derivative-free optimization of noisy functions. Problems of this form arise in a plethora of science, engineering, and artificial intelligence applications such as simulation optimization and reinforcement learning. The method takes advantage of the scalability and power of BFGS updating, employs an adaptive procedure for choosing the differencing interval based on an estimate of the noise, has sound theoretical properties, and is competitive with the state-of-the-art methods in practice.