Evaluation Complexity Of Algorithms For Nonconvex Optimization

A popular way to assess the “effort” needed to solve a problem is to count how many evaluations of the problem functions (and their derivatives) are required. In many cases, this is often the dominating computational cost. Given an optimization problem satisfying reasonable assumptions—and given access to problem-function values and derivatives of various degrees—how many evaluations might be required to approximately solve the problem? Evaluation Complexity of Algorithms for Nonconvex Optimization: Theory, Computation, and Perspectives addresses this question for nonconvex optimization problems, those that may have local minimizers and appear most often in practice. This is the first book on complexity to cover topics such as composite and constrained optimization, derivative-free optimization, subproblem solution, and optimal (lower and sharpness) bounds for nonconvex problems. It is also the first to address the disadvantages of traditional optimality measures and propose useful surrogates leading to algorithms that compute approximate high-order critical points, and to compare traditional and new methods, highlighting the advantages of the latter from a complexity point of view. This is the go-to book for those interested in solving nonconvex optimization problems. It is suitable for advanced undergraduate and graduate students in courses on advanced numerical analysis, data science, numerical optimization, and approximation theory.

This book details algorithms for large-scale unconstrained and bound constrained optimization. It shows optimization techniques from a conjugate gradient algorithm perspective as well as methods of shortest residuals, which have been developed by the author.

Everything should be made as simple as possible, but not simpler. (Albert Einstein, Readers Digest, 1977) The modern practice of creating technical systems and technological processes of high effi.ciency besides the employment of new principles, new materials, new physical effects and other new solutions ( which is very traditional and plays the key role in the selection of the general structure of the object to be designed) also includes the choice of the best combination for the set of parameters (geometrical sizes, electrical and strength characteristics, etc.) concretizing this general structure, because the Variation of these parameters ( with the structure or linkage being already set defined) can essentially affect the objective performance indexes. The mathematical tools for choosing these best combinations are exactly what is this book about. With the advent of computers and the computer-aided design the pro bations of the selected variants are usually performed not for the real examples ( this may require some very expensive building of sample op tions and of the special installations to test them ), but by the analysis of the corresponding mathematical models. The sophistication of the mathematical models for the objects to be designed, which is the natu ral consequence of the raising complexity of these objects, greatly com plicates the objective performance analysis. Today, the main (and very often the only) available instrument for such an analysis is computer aided simulation of an object's behavior, based on numerical experiments with its mathematical model.

This concise, self-contained volume introduces convex analysis and optimization algorithms, with an emphasis on bridging the two areas. It explores cutting-edge algorithms—such as the proximal gradient, Douglas–Rachford, Peaceman–Rachford, and FISTA—that have applications in machine learning, signal processing, image reconstruction, and other fields. An Introduction to Convexity, Optimization, and Algorithms contains algorithms illustrated by Julia examples and more than 200 exercises that enhance the reader’s understanding of the topic. Clear explanations and step-by-step algorithmic descriptions facilitate self-study for individuals looking to enhance their expertise in convex analysis and optimization. Designed for courses in convex analysis, numerical optimization, and related subjects, this volume is intended for undergraduate and graduate students in mathematics, computer science, and engineering. Its concise length makes it ideal for a one-semester course. Researchers and professionals in applied areas, such as data science and machine learning, will find insights relevant to their work.

Built on the framework of the successful first edition, this book serves as a modern introduction to the field of optimization. The author’s objective is to provide the foundations of theory and algorithms of nonlinear optimization as well as to present a variety of applications from diverse areas of applied sciences. Introduction to Nonlinear Optimization gradually yet rigorously builds connections between theory, algorithms, applications, and actual implementation. The book contains several topics not typically included in optimization books, such as optimality conditions in sparsity constrained optimization, hidden convexity, and total least squares. Readers will discover a wide array of applications such as circle fitting, Chebyshev center, the Fermat–Weber problem, denoising, clustering, total least squares, and orthogonal regression. These applications are studied both theoretically and algorithmically, illustrating concepts such as duality. Python and MATLAB programs are used to show how the theory can be implemented. The extremely popular CVX toolbox (MATLAB) and CVXPY module (Python) are described and used. More than 250 theoretical, algorithmic, and numerical exercises enhance the reader's understanding of the topics. (More than 70 of the exercises provide detailed solutions, and many others are provided with final answers.) The theoretical and algorithmic topics are illustrated by Python and MATLAB examples. This book is intended for graduate or advanced undergraduate students in mathematics, computer science, electrical engineering, and potentially other engineering disciplines.

The only book offering solved exercises for integer and combinatorial optimization, this book contains 102 classroom tested problems of varying scope and difficulty chosen from a plethora of topics and applications. It has an associated website containing additional problems, lecture notes, and suggested readings. Topics covered include modeling capabilities of integer variables, the Branch-and-Bound method, cutting planes, network optimization models, shortest path problems, optimum tree problems, maximal cardinality matching problems, matching-covering duality, symmetric and asymmetric TSP, 2-matching and 1-tree relaxations, VRP formulations, and dynamic programming. Problems and Solutions for Integer and Combinatorial Optimization: Building Skills in Discrete Optimization is meant for undergraduate and beginning graduate students in mathematics, computer science, and engineering to use for self-study and for instructors to use in conjunction with other course material and when teaching courses in discrete optimization.

Moment and polynomial optimization is an active research field used to solve difficult questions in many areas, including global optimization, tensor computation, saddle points, Nash equilibrium, and bilevel programs, and it has many applications. The author synthesizes current research and applications, providing a systematic introduction to theory and methods, a comprehensive approach for extracting optimizers and solving truncated moment problems, and a creative methodology for using optimality conditions to construct tight Moment-SOS relaxations. This book is intended for applied mathematicians, engineers, and researchers entering the field. It can be used as a textbook for graduate students in courses on convex optimization, polynomial optimization, and matrix and tensor optimization.

Recent results on non-convex multi-objective optimization problems and methods are presented in this book, with particular attention to expensive black-box objective functions. Multi-objective optimization methods facilitate designers, engineers, and researchers to make decisions on appropriate trade-offs between various conflicting goals. A variety of deterministic and stochastic multi-objective optimization methods are developed in this book. Beginning with basic concepts and a review of non-convex single-objective optimization problems; this book moves on to cover multi-objective branch and bound algorithms, worst-case optimal algorithms (for Lipschitz functions and bi-objective problems), statistical models based algorithms, and probabilistic branch and bound approach. Detailed descriptions of new algorithms for non-convex multi-objective optimization, their theoretical substantiation, and examples for practical applications to the cell formation problem in manufacturing engineering, the process design in chemical engineering, and business process management are included to aide researchers and graduate students in mathematics, computer science, engineering, economics, and business management.