Stochastic Optimization Methods

This book examines optimization problems that in practice involve random model parameters. It details the computation of robust optimal solutions, i.e., optimal solutions that are insensitive with respect to random parameter variations, where appropriate deterministic substitute problems are needed. Based on the probability distribution of the random data and using decision theoretical concepts, optimization problems under stochastic uncertainty are converted into appropriate deterministic substitute problems. Due to the probabilities and expectations involved, the book also shows how to apply approximative solution techniques. Several deterministic and stochastic approximation methods are provided: Taylor expansion methods, regression and response surface methods (RSM), probability inequalities, multiple linearization of survival/failure domains, discretization methods, convex approximation/deterministic descent directions/efficient points, stochastic approximation and gradient procedures and differentiation formulas for probabilities and expectations. In the third edition, this book further develops stochastic optimization methods. In particular, it now shows how to apply stochastic optimization methods to the approximate solution of important concrete problems arising in engineering, economics and operations research.

This book covers not only foundational materials but also the most recent progresses made during the past few years on the area of machine learning algorithms. In spite of the intensive research and development in this area, there does not exist a systematic treatment to introduce the fundamental concepts and recent progresses on machine learning algorithms, especially on those based on stochastic optimization methods, randomized algorithms, nonconvex optimization, distributed and online learning, and projection free methods. This book will benefit the broad audience in the area of machine learning, artificial intelligence and mathematical programming community by presenting these recent developments in a tutorial style, starting from the basic building blocks to the most carefully designed and complicated algorithms for machine learning.

This book addresses stochastic optimization procedures in a broad manner. The first part offers an overview of relevant optimization philosophies; the second deals with benchmark problems in depth, by applying a selection of optimization procedures. Written primarily with scientists and students from the physical and engineering sciences in mind, this book addresses a larger community of all who wish to learn about stochastic optimization techniques and how to use them.

Among all aspects of engineering, design is the most important step in developing a new product. A systematic approach to managing design issues can only be accomplished by applying mathematical optimization methods. Furthermore, due to the practical issues in engineering problems, there are limitations in using traditional methods. As such, stochastic optimization methods such as differential evolution, simulated annealing, and genetic algorithms are preferable in finding solutions in design optimization problems. This book reviews mechanical engineering design optimization using stochastic methods. It introduces students and design engineers to practical aspects of complicated mathematical optimization procedures, and outlines steps for wide range of selected engineering design problems. It shows how engineering structures are systematically designed. Many new engineering design applications based on stochastic optimization techniques in automotive, energy, military, naval, manufacturing process and fluids-heat transfer, are described in the book. For each design optimization problem described, background is provided for understanding the solutions. There are very few books on optimization that include engineering applications. They cover limited applications, and that too of well-known design problems of advanced and niche nature. Common problems are hardly addressed. Thus, the subject has remained fairly theoretical. To overcome this, each chapter in this book is contributed by at least one academic and one industrial expert researcher.

Advancements in the technology and availability of data sources have led to the `Big Data' era. Working with large data offers the potential to uncover more fine-grained patterns and take timely and accurate decisions, but it also creates a lot of challenges such as slow training and scalability of machine learning models. One of the major challenges in machine learning is to develop efficient and scalable learning algorithms, i.e., optimization techniques to solve large scale learning problems. Stochastic Optimization for Large-scale Machine Learning identifies different areas of improvement and recent research directions to tackle the challenge. Developed optimisation techniques are also explored to improve machine learning algorithms based on data access and on first and second order optimisation methods. Key Features: Bridges machine learning and Optimisation. Bridges theory and practice in machine learning. Identifies key research areas and recent research directions to solve large-scale machine learning problems. Develops optimisation techniques to improve machine learning algorithms for big data problems. The book will be a valuable reference to practitioners and researchers as well as students in the field of machine learning.

This volume presents a collection of contributions dedicated to applied problems in the financial and energy sectors that have been formulated and solved in a stochastic optimization framework. The invited authors represent a group of scientists and practitioners, who cooperated in recent years to facilitate the growing penetration of stochastic programming techniques in real-world applications, inducing a significant advance over a large spectrum of complex decision problems. After the recent widespread liberalization of the energy sector in Europe and the unprecedented growth of energy prices in international commodity markets, we have witnessed a significant convergence of strategic decision problems in the energy and financial sectors. This has often resulted in common open issues and has induced a remarkable effort by the industrial and scientific communities to facilitate the adoption of advanced analytical and decision tools. The main concerns of the financial community over the last decade have suddenly penetrated the energy sector inducing a remarkable scientific and practical effort to address previously unforeseeable management problems. Stochastic Optimization Methods in Finance and Energy: New Financial Products and Energy Markets Strategies aims to include in a unified framework for the first time an extensive set of contributions related to real-world applied problems in finance and energy, leading to a common methodological approach and in many cases having similar underlying economic and financial implications. Part 1 of the book presents 6 chapters related to financial applications; Part 2 presents 7 chapters on energy applications; and Part 3 presents 5 chapters devoted to specific theoretical and computational issues.

Optimization problems arising in practice involve random parameters. For the computation of robust optimal solutions, i.e., optimal solutions being insensitive with respect to random parameter variations, deterministic substitute problems are needed. Based on the distribution of the random data, and using decision theoretical concepts, optimization problems under stochastic uncertainty are converted into deterministic substitute problems. Due to the occurring probabilities and expectations, approximative solution techniques must be applied. Deterministic and stochastic approximation methods and their analytical properties are provided: Taylor expansion, regression and response surface methods, probability inequalities, First Order Reliability Methods, convex approximation/deterministic descent directions/efficient points, stochastic approximation methods, differentiation of probability and mean value functions. Convergence results of the resulting iterative solution procedures are given.

This textbook provides an introduction to convex duality for optimization problems in Banach spaces, integration theory, and their application to stochastic programming problems in a static or dynamic setting. It introduces and analyses the main algorithms for stochastic programs, while the theoretical aspects are carefully dealt with. The reader is shown how these tools can be applied to various fields, including approximation theory, semidefinite and second-order cone programming and linear decision rules. This textbook is recommended for students, engineers and researchers who are willing to take a rigorous approach to the mathematics involved in the application of duality theory to optimization with uncertainty.