Optimization and Variational Analysis
45 papers on this topic
Plug-and-Play image reconstruction is a convergent regularization method
Non-uniqueness and instability are characteristic features of image reconstruction processes. As a result, it is necessary to develop regularization methods that can be used to compute reliable...
Differentiable Convex Optimization Layers
Recent work has shown how to embed differentiable optimization problems (that is, problems whose solutions can be backpropagated through) as layers within deep learning architectures. This method...
Time-Varying Convex Optimization: Time-Structured Algorithms and Applications
Optimization underpins many of the challenges that science and technology face on a daily basis. Recent years have witnessed a major shift from traditional optimization paradigms grounded on batch...
Randomized Forward Mode of Automatic Differentiation For Optimization Algorithms
We present a randomized forward mode gradient (RFG) as an alternative to backpropagation. RFG is a random estimator for the gradient that is constructed based on the directional derivative along a...
Recent Theoretical Advances in Non-Convex Optimization
Motivated by recent increased interest in optimization algorithms for non-convex optimization in application to training deep neural networks and other optimization problems in data analysis, we...
A Sequential Quadratic Programming Approach to the Solution of Open-Loop Generalized Nash Equilibria
Dynamic games can be an effective approach to modeling interactive behavior between multiple non-cooperative agents and they provide a theoretical framework for simultaneous prediction and control...
Randomized block subsampling Kaczmarz-Motzkin method
By introducing a subsampling strategy, we propose a randomized block Kaczmarz-Motzkin method for solving linear systems. Such strategy not only determines the block size, but also combines and...
Stochastic Inexact Augmented Lagrangian Method for Nonconvex Expectation Constrained Optimization
Many real-world problems not only have complicated nonconvex functional constraints but also use a large number of data points. This motivates the design of efficient stochastic methods on...
Provably Faster Algorithms for Bilevel Optimization
Bilevel optimization has been widely applied in many important machine learning applications such as hyperparameter optimization and meta-learning. Recently, several momentum-based algorithms have...
Better Regularization for Sequential Decision Spaces: Fast Convergence Rates for Nash, Correlated, and Team Equilibria
We study the application of iterative first-order methods to the problem of computing equilibria of large-scale two-player extensive-form games. First-order methods must typically be instantiated...
Neural Fixed-Point Acceleration for Convex Optimization
Fixed-point iterations are at the heart of numerical computing and are often a computational bottleneck in real-time applications that typically need a fast solution of moderate accuracy. We...
A General Descent Aggregation Framework for Gradient-based Bi-level Optimization
In recent years, a variety of gradient-based methods have been developed to solve Bi-Level Optimization (BLO) problems in machine learning and computer vision areas. However, the theoretical...
Accelerated Primal-Dual Methods for Convex-Strongly-Concave Saddle Point Problems
We investigate a primal-dual (PD) method for the saddle point problem (SPP) that uses a linear approximation of the primal function instead of the standard proximal step, resulting in a linearized...
Modified Gauss-Newton Algorithms under Noise
Gauss-Newton methods and their stochastic version have been widely used in machine learning and signal processing. Their nonsmooth counterparts, modified Gauss-Newton or prox-linear algorithms,...
Improved Bilevel Model: Fast and Optimal Algorithm with Theoretical Guarantee
Due to the hierarchical structure of many machine learning problems, bilevel programming is becoming more and more important recently, however, the complicated correlation between the inner and...
BilevelJuMP.jl: Modeling and Solving Bilevel Optimization in Julia
In this paper we present BilevelJuMP, a new Julia package to support bilevel optimization within the JuMP framework. The package is a Julia library that enables the user to describe both upper and...
Linear optimization over homogeneous matrix cones
A convex cone is homogeneous if its automorphism group acts transitively on the interior of the cone, i.e., for every pair of points in the interior of the cone, there exists a cone automorphism...
An Equivalent Circuit Workflow for Unconstrained Optimization
We introduce a new workflow for unconstrained optimization whereby objective functions are mapped onto a physical domain to more easily design algorithms that are robust to hyperparameters and...