Recently, the internationally renowned academic journal "SIAM Journal on Optimization" published a research paper titled “A Minimization Approach for Minimax Optimization with Coupled Constraints,” authored by Associate Professor Xiaoyin Hu from the Department of Applied Mathematics at the School of Mathematical Sciences, Shenzhen University, in collaboration with Professor Kim-Chuan Toh, Dr. Shiwei Wang from the National University of Singapore, and Assistant Professor Nachuan Xiao from The Chinese University of Hong Kong, Shenzhen.

With the development of adversarial training, generative adversarial networks, two-level optimization, and resource allocation problems, minimax optimization problems have become an important model in machine learning and operations research. However, minimax problems typically require specially designed descent-ascent algorithms, and their convergence theories often need to be re-established. Furthermore, when the inner maximization problem contains constraints coupled with outer variables, the problem structure becomes even more complex, posing significant challenges for existing methods in terms of both theoretical guarantees and algorithmic implementation. To address this challenge, this paper proposes a novel method for transforming non-convex-strongly convex minimax problems into minimization problems. First, using the Lagrangian duality, we equivalently transform a non-convex-strongly convex minimax problem with coupled convex constraints into a non-convex-strongly convex minimax problem with uncoupled constraints. Subsequently, based on forward-backward envelopes, an explicit penalty function minimization problem is formulated, and it is proven that the first-order stable points of this minimization problem correspond exactly to the first-order extremum points of the original minimax problem. Consequently, solving non-convex-strongly convex minimax problems can be transformed into solving a minimization problem, thereby enabling the direct application of mature and efficient minimization algorithms—such as L-BFGS-B and TNC—and their convergence theories to the solution of non-convex-strongly convex minimax problems. Furthermore, this paper proves that the classical gradient descent-ascent method can be interpreted as a descent method for this minimization problem, thereby providing a new analytical perspective on its global convergence. Numerical experiments validate the efficiency of the proposed algorithm.

Full text link:

https://epubs.siam.org/doi/abs/10.1137/24M1689727?af=R

The internationally renowned academic journal "Mathematics of Operations Research" has published online a research paper titled “Developing Lagrangian-Based Methods for Nonsmooth Nonconvex Optimization,” co-authored by Associate Professor Xiaoyin Hu from the School of Mathematical Sciences at Shenzhen University, Assistant Professor Nachuan Xiao from The Chinese University of Hong Kong, Shenzhen, Dr. Kuangyu Ding from Purdue University, and Professor Kimchuan Toh from the National University of Singapore.

Non-convex, non-smooth constrained optimization is a significant research topic in operations research and machine learning, widely encountered in scenarios such as deep neural network training, signal processing, and engineering optimization. In particular, deep neural networks often employ ReLU as the activation function, and their objective functions are typically non-smooth and non-Clarke regular. Additionally, constraints on decision variables in practical problems introduce complex constraints to non-smooth optimization problems. The convergence theory of existing Lagrangian class methods largely relies on structural conditions such as the smoothness or weak convexity of the objective function and constraint mappings. When the objective function and constraints lack Clarke regularity, designing Lagrangian class methods with convergence guarantees becomes extremely challenging.

To address this issue, this paper proposes a unified Lagrangian-class method framework for embedding stochastic subgradient methods into constrained optimization algorithms. In each iteration, this framework performs a single stochastic subgradient-class update on the original variables and handles constraint information through modified dual updates. Consequently, common stochastic optimization methods such as proximal stochastic gradient descent, proximal momentum stochastic gradient descent, and proximal ADAM can be embedded as black-box modules within Lagrangian class methods. Theoretically, the paper proves that, under appropriate conditions, the proposed framework inherits the global convergence properties of the embedded stochastic subgradient method in the unconstrained setting and can be generalized to stochastic optimization problems with non-smooth expected constraints. This result extends the applicability of Lagrangian-class methods to non-convex and non-smooth optimization problems and provides a new analytical pathway for understanding the convergence behavior of modern stochastic optimization algorithms in complex constrained problems. The paper also conducts preliminary numerical experiments on constrained deep learning tasks, comparing the performance of Lagrangian-class methods embedded with proximal stochastic gradient descent and proximal ADAM against relevant stochastic augmented Lagrangian methods. The results demonstrate that the proposed framework can efficiently solve non-convex and non-smooth constrained optimization problems by combining with existing proximal gradient methods.

Full text link:

https://doi.org/10.1287/moor.2024.0479