Understanding Convexity and Steepest Descent in ECE490 Homework 2: A Tutorial with Trend Analogies

Introduction: Why This Matters in 2026

In May 2026, optimization algorithms power everything from training large language models like GPT-5 to real-time strategy in esports. The ECE490 homework 2 p0 assignment focuses on fundamental convex optimization concepts that are essential for understanding how gradient-based methods work. This tutorial breaks down each problem with clear explanations and timely examples, helping you master the theory without copying solutions.

1. Convexity of Descent Directions

Problem 1 asks: Given a continuously differentiable function f with non-zero gradient at a point x, prove that the set of descent directions is convex. A descent direction d satisfies ∇f(x)Td < 0. The set is convex because if d1 and d2 are descent directions, then any convex combination d = λd1 + (1−λ)d2 also yields a negative inner product: ∇f(x)Td = λ∇f(x)Td1 + (1−λ)∇f(x)Td₂ < 0.

Think of this like choosing moves in a strategy game: if two moves both reduce your opponent's health, any mix of them also reduces health. This convexity ensures that the set of improving directions is well-behaved for optimization.

2. Strong Convexity and Its Implications

Problem 2: For an m-strongly convex function, prove that f(y) ≥ f(x) + ∇f(x)T(y−x) + (m/2)‖y−x‖2. The hint uses the inequality [∇f(x)−∇f(y)]T(x−y) ≥ m‖x−y‖2. This is a key property that guarantees fast convergence in optimization algorithms, similar to how a well-tuned AI model quickly learns from data with strong regularization.

3. Steepest Descent Convergence Rate

Problem 3 combines strong convexity and smoothness (L-smooth) to derive the linear convergence rate of steepest descent with constant step size: ‖xk − x*‖ ≤ (κ−1)/(κ+1)k ‖x0 − x*‖, where κ = L/m is the condition number. This rate is exponential, and the condition number measures how ill-conditioned the problem is. In practice, a high condition number (like a steep valley in a game's terrain) slows down convergence, but preconditioning can help.

4. Least-Squares Optimization and Regularization

Problem 4 deals with underdetermined least-squares: minx (1/2)‖Ax − b‖2 where A is N×d with N < d and full row rank.

(a) Solution Space

If there exists a z with Az = b, then the solution set is {z + v : v ∈ ker(A)}. The nullspace contains all vectors orthogonal to the rows of A. This is like having multiple strategies that achieve the same goal – the differences lie in the nullspace.

(b) Lipschitz Constant of Gradient

The gradient ∇f(x) = AT(Ax−b) is L-smooth with L = λmax(ATA). Since rank(A) = N, AAT is invertible and L = largest eigenvalue of ATA.

(c) Iterations for Steepest Descent

With optimal step size (1/L), steepest descent finds a solution with ‖Axk−b‖2 ≤ ε in O(log(1/ε)) iterations. Specifically, after k iterations, the residual norm decreases geometrically: ‖Axk−b‖2 ≤ (1 − 1/κ)k ‖b‖2, where κ = L/λmin+(ATA). The number of iterations to reach ε is roughly κ log(‖b‖2/ε).

(d) Regularized Problem

The regularized problem minx (1/2)‖Ax−b‖2 + (µ/2)‖x‖2 has a unique minimizer xµ = (ATA + µI)−1ATb. This is the ridge regression solution, widely used in machine learning to prevent overfitting.

(e) Convergence for Regularized Problem

The condition number of the regularized Hessian is (L+µ)/(µ), which is better conditioned than the original. Steepest descent converges linearly with rate (κµ−1)/(κµ+1) where κµ = (L+µ)/µ. The number of iterations to achieve fµ(xk) − fµ(xµ) ≤ ε is O(κµ log(1/ε)).

(f) Bound on Original Objective

If fµ(x̂) − fµ(xµ) ≤ ε, then f(x̂) ≤ f(xµ) + ε + (µ/2)‖xµ‖2. This bound shows how regularization affects the original objective.

Real-World Connections

In 2026, optimization is at the heart of AI training. For example, the steepest descent method is analogous to a player in a battle royale game moving directly toward the safe zone – the gradient points to the steepest increase, so moving opposite reduces the loss. Strong convexity ensures the safe zone is a single point, not a line. Regularization is like adding a penalty for risky moves, keeping the player's path stable.

Understanding these concepts is crucial for designing efficient algorithms in machine learning, robotics, and finance. The ECE490 homework builds a strong foundation for more advanced topics like stochastic gradient descent and Adam optimizer.

Study Tips for ECE490

• Review matrix calculus and eigenvalues – they are used extensively.
• Practice proving convexity using definitions and inequalities.
• Simulate steepest descent on simple quadratic functions to see convergence.
• Connect theory to applications: least-squares is used in linear regression, which is a building block of neural networks.

By mastering these problems, you'll be well-prepared for exams and real-world optimization challenges.