Mastering Multivariable Calculus and Linear Algebra for ECE 490: A Homework 0 Tutorial

Introduction: Why ECE 490 Homework 0 Matters

ECE 490 is a challenging course that builds on foundational knowledge from multivariable calculus and linear algebra. The ECE 490 homework 0 is a diagnostic problem set designed to help you assess your readiness. This tutorial will guide you through the core concepts, providing clear explanations and step-by-step proofs. Whether you are reviewing for the course or preparing for exams, mastering these topics is essential for success in ECE 490 and related fields like machine learning and optimization.

1. The Fundamental Theorem of Calculus and the Gradient

One of the first problems in ECE 490 homework 0 asks you to prove an identity involving the gradient. The key is to use the fundamental theorem of calculus along a line segment. Consider a differentiable function f : ℝn → ℝ. For fixed vectors x and y, define g(t) = f((1−t)x + ty). Then by the chain rule, g'(t) = ∇f((1−t)x + ty)T(y − x). Applying the fundamental theorem of calculus, we get:

f(y) - f(x) = ∫₀¹ ∇f((1-t)x + ty)ᵀ (y-x) dt

This is a powerful tool for proving inequalities and identities in ECE 490. Think of it like a gradient descent algorithm in machine learning: we move along the gradient direction to minimize a loss function. The integral form helps us relate the change in function value to the gradient along the path.

2. Quadratic Functions: Gradient and Hessian

A quadratic function of the form f(x) = ½ xTAx + bTx + c is central to optimization. The gradient is ∇f(x) = ½ (A + A^T)x + b. If A is symmetric, this simplifies to Ax + b. The Hessian is constant: ∇²f(x) = A. In ECE 490, you will often work with symmetric matrices. For example, in linear regression, the cost function is f(x) = ||Ax − b||² = (Ax − b)^T(Ax − b). Expanding, we get f(x) = x^T(A^TA)x − 2b^TAx + b^{Tb. The Hessian is 2ATA, which is positive semidefinite. This is why the least squares problem has a unique solution when ATA is invertible.}

3. Log-Sum-Exp Function: A Smooth Maximum

The log-sum-exp function f(x) = log(∑i=1n exp(xi)) is a smooth approximation of the maximum function. Its gradient is ∇f(x) = softmax(x), where the i-th component is exp(xi) / ∑j exp(xj). The Hessian is ∇²f(x) = diag(softmax(x)) − softmax(x) softmax(x)T. This matrix is positive semidefinite, making the function convex. In ECE 490, you will use this in convex optimization and machine learning for multiclass classification. For instance, the cross-entropy loss in neural networks uses the softmax function. Understanding its gradient and Hessian is crucial for training deep learning models like GPT or stable diffusion.

4. Least Squares: Uniqueness of the Minimizer

Given A ∈ ℝm×n and b ∈ ℝm, we want to minimize f(x) = ||Ax − b||². Setting the gradient to zero gives the normal equations: ATAx = ATb. If A^TA is invertible, the unique solution is x* = (A^TA)⁻¹A^{Tb. To prove uniqueness, note that the Hessian is 2ATA, which is positive definite when ATA is invertible. Thus, f is strictly convex, and any stationary point is the global minimum. This result is fundamental in data science and signal processing.}

5. Convexity and the Gradient Inequality

A differentiable function f is convex if and only if f(y) ≥ f(x) + ∇f(x)T(y − x) for all x,y. This is the first-order condition for convexity. In ECE 490, you will prove that this implies the Jensen's inequality: f(tx + (1−t)y) ≤ tf(x) + (1−t)f(y). This property is essential in optimization theory and machine learning. For example, in support vector machines, the hinge loss is convex, ensuring that the optimization problem has a global minimum.

6. Eigenvalues of a Rank-One Matrix

Consider the matrix formed by the outer product of the standard basis vectors: M = ∑i=1n eiei+1T (with cyclic condition). This is a permutation matrix. Its eigenvalues are the n-th roots of unity: λk = e2πik/n for k = 0,1,…,n-1. This problem tests your understanding of eigenvalues and eigenvectors. In ECE 490, you will encounter similar matrices in graph theory and network analysis.

7. Eigenvectors of uvT

Given vectors u,v ∈ ℝn with uTv = 0, consider A = uvT. Then Au = u(vTu) = 0, so u is an eigenvector with eigenvalue 0. More generally, any vector orthogonal to v is in the nullspace. The only nonzero eigenvalue is vTu with eigenvector u. This simple structure appears in principal component analysis (PCA) and recommender systems.

8. Singular Values from SVD

The singular value decomposition (SVD) of a matrix A ∈ ℝn×n is A = ∑i=1n σi uiviT, where σi are singular values and {ui}, {vi} are orthonormal bases. Then ATA = ∑i=1n σi² viviT, so the eigenvalues of ATA are σi². Thus, the singular values of A are exactly the σi. This result is fundamental in data compression, image processing, and recommender systems. For example, in Netflix's recommendation algorithm, SVD is used to factorize user-item matrices.

Conclusion

This tutorial covered the essential topics from ECE 490 homework 0. By mastering gradients, Hessians, convexity, and linear algebra, you will be well-prepared for the course. Remember to practice these proofs and computations. For further study, explore convex optimization, machine learning, and deep learning applications. Good luck!