Understanding Data Processing Inequality and Entropy in Information Theory: A Practical Guide for EE276 Homework

Introduction to Information Theory Concepts

Information theory is the backbone of modern communication, data compression, and machine learning. In this tutorial, we'll explore key concepts from the EE276 homework #2 p0 assignment, including the Data Processing Inequality, entropy, mutual information, and the Asymptotic Equipartition Property (AEP). These ideas are not just academic—they appear in everything from AI chatbots like ChatGPT-5 (released April 2026) to TikTok's recommender system and even sports analytics for the 2026 FIFA World Cup qualifiers.

1. Data Processing Inequality (DPI)

The Data Processing Inequality states that if X → Y → Z form a Markov chain, then I(X;Y) ≥ I(X;Z). This means no clever processing of Y can increase the information about X. Let's break down the required proofs.

1.1 Proving H(X|Y) = H(X|Y,Z)

Given the Markov condition p(z|y) = p(z|y,x), we have X → Y → Z. Then:

H(X|Y,Z) = H(X|Y) + I(X;Z|Y) but I(X;Z|Y)=0 since X and Z are independent given Y.

Thus H(X|Y) = H(X|Y,Z). Similarly for H(Z|Y).

1.2 Proving H(X|Y) ≤ H(X|Z)

Using the chain rule and the fact that conditioning reduces entropy:

H(X|Z) ≥ H(X|Y,Z) = H(X|Y).

So indeed H(X|Y) ≤ H(X|Z).

1.3 Mutual Information Inequalities

From the above, I(X;Y) = H(X) - H(X|Y) ≥ H(X) - H(X|Z) = I(X;Z). Similarly I(Y;Z) ≥ I(X;Z).

1.4 Conditional Mutual Information

I(X;Z|Y) = 0 follows directly from the Markov property.

2. Two Looks: Counterexamples with Binary Variables

Consider X, Y1, Y2 binary with I(X;Y1)=0 and I(X;Y2)=0. Does that imply I(X;Y1,Y2)=0? Not necessarily! For a counterexample, let X be uniform, Y1 = X ⊕ N1, Y2 = X ⊕ N2, where N1, N2 are independent fair coins. Then I(X;Y1)=0 (since Y1 is independent of X), same for Y2. But (Y1,Y2) together reveal X via XOR: Y1⊕Y2 = N1⊕N2, which is independent of X, so still I(X;Y1,Y2)=0? Actually, here it's still zero. A better counterexample: let X be uniform, Y1 = X, Y2 = X? Then I(X;Y1)=1, not zero. To get zero marginal but nonzero joint, let X be uniform, Y1 = X with probability 0.5 else independent, Y2 similarly. Then I(X;Y1)=0, I(X;Y2)=0, but (Y1,Y2) can reveal X when both equal X. So I(X;Y1,Y2) > 0. For part (b), I(Y1;Y2)=0 does not follow either; same counterexample works.

3. Prefix and Uniquely Decodable Codes

Given a code, we check if it's prefix-free. A prefix code has no codeword that is a prefix of another. If not, it may still be uniquely decodable. For example, the code {0, 01} is not prefix but is uniquely decodable because the only way to parse is: 0 then 01? Actually {0, 01} is not uniquely decodable because 01 can be parsed as 0+1? But if 1 is not a codeword, then 01 is only one codeword. However, typical counterexample: {0, 01, 10} is not prefix (0 is prefix of 01) but is uniquely decodable? Let's not digress. The assignment asks to argue via an algorithm: start from left, find the shortest codeword that matches, etc.

4. Relative Entropy and Cost of Miscoding

Given p and q distributions, compute H(p) = 2 bits, D(p||q) = 0.5 bits, etc. Then verify optimal codes: code C1 is optimal for p because it satisfies Kraft inequality with equality and has minimal expected length. Same for C2 under q. Then compute redundancy when using C2 for p: L - H(p) where L = average length under p using C2.

5. Strong Law of Large Numbers and AEP

For i.i.d. X_i ~ p, the log-likelihood ratio log(q(X_i)/p(X_i)) converges in probability to -D(p||q). So when p is true, the odds favoring q decay exponentially.

6. AEP Typical Set Properties

The typical set A_ε^(n) contains sequences whose empirical entropy is close to H. The set B_ε^(n) based on sample mean may not be typical. We show that P(A_ε^(n) ∩ B_ε^(n)) → 1 and that |A_ε^(n)| ≈ 2^{nH}.

7. AEP-like Limit and Bonus

Find limit of p(X_1,...,X_n)^{1/n} which tends to 2^{-H} in probability. For the set C_n(t), its size is ≤ 2^{nt}, and probability tends to 1 if t > H, else 0 if t < H.

Conclusion

Understanding these principles is crucial for fields like data compression (ZIP, JPEG), machine learning (mutual information for feature selection), and AI explainability. As we see in 2026, the same math underpins TikTok's algorithm and NFL play-calling analytics. Master these, and you'll ace EE276!