Building Noise-Robust Classifiers with Transition Matrix Estimation: A COMP5328 Guide

Introduction: Why Label Noise Matters in 2026

In the era of AI-generated data and crowdsourced annotations, label noise is everywhere. From social media sentiment tags to medical image labeling, noisy labels degrade model performance. This tutorial builds on the COMP5328 assignment theme: constructing classifiers that withstand class-conditional label noise. You'll implement two robust algorithms and a transition matrix estimator, tested on FashionMNIST and CIFAR datasets. Think of it as training a spam filter on mislabeled emails—your model must learn the true patterns despite the noise.

Understanding Label Noise and Transition Matrices

Label noise occurs when training labels differ from true labels. In class-conditional noise, the flip probability depends on the true class. A transition matrix T captures these probabilities: T[i][j] = P(noisy label = j | true label = i). For example, in the provided FashionMNIST0.3 dataset, T[0][0]=0.7, T[0][1]=0.3, meaning 30% of class 0 images are flipped to class 1. Clean test data remains unflipped.

Why Robustness Matters

Standard classifiers like softmax regression assume clean labels. When trained on noisy data, they overfit to noise, reducing test accuracy. Robust algorithms correct for noise using T. If T is known, you can adjust loss functions. If unknown (as with CIFAR), you must estimate T first.

Two Robust Classification Algorithms

You need at least two classifiers, one not taught in the course. Here are two effective options:

1. Forward Correction with Softmax

This method modifies the predicted probabilities using T. Let f(x) be the softmax output for clean labels. The noisy class probability is T^T f(x). Train by minimizing cross-entropy between noisy labels and T^T f(x). This is straightforward when T is known.

2. Loss Correction via Backward Reweighting

Another approach: invert T to correct the loss. For each sample, multiply the loss by T^{-1} applied to the noisy label indicator. This requires T to be invertible (which it usually is). Backward correction is robust but can amplify noise if T is poorly estimated.

Estimating the Transition Matrix

For datasets without T (like CIFAR), you must estimate it. A common method uses a clean validation set (but here we only have noisy validation). Instead, use the noisy data itself with a two-stage approach:

1. Train a base classifier on noisy data (e.g., a simple CNN). Its predictions are noisy but can be used to estimate T.
2. Estimate T by comparing predicted noisy labels with true noisy labels on a held-out set. Specifically, let Q[i][j] = P(predicted noisy = j | noisy label = i). If the classifier is good, Q approximates T^2, but we need T. A trick: use the fact that T is stochastic and assume the classifier's confusion on clean data is identity (a strong assumption). Alternatively, use a small clean subset (not available here) or anchor points (samples almost certainly from a class). For simplicity, you can use the method from tutorial 9: train a classifier on noisy data, then compute the confusion matrix on a validation set, and normalize rows. That gives an estimate of T (though biased). To improve, use a second classifier trained on corrected labels iteratively.

Validate your estimator on the first two datasets by comparing to known T. For CIFAR, report your estimated T in the report.

Implementation Tips for COMP5328

Use Python 3 with NumPy, PyTorch or TensorFlow. Load datasets with np.load(). Split training+validation (80/20) randomly each run. Train at least 10 times per classifier, report mean and std of test accuracy.

Sample Code Skeleton

import numpy as np
import torch

data = np.load('FashionMNIST0.3.npz')
Xtr_val, Str_val = data['Xtr'], data['Str']
Xts, Yts = data['Xts'], data['Yts']
# Split: 80% train, 20% val
indices = np.random.permutation(len(Xtr_val))
train_idx, val_idx = indices[:int(0.8*len(indices))], indices[int(0.8*len(indices)):]
X_train, S_train = Xtr_val[train_idx], Str_val[train_idx]
X_val, S_val = Xtr_val[val_idx], Str_val[val_idx]
# Define model with forward correction
class ForwardModel(torch.nn.Module):
    def __init__(self, T):
        super().__init__()
        self.fc = torch.nn.Linear(784, 3)  # for FashionMNIST
        self.T = torch.tensor(T, dtype=torch.float32)
    def forward(self, x):
        logits = self.fc(x)
        clean_probs = torch.softmax(logits, dim=1)
        noisy_probs = torch.mm(clean_probs, self.T.T)
        return noisy_probs
# Train with cross-entropy on S_train
# Evaluate on Yts

Performance Evaluation

Compute top-1 accuracy on clean test data. For each classifier, run 10 random splits and report mean ± std. Compare the two algorithms: forward correction usually performs better when T is accurate; backward correction can be more sensitive to estimation errors.

Connecting to Real-World Trends

In 2026, label noise is critical for AI safety. For instance, autonomous vehicles trained on mislabeled road signs could cause accidents. Similarly, content moderation systems on social media rely on noisy user reports. Your assignment skills directly apply to building trustworthy AI.

Conclusion

This tutorial covered two robust classifiers and a transition matrix estimator for label noise. By mastering these, you'll handle real-world noisy datasets. Remember to validate your estimator on known T and report both accuracy and estimated T for CIFAR. Good luck with your COMP5328 assignment!