Mastering K-Means & K-Medoids for Image Compression: A Step-by-Step Guide (ISYE 6740)

Introduction: Clustering in the Age of AI and Image Compression

In 2026, as AI-powered apps like Midjourney and DALL-E dominate creative workflows, understanding clustering algorithms is more relevant than ever. Whether you're compressing images for a social media platform or analyzing customer segments for a fintech startup, K-means and K-medoids are foundational tools. This tutorial breaks down the core concepts from ISYE 6740 homework 1-4, focusing on clustering theory and image compression implementation.

Understanding K-Means Clustering

Distortion Function and Centroid Update

The K-means algorithm minimizes the distortion function: J = Σn Σk rnk ||xn - μk||2. Here, rnk is 1 if point n belongs to cluster k, else 0. To find the optimal centroid μk, we take the derivative of J with respect to μk and set it to zero. This yields μk = (Σn rnk xn) / (Σn rnk), i.e., the mean of points in cluster k.

Convergence to Local Optimum

K-means converges in finite steps because the distortion function monotonically decreases at each iteration and is bounded below. The algorithm alternates between assignment (minimizing J over rnk) and update (minimizing J over μk). Since there are only finitely many possible assignments (KN), the algorithm must reach a fixed point.

Hierarchical Clustering: Linkage Metrics

Bottom-up hierarchical clustering merges clusters based on distance metrics. The three common linkages are:

• Single linkage: minimum distance between any pair of points in two clusters.
• Complete linkage: maximum distance between any pair.
• Average linkage: average distance between all pairs.

Which linkage yields clusters most similar to K-means? Since K-means minimizes within-cluster variance, average linkage (which considers all pairs) tends to produce spherical clusters similar to K-means, especially when K is a power of 2. For the two moons dataset, single linkage can separate the moons because it can follow the narrow gap, while complete linkage and average linkage may merge them.

Image Compression Using Clustering

Image compression reduces colors by clustering pixels in RGB space. Each pixel is a 3D point (R,G,B). K-means assigns each pixel to the nearest centroid, then replaces the pixel color with the centroid's color. This reduces the image to K colors.

Implementing K-Means (mykmeans.m)

Your implementation should:

1. Initialize centroids randomly or from K random pixels.
2. Assign each pixel to the nearest centroid using Euclidean distance.
3. Update centroids as the mean of assigned pixels.
4. Repeat until convergence (assignments stop changing).

Example code snippet:

% Initialize centroids
centroids = pixels(randperm(size(pixels,1), K), :);
for iter = 1:100
    % Assign
    distances = pdist2(pixels, centroids);
    [~, class] = min(distances, [], 2);
    % Update
    for k = 1:K
        centroids(k,:) = mean(pixels(class==k,:), 1);
    end
end

Implementing K-Medoids (mykmedoids.m)

K-medoids is robust to outliers because it uses actual data points as centers. Implementation steps:

• Initialize medoids randomly.
• Assign each point to the nearest medoid using a distance measure (e.g., Euclidean, Manhattan, or cosine).
• For each cluster, swap the medoid with each point in the cluster and compute total distance; keep the configuration that minimizes total distance.
• Repeat until convergence.

You can also use the PAM (Partitioning Around Medoids) algorithm. For distance, Euclidean works well for images, but Manhattan may yield different color palettes.

Practical Tips for Your Homework

• Use the provided skeleton code (homework2.m) to load images and visualize results.
• Test with small K (2,3) and large K (16,32). Observe how compression quality improves but file size increases.
• Measure convergence time: K-means is faster than K-medoids because medoid updates require O(N2) comparisons.
• Different initializations can lead to different local optima; run multiple times and pick the best.

Conclusion

Clustering is a powerful tool for unsupervised learning and compression. By implementing K-means and K-medoids, you gain insight into how algorithms trade off speed, robustness, and quality. As you work on your ISYE 6740 homework, remember that these concepts apply beyond images—to customer segmentation, anomaly detection, and even AI model compression. Happy coding!