~~~~~~~~~~Unsupervised Learning~~~~~~~~~~

Asst. Prof. Dr. Somsak Chanaim

International College of Digital Innovation, CMU

September 24, 2026

Unsupervised Learning

Unsupervised Learning is one of the approaches in Machine Learning that analyzes data without a target variable (no labels).

In other words, the model must discover the structure or hidden patterns in the data on its own, without being told in advance how the data should be grouped or related. In some cases,

we may not even know how many groups should exist in the data.

Warm-up: How Would You Group These Animals?

Observe these living creatures. If you had to categorize them, what criteria would you use?

🦁
Lion

🐬
Dolphin

πŸ¦…
Eagle

🐸
Frog

πŸ•
Dog

🦈
Shark

πŸ¦‡
Bat

🐊
Crocodile

  • How many legs?
  • Terrestrial, aquatic, or amphibian?
  • Can it fly?
  • Wild or domesticated?

Defining Features

Before applying clustering algorithms, we must represent each entity as a Vector of Features.

Interactive Clustering Exploration

Key Takeaways on Clustering

  1. Unsupervised Learning: No ground-truth labels are provided in advance. Data points are partitioned strictly based on feature similarity (or distance).
  2. Feature Space Dictates the Outcome: Changing the selected feature set (e.g., from Habitat to Wings) reshapes the clusters entirely (e.g., bringing Eagle πŸ¦… and Bat πŸ¦‡ into the exact same group).
  3. Distance Metrics: In practice, algorithms treat each observation as a vector in multidimensional space (x1,x2,x3,…)(x_1, x_2, x_3, \dots) and minimize distances between data points and cluster centroids.

Principles of Unsupervised Learning

Instead of learning from examples with predefined answers (labels) as in Supervised Learning, in the case of Unsupervised Learning, the model works by:

  • Analyzing the structure of the data

  • Discovering hidden patterns or relationships

  • Grouping similar data together

  • Reducing data dimensionality to simplify analysis

Main Types of Unsupervised Learning

1. Clustering

  • Clustering automatically groups data so that items within the same group are similar to each other, while being different from items in other groups.

  • Example algorithms include:

    • (k)-means clustering
    • Hierarchical clustering
    • DBSCAN (Density-Based Spatial Clustering)

2. Dimensionality Reduction

  • Used to reduce the number of variables in the data while preserving as much important information as possible.

  • Useful for addressing the β€œCurse of Dimensionality” and allowing models to learn faster.

  • Example algorithms include:

    • PCA (Principal Component Analysis)

    • t-SNE (t-Distributed Stochastic Neighbor Embedding)

    • UMAP (Uniform Manifold Approximation and Projection)

3. Anomaly Detection

  • Used to identify values that deviate from the normal pattern in a dataset, such as detecting fraud or system errors.

  • Example algorithms include:

    • Isolation Forest

    • One-Class SVM

    • Autoencoders (Deep Learning)

4. Association Rule Learning

  • Used to discover relationships between variables in a dataset.
    Example: Market Basket Analysis, which identifies items that customers often purchase together.

  • Example algorithms include:

    • Apriori

    • FP-Growth

Real-World Applications of Unsupervised Learning

  • Customer Segmentation
    Companies can use clustering to group customers based on purchasing behavior, e.g., loyal customers, new customers, or churned customers.

  • Topic Modeling
    Unsupervised Learning can analyze and group articles or reviews. For example, LDA (Latent Dirichlet Allocation) is used to classify document categories.

  • Fraud Detection
    Anomaly Detection helps identify suspicious transactions, such as credit card use in very different locations within a short period.

  • Dimensionality Reduction
    PCA can be applied to reduce the number of variables before feeding data into a Machine Learning model, simplifying the process.

Advantages and Disadvantages of Unsupervised Learning

Advantages

  • Does not require labeled data, reducing the cost of data preparation.

  • Automatically discovers hidden patterns in the data.

  • Works well with large and complex datasets.

  • Useful for uncovering new knowledge that humans may not easily notice.

Disadvantages

  • Results are often harder to interpret compared to Supervised Learning.

  • Setting the number of clusters or certain parameters requires expertise.

  • May struggle with noisy or poorly prepared data.

Interactive: Clustering

K-Means Clustering

Clustering with kk-means (K-Means Clustering)

kk-means clustering is one of the most popular techniques for
Clustering in Unsupervised Learning.

It works by dividing data into kk groups based on similarity,
with each group represented by its own centroid (the central point).

How kk-means Works

flowchart TD
    A["Start"]
    B["Set number of<br/>clusters k"]
    C["Initialize<br/>k centroids"]
    D["Compute distances<br/>to centroids"]
    E["Assign each point to<br/>the nearest centroid"]
    F["Update centroids using<br/>cluster means"]
    G{"Converged or<br/>maximum iterations<br/>reached?"}
    H["Stop"]
    I["Output cluster labels<br/>and final centroids"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G

    G -- Yes --> H
    G -- No --> D

    H --> I

  1. Set the number of clusters kk:
    Specify the number of clusters before starting the algorithm.

  2. Initialize centroids:
    Select kk initial centroids, for example, by randomly selecting kk data points or using the kk-means++ initialization method.

  3. Compute distances and assign clusters:
    Calculate the Euclidean Distance between each data point and the centroids. Assign each point to the cluster with the nearest centroid.

  4. Update centroid positions:
    Recalculate each centroid as the mean of all data points assigned to its cluster.

  5. Repeat until convergence:
    Repeat Steps 3 and 4 until the cluster assignments or centroid positions no longer change significantly, or the maximum number of iterations is reached.

Objective Function: Minimize Squared Distances

J=βˆ‘j=1kβˆ‘π’™i∈Cjβˆ₯𝒙iβˆ’πjβˆ₯2 \boxed{ J = \sum_{j=1}^{k} \sum_{\mathbf{x}_i \in C_j} \left\| \mathbf{x}_i-\boldsymbol{\mu}_j \right\|^2 }

where CjC_j is the jj-th cluster and 𝝁j\boldsymbol{\mu}_j is the centroid of that cluster.

Goal: Minimize the sum of squared Euclidean distances between each data point and the centroid of its assigned cluster.

Distance Metrics in kk-means Clustering

In the process of kk-means clustering, calculating the distance between data points and centroids is crucial, as it determines which cluster each point belongs to.

By default, kk-means uses Euclidean Distance.
However, other distance metrics can also be applied depending on the nature of the data.

1. Euclidean Distance

d(x,y)=βˆ‘i=1n(xiβˆ’yi)2 d(x, y) = \sqrt{\sum_{i=1}^{n} (x_i - y_i)^2}

βœ… Easy to use and understand
βœ… Suitable for numerical quantitative data

❌ Sensitive to outliers
❌ Not suitable when features are on different scales (apply Standardization first)

2. Manhattan Distance

d(x,y)=βˆ‘i=1n|xiβˆ’yi| d(x, y) = \sum_{i=1}^{n} |x_i - y_i|

βœ… Suitable for data distributed in a grid-like structure (e.g., pixel images)
βœ… Less sensitive to outliers compared to Euclidean

❌ Not ideal for data with strong linear relationships or directional patterns

3. Cosine Similarity

cos⁑(ΞΈ)=xβ‹…y||x||β‹…||y||=βˆ‘i=1nxiyiβˆ‘i=1nxi2β‹…βˆ‘i=1nyi2 \cos(\theta) = \frac{x \cdot y}{||x|| \cdot ||y||} = \frac{\sum_{i=1}^{n} x_i y_i}{\sqrt{\sum_{i=1}^{n} x_i^2} \cdot \sqrt{\sum_{i=1}^{n} y_i^2}}

In general, Cosine Similarity measures the angle-based similarity between vectors rather than the distance.

βœ… Suitable for vector-based data such as documents, text, and embeddings βœ… Not affected by vector magnitude (e.g., documents of different lengths)

❌ Not suitable when magnitude itself is important for grouping

Distance Calculation with Different Methods

Example data:

x y
a 1 2
b 3 4
c 4 3

Euclidean Distance

Disab=22+22=8=2.828Disac=32+12=10=3.162Disbc=12+12=2=1.414 \begin{aligned} Dis_{ab}&=\sqrt{2^2+2^2}=\sqrt{8}=2.828\\ Dis_{ac}&=\sqrt{3^2+1^2}=\sqrt{10}=3.162\\ Dis_{bc}&=\sqrt{1^2+1^2}=\sqrt{2}=1.414 \end{aligned}

Manhattan Distance

Disab=|2|+|2|=4Disac=|3|+|1|=4Disbc=|1|+|1|=2 \begin{aligned} Dis_{ab}&= |2|+|2|= 4\\ Dis_{ac}&= |3|+|1|= 4\\ Dis_{bc}&= |1|+|1| = 2 \end{aligned}

Interactive: Distance

Example of kk-means

1. Sample Data

We have a dataset of 5 points in 2 dimensions (X, Y) and want to split them into k=2k=2 clusters.

Data Point X Y
A 2 10
B 2 5
C 8 4
D 5 8
E 7 5

2. Steps of kk-means Calculation

(1) Define kk and randomly select initial centroids

  • Set the number of clusters: k=2k = 2

  • Randomly choose initial centroids (suppose we pick A and C):

    • Centroid 1 (C1) = A = (2, 10)

    • Centroid 2 (C2) = C = (8, 4)

(2) Calculate the distance from each point to the centroids

We use Euclidean Distance:

d(x,y)=(X2βˆ’X1)2+(Y2βˆ’Y1)2 d(x, y) = \sqrt{(X_2 - X_1)^2 + (Y_2 - Y_1)^2}

Data Point d(P,C1)d(P, C1) d(P,C2)d(P, C2) Nearest Cluster
A (2,10) 0.00 8.49 C1
B (2,5) 5.00 6.08 C1
C (8,4) 8.49 0.00 C2
D (5,8) 3.61 5.00 C1
E (7,5) 7.07 1.41 C2

First-round result:

  • C1 cluster: {A, B, D}, C2 cluster: {C, E}

(3) Compute new centroids for each cluster

Centroid 1 (C1) {A, B, D}

C1x=2+2+53=93=3.00C1y=10+5+83=233=7.67\begin{aligned} C1_x &= \frac{2 + 2 + 5}{3} = \frac{9}{3} = 3.00\\ C1_y &= \frac{10 + 5 + 8}{3} = \frac{23}{3} = 7.67\end{aligned}

Centroid 2 (C2) {C, E}

C2x=8+72=152=7.50C2y=4+52=92=4.50\begin{aligned} C2_x &= \frac{8 + 7}{2} = \frac{15}{2} = 7.50\\ C2_y &= \frac{4 + 5}{2} = \frac{9}{2} = 4.50\end{aligned}

(4) Repeat steps 2 and 3

  • Recompute distances and reassign points to clusters.

  • Update the centroids, and continue iterating until the centroids no longer change (convergence).

3. Compute kk-means

Animation of K-means

Set up

K-mean (3D)

Choosing the Optimal kk in kk-means using Silhouette Score

Selecting the right number of clusters kk in kk-means is crucial. If kk is chosen too high or too low, clustering performance may suffer.

Silhouette Score is a metric for evaluating clustering quality, based on:

  • a(i)a(i): the average distance between a point and other points in the same cluster β†’ β€œcluster cohesion”
  • b(i)b(i): the average distance between a point and points in the nearest neighboring cluster β†’ β€œcluster separation”

S(i)=b(i)βˆ’a(i)max⁑(a(i),b(i)) S(i) = \frac{b(i) - a(i)}{\max(a(i), b(i))}

  • S(i)S(i) close to 1 β†’ The point is well clustered within its group.

  • S(i)S(i) close to 0 β†’ The point lies near the boundary between clusters.

  • S(i)S(i) close to -1 β†’ The point is likely assigned to the wrong cluster.

How to Use Silhouette Score to Choose kk

The average Silhouette Score indicates how appropriate the chosen kk is.

  1. Run kk-means with multiple values of kk (e.g., k=2,3,4,…,10k = 2, 3, 4, \dots, 10).

  2. Calculate the Silhouette Score for each kk.

  3. Select the kk that yields the highest average Silhouette Score.

Interpreting Silhouette Values by Cluster

  • Cluster 1 (Well-clustered): All three points have high positive Silhouette values (0.60–0.85). This means the points are tightly grouped and clearly separated from other clusters.

  • Cluster 2 (Borderline): One point is exactly at 0 and others are close to 0 (0.05–0.10). This indicates these points lie near the boundary between clusters, so their assignment is uncertain.

  • Cluster 3 (Poor cluster): The only point has a negative Silhouette value (-0.20). This shows the point is closer to another cluster than to its own, suggesting possible misclassification.

Interactive K-means & Silhouette Score

Interactive Silhouette Values

Advantages and Disadvantages of kk-means

Advantages of kk-means

βœ… Easy to implement and computationally efficient
βœ… Works well when data has clear cluster structures
βœ… Scales effectively to large datasets

Disadvantages of kk-means

❌ Requires specifying kk in advance
❌ Sensitive to outliers
❌ Performs poorly when clusters are non-spherical or vary in size

kk-means with Orange Data Mining

Example of kk-means in Orange (1)

Google Drive File

Google Drive File

Example of kk-means in Orange (2)

Google Drive File

Google Drive File

When Using Normalize in Orange Data Mining

⚠️ Be Careful When Interpreting Centroids

If Normalize columns is selected in the K-means widget in Orange Data Mining, the centroid values are expressed in the standardized scale.

Therefore, the centroid values should not be interpreted directly in the original units.

Standardization

z=xβˆ’xβ€ΎΟƒ z=\frac{x-\bar{x}}{\sigma}

where

  • xx = original value
  • xβ€Ύ\bar{x} = mean of the original variable
  • Οƒ\sigma = standard deviation of the original variable
  • zz = standardized value

Convert the Centroid Back to the Original Scale

If the standardized centroid value is zcz_c,

xc=zcσ+x‾ \boxed{ x_c=z_c\sigma+\bar{x} }

Example

Suppose Income has

xβ€Ύ=750,Οƒ=250 \bar{x}=750,\qquad \sigma=250

and Orange reports the centroid as

zc=βˆ’1.20 z_c=-1.20

Then,

xc=(βˆ’1.20)(250)+750=450 x_c=(-1.20)(250)+750 =450

Therefore,

Centroid of Income = 450 thousand THB per year

Remember: Normalization helps prevent variables with larger scales from dominating the clustering process.

However, when interpreting the cluster profile, convert the centroid values back to their original units.

Interpreting Clusters

After performing Clustering, we obtain Cluster Labels, such as
Cluster 1, Cluster 2, and Cluster 3.

However, the key questions are…

What are the characteristics of each cluster, and what do they mean in the real world?

Example: Customer Segmentation

A company has data on 12 customers with 3 features:

  • Age: Customer’s age
  • Annual Income: Annual income (thousand THB)
  • Spending Score: Customer spending score (0–100)

After applying kk-means with k=3k=3, the following results are obtained:

Customer Age Income Spending Cluster
C01 22 300 82 3
C02 25 360 88 3
C03 28 420 76 3
C04 31 450 81 3
C05 42 850 35 1
C06 45 920 28 1
C07 50 1,000 32 1
C08 48 950 40 1
C09 35 900 78 2
C10 38 1,050 85 2
C11 40 1,100 90 2
C12 36 980 82 2

Activity: What Do the Clusters Mean?

Cluster Profile

Cluster Avg. Age Avg. Income Avg. Spending
2 37.3 1,007.5 83.8
1 46.3 930.0 33.8
3 26.5 382.5 81.8

πŸ‘₯ Group Activity

Based on the cluster profiles above, discuss the following questions with your group:

  1. What are the characteristics of customers in Cluster 3?
  2. What are the characteristics of customers in Cluster 2?
  3. What are the characteristics of customers in Cluster 1?
  4. Suggest a descriptive name for each cluster.
  5. What marketing strategies could the company consider for each cluster?

From Clusters to Insights

🟦 Cluster 3

Young Active Spenders

  • Relatively young customers
  • Lower annual income
  • High spending scores

Possible Action

Promotions for young professionals
Discounts / Loyalty Programs
~

🟨 Cluster 1

High-Income Conservative Spenders

  • Older than customers in the other clusters
  • High annual income
  • Low spending scores

Possible Action

Investigate the reasons for low spending
Offer premium products that match customer needs

🟩 Cluster 2

High-Value Customers

  • Middle-aged customers
  • Very high annual income
  • High spending scores

Possible Action

VIP Programs
Personalized Offers
Customer Retention

Key Idea

Clustering does not automatically create knowledge.

Clusters→Cluster Profiles→Interpretation→Action \text{Clusters} \rightarrow \text{Cluster Profiles} \rightarrow \text{Interpretation} \rightarrow \text{Action}

Cluster labels such as 1, 2, and 3 have no inherent meaning.

Analysts must examine the characteristics of each cluster, interpret the results, and assign meaningful descriptive names before using the findings to support business decisions.

Orange Example

K-mean Example.ows

K-mean Example.ows

Example.xlsx

References