Evading ML-Based Network Intrusion Detection: Polymorphic Blending Attack Tutorial

Introduction: Why Machine Learning for Network Security?

In today's cybersecurity landscape, attackers constantly evolve to bypass signature-based detection. Machine learning (ML) offers a powerful alternative by learning patterns of normal traffic and flagging anomalies. However, as we'll explore, ML models can be deceived by carefully crafted adversarial inputs. This tutorial, inspired by the Cs6262 Project 5 assignment, guides you through building a PAYL (Profile Anomaly Detection for Yielding Low False Positives) model and then conducting a polymorphic blending attack to evade it. We'll use Python 3 and focus on HTTP and DNS protocols.

Understanding the PAYL Model

PAYL is a 1-gram byte frequency model. It computes for each payload length the mean and standard deviation of ASCII byte frequencies. During detection, it calculates the Mahalanobis distance of a new payload to the model. If the distance exceeds a threshold, the payload is flagged as anomalous.

The training workflow involves:

Splitting normal traffic into 75% training and 25% testing.
Grouping payloads by length.
For each length, computing mean frequency vector and standard deviation vector.

This model is effective because normal traffic exhibits consistent byte distributions, while attacks often contain unusual byte sequences.

Setting Up the Environment

You can use the provided Linux VM or your local machine. Ensure you have Python 3 and the necessary libraries: numpy, scipy, and scapy for pcap handling. Download the project files from Canvas (project5.zip). The PAYL directory contains wrapper.py and supporting code.

Tip: Check SETUP.txt for an overview of code components.

Task A: Training the PAYL Model

First, we need to train the model on normal traffic. Run the wrapper in training mode without arguments:

python3 wrapper.py

This reads the default normal traffic pcap, splits it, and generates models for each payload length. The output shows parameters like mSF (smoothing factor) and mTMD (threshold for Mahalanobis distance) for HTTP and DNS. Record these values in parameters.txt with two decimal precision.

For example, your parameters.txt might look like:

http_smoothing_factor = 0.01
dns_smoothing_factor = 0.02
http_threshold_for_mahalanobis = 10.50
dns_threshold_for_mahalanobis = 12.30
http_distance = 0.00
dns_distance = 0.00

Note: The distance values will be updated in Task B.

Task B: Testing the Model on Attack Payload

Now, test the model on your unique attack payload (download YOUR_GTUSERNAME.pcap from Canvas). Place it in the PAYL directory. Run:

python3 wrapper.py YOUR_GTUSERNAME.pcap

This computes the Mahalanobis distance for each packet. The model should flag the attack packets as anomalous (distance > threshold). Record the distance values in parameters.txt under http_distance and dns_distance.

Task C: Polymorphic Blending Attack

The goal is to modify the attack payload so that it appears normal to the PAYL model while preserving the malicious content. We assume the attacker has access to one normal packet (the artificial profile) and the attack payload. The blending attack works by adjusting the byte frequencies of the attack payload to match the normal profile's distribution, subject to constraints that the malicious semantics are preserved.

We'll use a technique from the paper: for each byte position, we can modify bytes that are not critical to the attack. The algorithm:

Compute the target mean and std from the normal profile for the given payload length.
For each byte in the attack payload, if it can be changed (e.g., not part of shellcode), replace it with a byte sampled from a Gaussian distribution with the target mean and std, clipped to ASCII range.
Repeat until the Mahalanobis distance of the modified payload falls below the threshold.

Implement this in a Python script. For demonstration, we'll use a simple approach: we replace all non-critical bytes with random bytes that satisfy the target distribution.

import numpy as np
from scipy.spatial.distance import mahalanobis

def blend_attack(attack_bytes, normal_mean, normal_std, threshold):
    # attack_bytes: list of ints
    # normal_mean, normal_std: arrays of length 256
    modified = attack_bytes.copy()
    # Identify critical bytes (e.g., fixed shellcode positions)
    # For simplicity, assume all bytes can be changed
    for i in range(len(modified)):
        # Sample from Gaussian, clip to 0-255
        new_byte = int(np.random.normal(normal_mean[modified[i]], normal_std[modified[i]]))
        new_byte = max(0, min(255, new_byte))
        modified[i] = new_byte
    # Compute Mahalanobis distance
    freq = np.bincount(modified, minlength=256) / len(modified)
    dist = mahalanobis(freq, normal_mean, np.linalg.inv(np.diag(normal_std**2)))
    return modified, dist

You'll need to iterate until distance < threshold. This is a simplified version; real attacks require careful handling to preserve exploit functionality.

Evaluating the Attack

After blending, run the modified payload through the PAYL model. It should now be classified as normal (distance < threshold). Verify by running:

python3 wrapper.py blended_payload.pcap

If successful, you've evaded the ML-based IDS!

Real-World Implications

Polymorphic blending attacks highlight a critical vulnerability in ML-based security systems. As AI becomes more prevalent in cybersecurity, understanding adversarial attacks is essential. This project mirrors challenges faced by companies like Cloudflare and Darktrace, where attackers use AI to craft evasive malware. Similar techniques appear in AI-powered phishing campaigns and adversarial examples in image recognition. By learning these concepts, you're better prepared to defend against next-gen threats.

Conclusion

You've built a PAYL model, tested it against attacks, and successfully evaded it using a polymorphic blending attack. This hands-on experience demonstrates both the power and limitations of ML in security. Continue exploring adversarial ML to stay ahead in the cybersecurity arms race.