Building a DIY Network-Based IDS with Vagrant, Snort, and Syscall Hooking

Introduction: Why Build Your Own Network IDS?

In today's cybersecurity landscape, threats evolve faster than signature-based tools can keep up. With the rise of AI-generated malware and polymorphic code, a DIY network-based intrusion detection system gives you granular control over what runs on your network. This tutorial builds on syscall hooking concepts to create a sandbox that inspects every executable traversing your network—just like how a Snort rule flags suspicious packets, but at the kernel level.

Lab Setup: Three Virtual Machines with Vagrant

We'll use Vagrant to spin up three headless machines: mal (attacker), vicky (victim with Snort), and sandy (sandbox with LKM). This mirrors real-world red team/blue team exercises. Ensure you have VirtualBox and Vagrant installed, then run:

vagrant up

The machines are networked on 192.168.50.0/24. SSH into each using:

vagrant ssh mal
vagrant ssh vicky
vagrant ssh sandy

Part 1: File Transfer via FTP

First, confirm that vicky can receive a file from mal over FTP. From mal, connect to vicky:

ftp 192.168.50.3

Login with vagrant/vagrant. Use put to send a test file. This step validates network connectivity and FTP setup—a common pitfall in network security projects.

Part 2: Writing a Snort Rule to Detect Executables

Your Snort rule should flag executable files (e.g., ELF headers) but ignore non-executables. Here's a sample rule:

alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"Executable file detected"; content:"|7f 45 4c 46|"; sid:1000001; rev:1;)

This rule looks for the ELF magic bytes (0x7f 'E' 'L' 'F'). Adjust it to match the specific binaries in your lab. Test with the provided samples—only one non-executable should pass through.

Part 3: Automating Log Transfer from Vicky to Sandy

Write a daemon on vicky that watches for new Snort log files (PCAPs) and uploads them to sandy via FTP. Use watchdog (Python library) or a simple shell script with inotifywait. Example script:

#!/bin/bash
while inotifywait -e create /var/log/snort/; do
  for f in /var/log/snort/*.pcap; do
    ftp -n 192.168.50.4 <

Ensure each log is sent only once by moving or renaming it after upload.

Part 4: Updating the LKM for Malicious Syscall Detection

On sandy, modify your Lab 3 kernel module to detect only the malicious syscalls identified earlier (e.g., execve, open, write). The sample hooks.c provides a starting point. Insert the LKM:

insmod hooks.ko

Your module should print alerts to the kernel log when a malicious syscall is invoked.

Part 5: Sandbox Daemon to Analyze Binaries

Write a daemon on sandy that monitors a directory (e.g., /home/lab3/incoming/) for new PCAP files. When a file arrives, extract the binary using tcpdump or binwalk, then execute it with the LKM loaded. The daemon can be a Python script using watchdog or a cron job.

Part 6: Configuring rsyslog for IDS Alerts

Direct kernel messages from your LKM to /var/log/ids_alert.log. Add this to /etc/rsyslog.conf:

kern.*    /var/log/ids_alert.log

Restart rsyslog:

systemctl restart rsyslog

Now, when the LKM detects a malicious syscall, the alert appears in ids_alert.log.

Bonus: Behavioral Analysis and Zeek Integration

For Part a, define malicious behaviors (e.g., writing to /etc/passwd) and alert them via rsyslog. Then, have sandy send benign files back to vicky and drop malicious ones. For Part b, replace Snort with Zeek (formerly Bro) to detect executables. Zeek's scripting language allows custom analysis—a great way to expand your intrusion detection skills.

Deliverables Checklist

• Snort configuration file
• Daemon scripts for vicky and sandy
• LKM C file and Makefile
• Updated rsyslog configuration
• Report with screenshots of each step

Conclusion

By completing this project, you've built a functional network-based IDS that combines packet inspection with kernel-level monitoring. This hands-on experience is invaluable for cybersecurity students and professionals alike. As threats become more sophisticated, understanding how to customize detection mechanisms is a critical skill.