Binary Exploitation for CS6035: Control Flow Hijacking with Python and pwntools

Introduction to Binary Exploitation

Binary exploitation is a core cybersecurity skill that involves manipulating a compiled program's behavior by exploiting memory vulnerabilities. In the CS6035 Project 1: Binary Exploitation, you'll learn to hijack control flow using techniques like buffer overflows and return-oriented programming (ROP). This tutorial covers the essential concepts, tools, and scripting with Python and pwntools to craft exploits. With the 2026 cybersecurity landscape seeing increased attacks on legacy binaries, understanding these exploits is more relevant than ever.

Understanding Memory Layout and the Stack

Before writing exploits, you must understand how a program uses memory. The stack stores local variables, function parameters, and return addresses. When a function is called, a stack frame is pushed: return address, saved base pointer, and local variables. A buffer overflow occurs when data written to a buffer exceeds its allocated size, overwriting adjacent memory. For example, in a C program with a 64-byte buffer, writing 80 bytes can overwrite the return address, allowing you to redirect execution.

Real-World Analogy: Stack Overflow in a Gaming Tournament

Imagine a 2026 esports tournament bracket where each match stores player stats in a fixed-size slot. A bug allows a winning player's stats to overflow into the next slot, overwriting the opponent's data. This is exactly how a buffer overflow works—except in memory, you overwrite the return address to control where the program goes next.

Tools of the Trade: pwntools, GDB, objdump, ropper

The CS6035 project uses a VM with precompiled binaries. Key tools include:

• Python + pwntools: Automate exploit scripting. Functions like cyclic() generate patterns to find offset to the return address.
• GDB with pwndbg: Debug the binary step-by-step to observe register and memory changes.
• objdump: Disassemble binaries for static analysis—find function addresses and gadget locations.
• ropper: Search for ROP gadgets (e.g., pop rdi; ret) to chain instructions.

These tools are essential for both binary exploitation and reverse engineering workflows.

Stage 02: Stack Smashing – Crafting Your First Exploit

Stack smashing is the classic buffer overflow. The goal is to overwrite the return address to jump to a win function or shellcode. Here's a step-by-step approach:

1. Find the offset: Use cyclic(200) and run the binary in GDB. When it crashes, read the RIP value and use cyclic_find(value) to get the exact offset.
2. Locate the target address: Use objdump -t binary or nm to find the win function address.
3. Construct the payload: payload = b'A' * offset + p64(win_addr)
4. Send and exploit: Use pwntools' p.sendline(payload) and interact to get the flag.

For example, if offset is 72 and win address is 0x40123a, the exploit script is:

from pwn import *
p = process('./vuln')
payload = b'A'*72 + p64(0x40123a)
p.sendline(payload)
print(p.recvall())

This simple buffer overflow exploit demonstrates control flow hijacking.

Stage 03: Return-Oriented Programming (ROP)

When the binary has protections like NX (non-executable stack), you cannot jump to shellcode. Instead, you chain existing code snippets called gadgets. ROP is like assembling a 2026 AI chatbot where each gadget is a short instruction that performs a small task, and together they execute a full system call. For example, to call execve('/bin/sh'), you need gadgets to set RDI, RSI, RDX, and then syscall.

Finding Gadgets with ropper

Run ropper --file binary --search 'pop rdi'. Typical gadgets:

• pop rdi; ret – address 0x400c9e
• pop rsi; ret – address 0x400ca1
• syscall; ret – address 0x400d02

Then craft the ROP chain using pwntools' ROP class or manually:

rop = ROP(binary)
rop.call('execve', [binsh_addr, 0, 0])
payload = b'A'*offset + rop.chain()

This ROP exploit bypasses NX and is a common CTF challenge technique.

Stage 04: Final Flags – Putting It All Together

The final stage combines multiple vulnerabilities. You may need to leak addresses (ASLR bypass) or chain ROP with stack pivoting. For example, a 2026 finance app bug allowed attackers to leak a canary and then overflow a buffer—similar to how you'll leak a stack address and then compute offsets. Use %p format string leaks or read from /proc/flag via a ROP chain that calls open, read, write.

Scripting with Python and pwntools: Best Practices

Your exploit scripts (e.g., e.py) should be modular. Use context.log_level = 'debug' for verbose output. When debugging with GDB, attach with gdb.attach(p). For remote targets (though this project is local), use remote('host', port). Always test against the original binary in the VM—recompiled binaries will be fingerprinted and fail Gradescope.

Common Pitfalls and FAQ

Students often ask: Why does my exploit segfault? Check offset accuracy, endianness (use p64() for 64-bit), and ensure you're not overwriting important data. If using cyclic_find(), ensure the pattern is in the correct register (RIP on x86_64). For ROP, remember that gadgets must end with ret to chain.

Conclusion

Binary exploitation is a rewarding skill that combines low-level understanding with creative problem-solving. By mastering stack smashing and ROP, you'll be prepared for advanced binary exploitation challenges in CTFs and real-world security assessments. The CS6035 project provides a safe environment to practice these techniques. Good luck, and may your exploits run without segfaults!