Building a Custom Shell in C: From Tokenizer to Advanced Features

Introduction to Shell Programming

Building a custom shell is a rite of passage for C programmers. It teaches you process management, string parsing, and system calls. In this tutorial, we'll walk through creating a shell similar to the CS3650 project, covering tokenization, basic command execution, built-in commands, and advanced features like sequencing, redirection, and pipes.

Understanding the Shell Tokenizer

The first step is to break user input into tokens. Tokens are meaningful chunks like commands, arguments, and operators. Our tokenizer must handle special characters: ( ) < > ; | and whitespace. Quoted strings are treated as single tokens.

Tokenization Example

Input: echo "hello world" | sort
Tokens: ["echo", "hello world", "|", "sort"]

Implement a function tokenize(char *line) that returns an array of token strings. Use a state machine to track whether you're inside quotes. For simplicity, you can allocate a fixed-size array (e.g., 256 tokens) and parse character by character.

Basic Shell Implementation

Your shell should display a prompt, read input, tokenize it, and execute commands. The basic requirements include:

• Print Welcome to mini-shell on startup.
• Prompt: shell $
• Support commands with arguments.
• Handle double quotes as a single argument.
• Run child processes in the foreground.
• Implement exit and handle Ctrl-D.
• Print error if command not found.

Fork and Exec

Use fork() to create a child process and execvp() to run the command. The parent waits for the child to finish. Here's a minimal example:

pid_t pid = fork();
if (pid == 0) {
    execvp(args[0], args);
    perror("exec failed");
    exit(1);
} else {
    waitpid(pid, NULL, 0);
}

Built-in Commands

Implement built-ins like cd, source, prev, and help. These run without forking.

cd

Change directory using chdir(). Handle errors if the path doesn't exist.

source

Read a script file line by line and execute each line as a command. This allows batch execution.

prev

Store the previous command line and re-execute it. Useful for repeating commands quickly.

help

Print a list of built-in commands and their descriptions.

Advanced Features: Sequencing, Redirection, and Pipes

These operators allow powerful command chaining. Implement them by parsing the token list and setting up file descriptors accordingly.

Sequencing with ;

Split commands at semicolons and execute each sequentially. This is straightforward: after tokenizing, look for ; tokens and treat each segment as a separate command.

Input Redirection (<)

Redirect stdin from a file. Use open() and dup2() to replace stdin with the file descriptor.

int fd = open(filename, O_RDONLY);
dup2(fd, STDIN_FILENO);
close(fd);

Output Redirection (>)

Redirect stdout to a file. Use open() with O_WRONLY | O_CREAT | O_TRUNC.

Pipes (|)

Connect stdout of one command to stdin of another. Use pipe() and fork() multiple children. For each pipe segment, create a pipe and redirect the appropriate ends.

Combining Operators

Operators can be combined, e.g., sort < input.txt | uniq > output.txt. Your parser must respect operator precedence: pipes bind tighter than redirection and sequencing. A common strategy is to first split by ;, then by |, and handle redirections within each segment.

Testing Your Shell

Test with various commands: ls -la, echo hello world, pwd, cd /tmp, cat file.txt | wc -l, sort < data.txt > sorted.txt. Ensure error messages are clear.

Trend Connection: AI and Shell Automation

In 2026, many developers use AI assistants to generate shell commands. Your custom shell can be extended to integrate with AI models for natural language command interpretation. For example, you could add a built-in ai command that sends a prompt to an LLM and executes the suggested command. This blends classic systems programming with modern AI trends.

Conclusion

Building a shell from scratch is a rewarding project that deepens your understanding of operating systems. By implementing tokenization, process control, and I/O redirection, you gain practical skills applicable to many low-level programming tasks. Start with the tokenizer, build the basic shell, then add advanced features. Happy coding!