Mastering Normal Mapping and Shadow Mapping in OpenGL: A Step-by-Step Implementation Guide

Introduction to Normal Mapping and Shadow Mapping

In modern 3D graphics, adding surface details and realistic shadows is essential for creating immersive virtual worlds. Normal mapping (also called bump mapping) simulates fine surface geometry by modifying per-pixel normals, while shadow mapping determines which areas are lit or in shadow. This tutorial walks through implementing both techniques in OpenGL, based on a typical assignment where you have spheres on a plane lit by a single directional light. We'll cover tangent space calculation, TBN matrix, lighting in tangent space, depth map generation from the light's view, shadow testing, and percentage-closer filtering (PCF) for softer shadows. By the end, you'll have a solid foundation for adding these effects to your own projects.

Understanding Tangent Space and the TBN Matrix

Normal maps store normals in a local coordinate system called tangent space. This space is defined per vertex by three vectors: the normal (N), tangent (T), and bitangent (B). The tangent vector points in the direction of the texture's u-coordinate, the bitangent along the v-coordinate, and the normal is perpendicular to the surface. To use a normal map, we must transform lighting vectors (light direction, view direction) from world space to tangent space using the TBN matrix (a 3x3 matrix with columns T, B, N). This allows the normal map's perturbed normals to interact correctly with the light.

Calculating Tangent and Bitangent Vectors

For a triangle with vertices v0, v1, v2 and texture coordinates (u0,v0), (u1,v1), (u2,v2), we can compute the tangent vector using the edge vectors and texture deltas. Let E1 = v1 - v0, E2 = v2 - v0, and deltaU1 = u1 - u0, deltaV1 = v1 - v0, deltaU2 = u2 - u0, deltaV2 = v2 - v0. Then solve: T = (deltaV2 * E1 - deltaV1 * E2) / (deltaU1*deltaV2 - deltaU2*deltaV1). The bitangent can be computed as cross(N, T) or from a similar formula. In practice, we often average tangents per vertex and orthogonalize using Gram-Schmidt.

Implementing Normal Mapping in the Shader

In the vertex shader, we compute the TBN matrix and transform the light vector (from light position to fragment) and view vector (from camera to fragment) into tangent space. These are passed to the fragment shader as varyings. In the fragment shader, we sample the normal map (usually a texture with RGB encoding normals as (x,y,z) mapped to [0,1]), unpack it to [-1,1], and use it as the per-pixel normal in tangent space. Then we compute diffuse and specular lighting using the tangent-space light and view vectors. The result is a surface that appears to have fine bumps and grooves, even on a flat mesh.

// Vertex shader snippet
out vec3 fragPos_tangent;
out vec3 lightDir_tangent;
out vec3 viewDir_tangent;

void main() {
    vec3 T = normalize(gl_NormalMatrix * tangent.xyz);
    vec3 B = normalize(gl_NormalMatrix * bitangent.xyz);
    vec3 N = normalize(gl_NormalMatrix * normal.xyz);
    mat3 TBN = mat3(T, B, N);
    // Transform vectors
    vec3 fragPos_world = (gl_ModelViewMatrix * gl_Vertex).xyz;
    vec3 lightPos_world = lightPosition;
    vec3 lightDir_world = lightPos_world - fragPos_world;
    vec3 viewDir_world = -fragPos_world; // camera at origin in view space
    fragPos_tangent = fragPos_world * TBN;
    lightDir_tangent = lightDir_world * TBN;
    viewDir_tangent = viewDir_world * TBN;
    gl_Position = ftransform();
}

Shadow Mapping Basics: Two-Pass Rendering

Shadow mapping involves two rendering passes. In the first pass, we render the scene from the light's point of view using a simple depth shader, storing the depth values in a texture (the shadow map). This tells us the closest surface to the light along each direction. In the second pass, we render the scene from the camera's view. For each fragment, we compute its position in light space (using the light's view-projection matrix), compare its depth to the stored depth in the shadow map, and if the fragment's depth is greater than the stored depth, it is in shadow. We then attenuate the diffuse term (but not ambient) accordingly.

Generating the Depth Map

To generate the depth map, we set up a framebuffer object (FBO) with a depth texture attachment. Then we render the scene with a simple shader that outputs depth. The light's view-projection matrix is constructed from the light's position and direction. For a directional light, we use an orthographic projection. The scene is rendered from the light's perspective, and the depth values are written to the texture.

// Depth vertex shader (depth_v.glsl)
void main() {
    gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
}

// Depth fragment shader (depth_f.glsl)
void main() {
    // No need to write anything; depth is automatically stored
}

Shadow Testing in the Fragment Shader

In the second pass, we have access to the shadow map texture. For each fragment, we compute its light-space position (light_frag_pos). We then sample the shadow map at the corresponding texture coordinates (light_frag_pos.xy / light_frag_pos.w, mapped to [0,1]). We compare the sampled depth (closest) with the fragment's depth (light_frag_pos.z / light_frag_pos.w). If the fragment's depth is greater than the sampled depth plus a small bias (to avoid shadow acne), the fragment is in shadow. The bias is needed because of floating-point precision. We output a shadow factor (0 for shadow, 1 for lit) that we multiply with the diffuse term.

float calculateShadow(vec4 light_frag_pos, sampler2D shadowMap) {
    vec3 projCoords = light_frag_pos.xyz / light_frag_pos.w;
    projCoords = projCoords * 0.5 + 0.5;
    float closestDepth = texture(shadowMap, projCoords.xy).r;
    float currentDepth = projCoords.z;
    float bias = 0.005;
    return currentDepth - bias > closestDepth ? 0.0 : 1.0;
}

Percentage-Closer Filtering (PCF) for Soft Shadows

Basic shadow maps produce hard, aliased edges. PCF smooths shadows by sampling the shadow map multiple times around the fragment's projected coordinates and averaging the results. A common approach is to sample a 3x3 grid of texels around the fragment's position. This softens shadow boundaries and reduces aliasing. The cost is 9 texture lookups per fragment, but it's manageable. The shadow factor becomes the average of the individual shadow tests.

float pcfShadow(vec4 light_frag_pos, sampler2D shadowMap) {
    vec3 projCoords = light_frag_pos.xyz / light_frag_pos.w;
    projCoords = projCoords * 0.5 + 0.5;
    float shadow = 0.0;
    vec2 texelSize = 1.0 / textureSize(shadowMap, 0);
    for(int x = -1; x <= 1; ++x) {
        for(int y = -1; y <= 1; ++y) {
            vec2 offset = vec2(x, y) * texelSize;
            float closest = texture(shadowMap, projCoords.xy + offset).r;
            float current = projCoords.z;
            float bias = 0.005;
            shadow += (current - bias > closest) ? 0.0 : 1.0;
        }
    }
    return shadow / 9.0;
}

Putting It All Together: Scene Setup and Interaction

In the assignment, you are given a scene with spheres on a plane. The config.txt file provides rotation and translation matrices for initial positions. You can use keyboard/mouse to change light position or object transforms. The normal mapping is toggled with 'm'/'M'. When enabled, you should see the spheres appear bumpy due to the normal map. The shadow mapping should cast realistic shadows from the directional light. Pay attention to the TBN calculation: ensure tangent and bitangent are correct per vertex. Also, the shadow map resolution affects quality; use a 1024x1024 or 2048x2048 texture. For PCF, you can adjust the bias and filter size for best results.

Common Pitfalls and Debugging Tips

If your normal mapping looks wrong (e.g., lighting appears flat or distorted), check that the TBN matrix is orthonormal and that you are transforming the light and view vectors correctly. A common mistake is forgetting to normalize vectors after transformation. For shadow mapping, if you see no shadows or everything is shadowed, verify the light's view-projection matrix and the depth comparison. Shadow acne appears as dark stripes on lit surfaces; increase bias. Peter panning (shadows detaching from objects) occurs if bias is too high; use a smaller bias or slope-based bias. Also, ensure the shadow map is bound and sampled correctly.

Connecting to Real-World Applications

Normal mapping and shadow mapping are used extensively in games, VR, and simulations. For example, in the popular game Fortnite, normal maps add detail to character skins and environments, while shadow maps create dynamic shadows that enhance realism. In architectural visualization, these techniques bring renders to life. Even in AI-generated content, such as text-to-3D models, these lighting tricks are applied to make outputs look more polished. Understanding these fundamentals will help you in graphics programming, whether you're building a game engine, a 3D modeling tool, or a virtual reality experience.

Conclusion

Implementing normal mapping and shadow mapping in OpenGL involves understanding tangent space, constructing the TBN matrix, performing two-pass shadow rendering, and optionally applying PCF for softer shadows. By following the steps outlined here, you can add these effects to your own scenes. Remember to test incrementally and debug using visual feedback. With practice, you'll be able to create visually stunning graphics that rival commercial applications. Good luck with your assignment!