Understanding Machine Precision and Finite Difference Errors in Numerical Analysis

Introduction to Numerical Analysis and Machine Precision

Numerical analysis is the study of algorithms that use numerical approximation for problems of continuous mathematics. In today's world, from training large AI models to simulating financial markets, understanding the limits of floating-point arithmetic is crucial. The machine precision (often denoted as ε) is the smallest number such that 1 + ε > 1 in a given floating-point system. For IEEE double-precision, which uses 64 bits, the machine precision is 2-52 ≈ 2.22 × 10-16. This means any relative error smaller than ε is essentially lost due to rounding.

Assignment Overview: Math1426 Homework 1

This tutorial covers three key tasks from the assignment: defining machine precision, finding coefficients for an O(h²) finite difference approximation, and analyzing error plots for f(x)=sin(x). We'll use Python for implementation, but the concepts apply to any language.

Task 1: Machine Precision in IEEE Double-Precision

In the IEEE 754 double-precision format, numbers are represented with 1 sign bit, 11 exponent bits, and 52 mantissa bits. The machine epsilon is the gap between 1 and the next representable number, i.e., 2-52. This limits the accuracy of numerical computations. For example, in machine learning, gradient descent updates can be affected by precision when learning rates are very small.

Task 2: Finding Coefficients for O(h²) Finite Difference

We consider the formula: f'(x) ≈ (a f(x+h) + b f(x)) / h. Using Taylor expansion about x:

• f(x+h) = f(x) + h f'(x) + h²/2 f''(x) + h³/6 f'''(x) + O(h⁴)

Substituting into the formula gives: (a f(x) + a h f'(x) + a h²/2 f''(x) + ... + b f(x)) / h = (a+b)/h f(x) + a f'(x) + a h/2 f''(x) + ...

For the approximation to be consistent, we need the coefficient of f(x) to be zero: a+b = 0. Then the leading error term is from f''(x): a h/2 f''(x). To achieve O(h²) error, we need the coefficient of f''(x) to be zero as well: a h/2 = 0 → a=0, but that gives trivial solution. Wait, re-evaluate: Actually, the term with f''(x) is of order h, not h². To get O(h²), we need to cancel the O(h) term. So we require a = 0? That would give zero derivative. Let's derive properly.

We want: f'(x) = (a f(x+h) + b f(x))/h + O(h²). Expand:

a f(x+h) + b f(x) = a(f + h f' + h²/2 f'' + h³/6 f''') + b f
= (a+b) f + a h f' + a h²/2 f'' + a h³/6 f''' + ...

Divide by h: (a+b)/h f + a f' + a h/2 f'' + a h²/6 f''' + ...

For this to equal f' + O(h²), we need:

• Coefficient of f: (a+b)/h = 0 → a+b = 0
• Coefficient of f': a = 1
• Coefficient of f'': a h/2 = 0 for O(h) term to vanish → but that forces a=0, contradiction. So we cannot achieve O(h²) with only two points? Actually, the formula is a forward difference if b = -a, so a=1, b=-1 gives first-order. To get second-order, we need three points. The assignment says a,b are real-valued coefficients, but maybe it's a trick: the only way is to have a=0, b=0? No.

Let's re-read the assignment: Consider the numerical difference formula with real-valued coefficients a,b ∈ R. For which values of a and b is the mathematical approximation error of order O(h²)? The formula is not explicitly given, but from context it's likely f'(x) ≈ (a f(x+h) + b f(x))/h. From above, we get first-order unless a=0 (trivial). Perhaps the formula is centered? Actually, common second-order formula: f'(x) ≈ (f(x+h) - f(x-h))/(2h) which has coefficients a=1/2, b=-1/2? Wait, that's three points? No, it uses f(x+h) and f(x-h). But here we have only f(x+h) and f(x). So it's impossible to get second-order. The correct answer: no such a,b exist except the trivial zero? But the assignment likely expects a=1, b=-1 gives first-order. Maybe the formula is f'(x) ≈ (a f(x+h) + b f(x-h))/h? That would be symmetric. Let's assume it's the latter for the sake of the tutorial.

Given the ambiguity, we'll explain the general approach: expand Taylor series, set up equations for coefficients, and solve for a,b to cancel lower-order terms. For a two-point formula, you can only get first-order; for three-point, you can get second-order. We'll provide the solution for the common centered difference as an example.

Task 3: Error Analysis with Python Plotting

We'll implement the finite difference formula (assuming the standard forward difference: a=1, b=-1) and compute the numerical evaluation error for f(x)=sin(x) at x=0 and x=1. The error consists of truncation error (from Taylor series) and rounding error (due to machine precision). We'll plot both on a log-log scale.

import numpy as np
import matplotlib.pyplot as plt

def forward_diff(f, x, h):
    return (f(x+h) - f(x)) / h

x_vals = [0, 1]
h_vals = np.logspace(-16, 0, 100)
for x in x_vals:
    errors = []
    for h in h_vals:
        approx = forward_diff(np.sin, x, h)
        exact = np.cos(x)
        errors.append(abs(approx - exact))
    plt.loglog(h_vals, errors, label=f'x={x}')
plt.xlabel('h')
plt.ylabel('absolute error')
plt.legend()
plt.grid(True)
plt.show()

The plot will show that for large h, truncation error dominates (error ~ O(h)), and for very small h, rounding error dominates (error ~ ε/h). The optimal h is around sqrt(ε) ≈ 10-8. At x=0, sin(x) ≈ x, so cancellation error may be less; at x=1, the derivative is cos(1) ≈ 0.54, and the error behavior is similar but with different constants.

Interpreting Results and Recommendations

From the log-log plot, you'll see a V-shaped curve. The minimum error occurs at h ≈ 10-8. For h larger than that, error increases due to truncation; for h smaller, error increases due to rounding. Therefore, we recommend using h around 10-8 for this formula when using double precision. This is a classic trade-off in numerical differentiation, analogous to choosing the right learning rate in AI training: too high and you overshoot, too low and progress stalls due to precision.

Connection to Current Trends

In 2026, numerical precision is more relevant than ever. Large language models like GPT-5 use mixed-precision training to balance speed and accuracy. Financial algorithms for high-frequency trading rely on minimizing rounding errors. Even in gaming, physics engines use floating-point arithmetic; understanding machine epsilon helps developers avoid glitches. This assignment gives you foundational skills for such real-world applications.

Key Takeaways

• Machine precision ε ≈ 2.2e-16 limits achievable accuracy.
• Finite difference formulas have a trade-off between truncation and rounding error.
• Optimal step size h ≈ sqrt(ε) minimizes total error.
• Python's matplotlib is a powerful tool for error analysis.

By mastering these concepts, you'll be better prepared for advanced topics in numerical analysis, scientific computing, and data science.