Camera Calibration: Boring but Essential
Nobody gets excited about camera calibration. It's tedious, fiddly, and involves waving a checkerboard at a camera for twenty minutes. But every 3D vision application — depth estimation, SLAM, augmented reality, robotic grasping — depends on accurate calibration. A 1-pixel error in the principal point propagates to centimeter-level errors in 3D reconstruction at typical working distances.
Intrinsic Parameters
A camera's intrinsic parameters describe how it projects 3D points onto the 2D sensor. The pinhole model has five parameters: focal lengths (fx, fy), principal point (cx, cy), and skew (almost always zero for modern cameras).
import numpy as np
# Typical camera matrix for a 1920x1080 industrial camera
K = np.array([
[1200.0, 0.0, 960.0],
[ 0.0, 1200.0, 540.0],
[ 0.0, 0.0, 1.0]
])Real lenses aren't pinholes. They introduce distortion — barrel distortion where straight lines curve outward, and tangential distortion from imperfect lens alignment. OpenCV's calibration uses 5-14 distortion coefficients depending on the model. For most industrial cameras with low-distortion lenses, 5 coefficients (k1, k2, p1, p2, k3) suffice.
Calibration with ChArUco Boards
Calibration requires images of a known pattern from multiple viewpoints. ChArUco boards (checkerboard + ArUco markers) are better than plain checkerboards because they work even when partially occluded.
import cv2
aruco_dict = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_6X6_250)
board = cv2.aruco.CharucoBoard((8, 6), 0.04, 0.03, aruco_dict)
all_corners, all_ids = [], []
image_size = None
for img_path in calibration_images:
img = cv2.imread(img_path)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
image_size = gray.shape[::-1]
corners, ids, _ = cv2.aruco.detectMarkers(gray, aruco_dict)
if ids isn't None and len(ids) > 4:
ret, charuco_corners, charuco_ids = cv2.aruco.interpolateCornersCharuco(
corners, ids, gray, board
)
if ret > 6:
all_corners.append(charuco_corners)
all_ids.append(charuco_ids)
ret, K, dist, rvecs, tvecs = cv2.aruco.calibrateCameraCharuco(
all_corners, all_ids, board, image_size, None, None
)
print(f"Reprojection error: {ret:.4f} pixels")The reprojection error tells you how good your calibration is. Below 0.5 pixels is excellent. Between 0.5-1.0 is acceptable for most applications. Above 1.0, something went wrong.
Extrinsic Calibration
Extrinsic parameters describe the camera's position and orientation in the world — a rotation matrix R and translation vector t. For multi-camera systems, you need the relative pose between cameras.
Stereo calibration estimates the rigid transformation between two cameras. For multi-camera rigs with 3+ cameras, pairwise stereo calibration accumulates error. Bundle adjustment over all cameras simultaneously works better, minimizing total reprojection error across all camera pairs. GTSAM and Ceres Solver handle the nonlinear optimization.
Geometric Transforms for 3D Work
With calibrated cameras, projecting between 2D and 3D uses standard linear algebra. The projection equation maps a 3D world point to a 2D image point through the camera matrix and extrinsic transform.
We covered a related topic in Prompt Engineering as Software Engineering: Version Control .
For the inverse — going from 2D back to 3D — you need depth information. Stereo triangulation uses correspondences between two views. The key relationship: depth = baseline * focal_length / disparity. In practice, stereo matching is the hard part. Block matching is fast but noisy. Semi-global matching (SGM) gives cleaner results at moderate cost. Neural stereo methods produce the best results but require GPU inference.
Common Pitfalls
Temperature changes affect calibration. Metal camera mounts expand and contract with temperature. In a warehouse robotics system, calibration drifted enough overnight (10 degrees C swing) to cause 5mm positioning errors. We added daily auto-recalibration using fiducial markers permanently mounted in the workspace.
Rolling shutter distortion is another calibration killer. Consumer cameras read pixels row-by-row, not simultaneously. Fast camera motion during exposure creates geometric distortion your calibration model doesn't account for. Either use global shutter cameras or add rolling shutter compensation.
Calibration isn't glamorous, but it's the foundation. I've seen teams spend months tuning their 3D reconstruction algorithm when the real problem was a sloppy calibration done once and never verified. Measure your reprojection error. Check for drift. Recalibrate regularly.
Multi-Camera Rig Calibration
Real-world 3D vision systems rarely use a single camera. Stereo rigs, structured light scanners, and multi-view reconstruction all require knowing the precise relative positions of multiple cameras. The calibration process scales in complexity: for N cameras, you have N intrinsic calibrations plus N*(N-1)/2 pairwise extrinsic calibrations.
Sequential pairwise calibration — calibrating camera 1-2, then 2-3, then 3-4 — accumulates error along the chain. Camera 4's pose relative to camera 1 carries the sum of three calibration errors. For rigs with more than 3 cameras, joint optimization is essential.
import gtsam
import numpy as np
def bundle_adjust_cameras(camera_poses, observations, K_matrices):
graph = gtsam.NonlinearFactorGraph()
initial = gtsam.Values()
noise_model = gtsam.noiseModel.Isotropic.Sigma(2, 1.0)
for cam_id, pose in enumerate(camera_poses):
initial.insert(gtsam.symbol("c", cam_id), pose)
for point_id, obs_list in enumerate(observations):
initial.insert(gtsam.symbol("p", point_id),
gtsam.Point3(*triangulate_initial(obs_list, camera_poses)))
for cam_id, pixel in obs_list:
K = K_matrices[cam_id]
cal = gtsam.Cal3_S2(K[0,0], K[1,1], 0, K[0,2], K[1,2])
factor = gtsam.GenericProjectionFactorCal3_S2(
gtsam.Point2(*pixel), noise_model,
gtsam.symbol("c", cam_id), gtsam.symbol("p", point_id), cal
)
graph.add(factor)
optimizer = gtsam.LevenbergMarquardtOptimizer(graph, initial)
result = optimizer.optimize()
return resultGTSAM handles the graph optimization efficiently, converging in seconds for typical multi-camera setups (4-8 cameras, 50-100 calibration points). The initial estimates from pairwise calibration seed the optimizer, and joint optimization refines them to minimize total reprojection error across all cameras simultaneously.
This connects to the ideas in Image Quality Assessment Metrics for Automated Visual Inspec.
Dynamic Calibration for Moving Cameras
Robotic systems mount cameras on moving arms or platforms. The intrinsics stay constant, but the extrinsics change with every movement. Hand-eye calibration determines the fixed transform between the camera and the robot's end-effector, so you can compute the camera pose for any robot configuration.
The classic hand-eye calibration formulation (AX=XB) requires at least 3 robot poses with corresponding camera observations of a calibration target. In practice, we use 15-20 poses spread across the robot's workspace. The additional poses improve accuracy and reveal any systematic errors in the robot's kinematic model.
Lens Selection and Its Impact on Calibration
Lens choice affects calibration difficulty and accuracy. Wide-angle lenses (below 50 degrees FOV) have minimal distortion and calibrate easily. Ultra-wide and fisheye lenses (above 120 degrees) require specialized distortion models and more calibration images for accurate parameter estimation.
For stereo systems, matching lenses between cameras is critical. Even two lenses of the same model can have slightly different focal lengths due to manufacturing tolerances. We characterize each lens individually, never assume identical intrinsics between cameras, and verify that the stereo baseline and convergence angle produce useful depth estimates across the working volume before final installation.
Telecentric lenses deserve mention for industrial metrology. They project all points at the same magnification regardless of depth, eliminating perspective distortion. Calibration is simpler (no perspective projection matrix), but the lenses are expensive and their narrow depth of field limits application range.
Practical Tips for Reliable Calibration
After calibrating hundreds of cameras across different projects, I've accumulated a set of practical rules that aren't in the textbooks:
Use at least 20 calibration images with the pattern covering different regions of the frame. The corners and edges are where distortion is strongest, and under-sampling those regions produces inaccurate distortion coefficients. I've seen teams take 50 images all from the center of the frame and wonder why their undistorted images have wavy edges.
See also: Speech Recognition Pipeline Optimization for Low-Resource La.
Vary the pattern distance from the camera. Close-up images (pattern fills 80% of the frame) help estimate the principal point and focal length. Far images (pattern at 20% of frame) help estimate distortion coefficients. A good calibration set has images at three distinct distances.
Rotate the pattern out of plane — don't just translate it. The calibration algorithm needs viewpoint diversity to separate intrinsic from extrinsic parameters. At least 5 of your calibration images should have the pattern tilted at 20-45 degrees from frontal.
Automated Calibration Verification
We run automated verification after every calibration. The verification captures a fresh set of images (not used in calibration) and computes the reprojection error on those images. If the verification error is more than 1.5x the calibration error, something went wrong — typically the calibration overfit to the specific images used.
def verify_calibration(K, dist, verification_images, board):
errors = []
for img_path in verification_images:
img = cv2.imread(img_path)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
corners, ids, _ = cv2.aruco.detectMarkers(gray, aruco_dict)
if ids is None or len(ids) < 6:
continue
ret, charuco_corners, charuco_ids = cv2.aruco.interpolateCornersCharuco(
corners, ids, gray, board
)
if ret < 6:
continue
# Compute reprojection error for this image
obj_points, img_points = board.matchImagePoints(charuco_corners, charuco_ids)
ret, rvec, tvec = cv2.solvePnP(obj_points, img_points, K, dist)
projected, _ = cv2.projectPoints(obj_points, rvec, tvec, K, dist)
error = np.sqrt(((img_points - projected.reshape(-1, 2))**2).sum(axis=1)).mean()
errors.append(error)
return np.mean(errors), np.std(errors)We also check for systematic errors by plotting the reprojection residuals as vectors overlaid on the image. Random residuals indicate noise-limited calibration — you can't do better. Systematic patterns (all vectors pointing outward from the center, or all pointing in one direction) indicate model mismatch — you need more distortion coefficients or a different distortion model.
For production systems, we store calibration parameters with timestamps and automatically invalidate them after a configurable period (typically 30 days for fixed installations, immediately for any camera that's been physically touched). The system won't run 3D algorithms with expired calibration, forcing recalibration before any accuracy-critical operations resume.
Depth Estimation Without Stereo
Not every 3D application needs stereo cameras. Monocular depth estimation has improved dramatically with models like MiDaS and Depth Anything. These models predict relative depth from a single image using learned priors about scene geometry. The depth map isn't metrically accurate — you get relative depth ordering, not absolute distances — but for applications like AR occlusion and 3D photo effects, relative depth is sufficient.
For metric depth from a single camera, you need either known scene constraints (a ground plane at known height) or a depth sensor as calibration reference. Intel RealSense and Azure Kinect provide metric depth natively, but their range is limited (typically 0.3-4 meters indoors). For outdoor and long-range applications, stereo remains the practical choice despite its calibration requirements.
Time-of-flight (ToF) sensors offer another alternative. They measure depth by timing infrared light pulses. Apple's LiDAR scanner in recent iPhones and iPads uses ToF technology, achieving centimeter-level accuracy within 5 meters. The main limitations are outdoor performance (direct sunlight overwhelms the infrared pulses) and resolution (typically 320x240, much lower than camera images). We use ToF sensors as depth ground truth for training monocular depth estimation models — the ToF provides sparse but accurate depth while the camera provides high-resolution imagery.