Object Detection Model Selection: YOLO...

Picking an Object Detector for Production

Every few months a new object detection paper lands on arXiv claiming modern mAP on COCO. The benchmark table reshuffles. And none of it tells you which model to actually ship. I've deployed all three major families — YOLO variants, DETR-style transformers, and EfficientDet — in production systems ranging from warehouse robotics to traffic monitoring. The right choice depends on constraints the papers don't measure.

YOLO: Speed Where It Counts

YOLO's appeal is obvious. It's fast. YOLOv8 runs inference on a single 640x640 image in roughly 6ms on an A100, and even the nano variant clocks under 2ms on a T4. For applications where you're processing 30 frames per second from multiple camera feeds, that speed gap compounds quickly.

But speed isn't YOLO's only advantage. The architecture is straightforward to export. ONNX conversion works reliably, TensorRT optimization is well-documented, and the Ultralytics library handles most of the boilerplate:

from ultralytics import YOLO

model = YOLO("yolov8m.pt")
model.export(format="engine", device=0, half=True)

trt_model = YOLO("yolov8m.engine")
results = trt_model.predict("frame.jpg", conf=0.4, iou=0.5)

The catch? YOLO struggles with small objects in dense scenes. In a recent warehouse project, we had pallets stacked three high with labels roughly 15x15 pixels in the frame. YOLOv8 medium missed about 22% of those labels even after fine-tuning. Switching to a higher input resolution (1280) helped, but inference time tripled.

DETR and Its Descendants

DETR changed the game by eliminating anchor boxes and NMS post-processing. The model directly predicts a set of detections using a transformer decoder. In theory, this is cleaner. In practice, the original DETR was painfully slow to train — 300 epochs minimum — and the inference latency made real-time applications impossible.

RT-DETR fixed the latency problem. It's genuinely competitive with YOLO on speed while retaining the transformer's strength at detecting small, occluded objects. On our warehouse label dataset, RT-DETR-L caught 91% of those tiny labels versus YOLO's 78%.

The downside is operational complexity. DETR models are harder to optimize for edge deployment. TensorRT conversion works but requires more manual intervention. The attention mechanism doesn't play nicely with INT8 quantization — we saw a 7 mAP drop with naive quantization versus just 2 mAP for YOLOv8. And debugging is harder. When YOLO misses something, you can usually trace it to anchor assignments or NMS thresholds. When DETR misses something, you're staring at attention maps trying to understand why the decoder didn't attend to that region.

This connects to the ideas in Anomaly Detection in High-Dimensional Industrial Sensor Data.

EfficientDet: The Middle Ground

EfficientDet doesn't get the attention it deserves. The BiFPN feature pyramid is genuinely clever — it learns weighted connections between feature scales instead of using a fixed top-down/bottom-up path. For multi-scale detection problems, this matters.

I've found EfficientDet-D3 hits a sweet spot for many production scenarios. It's faster than DETR variants (though slower than YOLO nano/small), and its multi-scale handling outperforms YOLO on datasets with extreme size variation. A traffic monitoring system we built needed to detect both trucks and license plates in the same frame — EfficientDet handled the 50:1 scale ratio cleanly while YOLO needed two separate models.

The ecosystem is the problem. EfficientDet's reference implementation is in TensorFlow, and while PyTorch ports exist, they don't get the same maintenance as Ultralytics. Export pipelines break more often. Community support is thinner. If something weird happens during training, you're reading source code, not Stack Overflow answers.

Training Data Matters More Than Architecture

Model architecture matters less than training data quality. I've seen a well-tuned YOLOv5 beat a poorly trained DETR every time. A few things that consistently matter across architectures:

Annotation consistency — one annotator drawing tight boxes while another draws loose ones will kill your model. We use a 3-pixel tolerance check across annotators.
Class imbalance strategy — not just oversampling rare classes, but ensuring the rare class appears at different scales and positions in the training images.
Negative examples — images with zero target objects. Without these, the model hallucinates detections on clean backgrounds. We target 10-15% negatives in every training split.
Edge case curation — systematically collecting failure cases from production and adding them back to training data. This feedback loop matters more than architecture choice after the first deployment.

The Decision Framework We Use

After deploying about a dozen detection systems, our team settled on a practical decision tree. If your latency budget is under 10ms per frame and the objects are medium-to-large, go with YOLO. If accuracy on small or occluded objects is the primary metric and you can afford 20-30ms latency, RT-DETR. If you've got extreme multi-scale requirements and your team is comfortable with TensorFlow, EfficientDet.

One pattern that's worked well: prototype with YOLOv8 first. Always. It trains in hours, not days. The tooling is excellent. If YOLOv8 meets your accuracy requirements — and it often does — ship it. Only switch to DETR or EfficientDet when you've proven YOLO can't hit the accuracy bar with reasonable data and augmentation effort.

NMS Tuning for Dense Scenes

YOLO requires Non-Maximum Suppression as a post-processing step. The default IoU threshold of 0.7 works for most cases, but crowded scenes need tuning. We've built a per-class NMS threshold configuration:

NMS_THRESHOLDS = {
 "vehicle": 0.5,
 "pedestrian": 0.45,
 "license_plate": 0.3,
 "default": 0.7
}

def class_aware_nms(predictions, iou_thresholds):
 results = []
 for cls_id, cls_preds in group_by_class(predictions):
 cls_name = CLASS_NAMES.get(cls_id, "default")
 threshold = iou_thresholds.get(cls_name, iou_thresholds["default"])
 results.extend(nms(cls_preds, threshold))
 return results

DETR doesn't need NMS at all — the set prediction mechanism handles duplicate suppression natively. That's one fewer hyperparameter to tune, which matters more than people think when you're maintaining multiple deployed models.

Production Monitoring and Retraining

Whichever model you choose, production drift is real. Lighting changes seasonally. Camera angles shift from vibration. New object variants appear that weren't in the training set. We run weekly accuracy audits on a rotating sample of production frames, comparing model predictions against human annotations.

When accuracy drops below our SLO threshold (typically 85-90% mAP depending on the application), we trigger a retraining cycle. The full pipeline — from drift detection to retrained model in production — takes about 48 hours in our setup. Most of that's human annotation time for the new failure cases. The model training itself is 4-6 hours for YOLO, 12-18 hours for DETR. That difference matters when you're responding to a production accuracy degradation.

Don't over-index on academic benchmarks. COCO mAP is a useful sanity check but it doesn't predict production performance on your specific data distribution. The only benchmark that matters is your own validation set, built from your actual camera feeds, with your actual edge cases.

Hardware Selection for Different Detection Workloads

The GPU you choose constrains your model options. An NVIDIA T4 (16GB, ~65 TOPS INT8) is the sweet spot for single-stream detection deployments in the cloud. It runs YOLOv8-medium at 10ms per frame and costs roughly $0.35/hour on spot instances. For multi-stream edge deployments, the Jetson Orin NX (8-16GB shared memory, ~100 TOPS INT8) handles 4-8 concurrent camera feeds depending on model complexity.

One non-obvious consideration: memory bandwidth matters more than raw TOPS for detection models. YOLOv8 is memory-bound, not compute-bound, at typical batch sizes. An A10 with 600 GB/s memory bandwidth outperforms an A100 per-dollar for single-image inference because the A100's extra compute capacity sits idle waiting for memory reads.

Batch Size and Throughput Trade-offs

Batching multiple images together improves GPU utilization dramatically. Processing 8 images at once through YOLOv8 takes roughly 15ms total — about 1.9ms per image versus 6ms for single-image inference. But batching adds latency: the last image in the batch waits for the first seven to arrive before processing starts.

For real-time applications, we use dynamic batching. The system collects frames until either the batch is full or a timeout expires (typically 20-50ms). This gives most of the throughput benefit while capping worst-case latency. NVIDIA Triton Inference Server implements this natively:

# Triton model config for dynamic batching
max_batch_size: 16
dynamic_batching {
 preferred_batch_size: [4, 8, 16]
 max_queue_delay_microseconds: 50000
}
instance_group [
 { count: 1, kind: KIND_GPU, gpus: [0] }
]

The preferred batch sizes should match your typical workload. If you're processing 6 cameras at 5fps each, you'll accumulate about 30 frames per second — a batch of 8 forms naturally every 260ms, well within most latency budgets.

Evaluation Beyond mAP

mAP at IoU 0.5 (mAP50) is the standard benchmark metric, but it hides important information. A model with 85% mAP50 might have 95% recall on large objects and 60% recall on small ones. Per-class and per-size evaluation reveals whether the model meets your actual requirements.

We report four numbers for every production model: mAP50 and mAP50-95 (standard benchmarks), recall at our operating precision threshold (typically 90% precision), and the confusion matrix between the most commonly confused class pairs. That confusion matrix tells you more about deployment readiness than any single number. If your model confuses "truck" with "bus" 15% of the time, that might be acceptable for traffic counting but catastrophic for a logistics application tracking specific vehicle types.

Object Detection Model Selection: YOLO vs DETR vs EfficientDet for Production