Modern CNNs: ResNet, EfficientNet and Vision Transformers Compared

Choosing between ResNet, EfficientNet, and Vision Transformers is no longer a question of which model tops ImageNet by 0.3%. In production, the decision affects training budget, inference latency, data requirements, and how painful your quantization pipeline will be. I have deployed all three families across medical imaging, retail shelf detection, and document understanding systems, and the trade-offs are consistent enough to be predictable — if you know where to look. This article compares these architectures not on leaderboard scores alone, but on how they behave when you actually have to train, fine-tune, and ship them. We will cover why ResNet still matters, how EfficientNet squeezes accuracy per FLOP, and where Vision Transformers earn their keep despite their appetite for data and memory.

Residual Learning in ResNet: Why Skip Connections Matter Beyond 50 Layers

In my experience, the most practical contribution of ResNet was not accuracy, it was trainability. Before residual networks, pushing a plain convolutional network past 30-40 layers reliably hurt performance. Training error increased with depth, which ruled out simple overfitting — it was an optimization failure. The degradation problem meant gradients vanished or exploded through long chains of convolutions and ReLUs, and even with careful initialization and Training Deep Networks: Learning Rate Schedules, Batch Normalization and Gradient Clipping strategies, deep plain nets stalled.

ResNet reframed the learning problem. Instead of asking a stack of layers to learn a desired mapping H(x), it asks them to learn the residual F(x) = H(x) - x, so the output becomes F(x) + x via an identity skip connection. If an identity mapping is optimal, the network can simply push F(x) toward zero, which is far easier than approximating identity through non-linear layers. This simple addition preserves gradient flow. During backpropagation, the gradient has a direct path through the skip, which mitigates vanishing gradients without adding parameters.

The Bottleneck Block and Downsampling Decisions

For ResNet-50 and deeper, the basic two-layer block is replaced by a bottleneck design: 1x1 reduction, 3x3 convolution, 1x1 expansion. A ResNet-50 bottleneck reduces 256 channels to 64, processes spatially at lower cost, then restores to 256. This cuts FLOPs by roughly 50% compared to two 3x3 convolutions at full width while keeping representational capacity. I've found that this detail matters when you adapt ResNet for dense prediction — using dilated convolutions in the later stages or replacing strided downsampling with average pooling can significantly improve segmentation mIoU without retraining from scratch.

Downsampling is another subtlety. The original ResNet uses a 1x1 convolution with stride 2 on the skip path when spatial dimensions change. Later variants (ResNet-D, ResNetV2) replace this with average pooling + 1x1 or move BatchNorm and ReLU to a pre-activation position. In practice, pre-activation (BN-ReLU-Conv) gives slightly better gradient behavior and makes the network a bit easier to fine-tune with lower learning rates, especially for transfer learning on small datasets.

import torch.nn as nn

class Bottleneck(nn.Module):
    expansion = 4
    def __init__(self, in_channels, mid_channels, stride=1, downsample=None):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels, mid_channels, kernel_size=1, bias=False)
        self.bn1 = nn.BatchNorm2d(mid_channels)
        self.conv2 = nn.Conv2d(mid_channels, mid_channels, kernel_size=3,
                               stride=stride, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(mid_channels)
        self.conv3 = nn.Conv2d(mid_channels, mid_channels * self.expansion,
                               kernel_size=1, bias=False)
        self.bn3 = nn.BatchNorm2d(mid_channels * self.expansion)
        self.relu = nn.ReLU(inplace=True)
        self.downsample = downsample

    def forward(self, x):
        identity = x
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.relu(self.bn2(self.conv2(out)))
        out = self.bn3(self.conv3(out))
        if self.downsample is not None:
            identity = self.downsample(x)
        out += identity
        return self.relu(out)

# Usage: ResNet-50 stage with stride 2 downsampling
# downsample = nn.Sequential(
#     nn.Conv2d(512, 1024, kernel_size=1, stride=2, bias=False),
#     nn.BatchNorm2d(1024)
# )

When I Still Reach for ResNet

ResNet-34 and ResNet-50 remain my default baseline for any new vision task. They converge reliably with SGD + momentum, are tolerant to imperfect hyperparameters, and have mature support for quantization and pruning. For datasets under 50k images, a ResNet-50 pretrained on ImageNet often outperforms a ViT-B/16 trained from scratch, and fine-tunes in far fewer epochs. If you need deterministic latency on edge devices or support for INT8 without extensive calibration, ResNet is still hard to beat. For a deeper look at attention-based successors, see Transformer Architecture from Scratch: Attention, Encoders and Decoders.

Compound Scaling in EfficientNet: From B0 to B7 Without Wasting FLOPs

EfficientNet started from a different question: if you have more compute, how should you spend it? Empirical scaling had been ad hoc — make the network deeper, or wider, or feed higher-resolution images. The EfficientNet authors showed that these dimensions are coupled. Scaling only depth improves accuracy but quickly saturates because deeper networks struggle to capture fine-grained features without sufficient width and resolution. Scaling only resolution wastes capacity if the network is too shallow to exploit it.

The solution was compound scaling. Given a baseline EfficientNet-B0 found via neural architecture search, they introduce a compound coefficient phi that jointly scales depth (alpha^phi), width (beta^phi), and input resolution (gamma^phi) under the constraint alpha * beta^2 * gamma^2 ≈ 2. This ensures FLOPs increase by roughly 2^phi. The coefficients alpha, beta, gamma were found by small grid search on B0 (values around 1.2, 1.1, 1.15). The result is a family from B0 (5.3M params, 0.39B FLOPs) to B7 (66M params, 37B FLOPs) where each step gives a predictable accuracy bump without the diminishing returns of one-dimensional scaling.

MBConv, Squeeze-and-Excitation, and Swish

EfficientNet-B0's building block is MBConv6, an inverted residual with depthwise separable convolution borrowed from MobileNetV2, plus a Squeeze-and-Excitation (SE) module. The flow is: 1x1 expansion (6x), 3x3 or 5x5 depthwise convolution, SE block that reweights channels based on global average pooled context, 1x1 projection, then a residual add if shapes match. Depthwise convolutions drastically reduce FLOPs and parameters, while the SE block recovers accuracy lost by that factorization by explicitly modeling channel interdependencies. In my experiments, disabling SE in B0 drops ImageNet top-1 by about 1% with minimal speed gain, so it is worth keeping even for latency-sensitive models.

Activation choice also matters. EfficientNet uses Swish (also called SiLU) instead of ReLU. Swish is smooth and non-monotonic, which helps optimization for deeper scaled models. I've found that switching Swish to ReLU6 for INT8 quantization readiness costs 0.5-0.8% accuracy but makes post-training quantization far more stable — a worthwhile trade on mobile. Training details are documented thoroughly in the PyTorch Documentation and the original EfficientNet implementation notes.

import timm

# EfficientNet-B3 with pretrained weights, adapting for transfer learning
model = timm.create_model('efficientnet_b3', pretrained=True, num_classes=0)
# Freeze early stem and blocks 0-2 for small data regimes
for name, param in model.named_parameters():
    if name.startswith('conv_stem') or name.startswith('blocks.0') or name.startswith('blocks.1'):
        param.requires_grad = False

# New classifier head for 10-class task with dropout
import torch.nn as nn
model.classifier = nn.Sequential(
    nn.Dropout(0.3),
    nn.Linear(model.num_features, 10)
)

# EfficientNet expects larger resolution per variant:
# B0: 224, B3: 300, B4: 380, B7: 600
# Always verify preprocessing: mean/std and interpolation matter

Practical Limits of Scaling EfficientNet

EfficientNet scales well on paper, but B5+ becomes memory-heavy due to 600px inputs and wide channels. Batch size must shrink, which destabilizes BatchNorm. I've had best results for B5 and above by switching to GroupNorm or using synchronized BatchNorm with gradient accumulation, and by using RMSProp or AdamW with exponential moving average of weights. For datasets where input objects are small (e.g., defect detection on 1024px images), EfficientNet's resolution scaling is genuinely useful. For 224px natural images, ResNet-50 vs EfficientNet-B3 is often a wash in accuracy, with EfficientNet winning on parameters but losing on straightforward deployability if your inference engine lacks optimized depthwise kernels.

Patch-Based Self-Attention in Vision Transformers: Trading Inductive Bias for Global Context

Vision Transformers remove convolutions entirely. An image is split into non-overlapping patches (typically 16x16), each patch is linearly projected to a token embedding, positional embeddings are added, and the resulting sequence is processed by a standard transformer encoder. A learnable [CLS] token is prepended, and its final state serves as the image representation for classification. This design makes almost no assumptions about locality or translation equivariance — properties that CNNs bake in. The model must learn spatial relationships from data.

That lack of inductive bias is both the strength and weakness. With enough data, the self-attention mechanism can model long-range dependencies from the first layer, whereas a CNN needs depth to grow its receptive field. On JFT-300M, a ViT-H/14 surpasses the best CNNs. On ImageNet-1k from scratch without strong augmentation, a ViT-B/16 underperforms a ResNet-50. I've found that threshold sits around 10-20 million labeled images, or ImageNet-1k with heavy regularization (RandAugment, Mixup, CutMix, stochastic depth) and distillation as in DeiT.

Positional Embeddings and Patch Size Trade-offs

Patch size controls sequence length and thus compute. For a 224x224 image, 16x16 patches yield 196 tokens; 32x32 yields 49 tokens. Smaller patches preserve fine detail and improve accuracy but attention scales quadratically with tokens (O(n^2 * d)). In practice, ViT-B/16 is about 4x slower to train than ViT-B/32 but 2-3% more accurate on ImageNet. Learned 1D positional embeddings work surprisingly well, but for variable resolutions at test time, you need to interpolate them. I've seen accuracy drop 5% when evaluating a 224px model at 384px without interpolating position embeddings correctly. Isotropic ViTs keep resolution constant throughout, unlike hierarchical variants like Swin which reintroduce locality and multi-scale features.

import timm

model = timm.create_model('vit_base_patch16_224', pretrained=True)

# Inspect patch embedding and interpolate pos_embed for higher resolution
# Useful when fine-tuning at 384px after pretraining at 224px
def interpolate_pos_embed(model, new_size=384):
    pos_embed = model.pos_embed  # [1, 197, 768] for 224
    # Implementation interpolates the grid portion (excluding CLS token)
    # using bicubic interpolation; see timm's checkpoint_filter_fn
    pass

# Optimizer: ViTs prefer AdamW with weight decay decoupled from gradient
# ViTs are sensitive to learning rate warmup
import torch.optim as optim
optimizer = optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.05)
# Use cosine decay with 10k steps warmup for stable early training

Why ViTs Win on Transfer for Large-Scale Pretraining

When pretrained on ImageNet-21k or larger, ViTs transfer exceptionally well. The global attention captures shape and context that helps on downstream tasks like fine-grained classification or multi-label tagging. However, for dense tasks like detection or segmentation, plain ViT requires adaptation (e.g., ViTDet) because it lacks a feature pyramid. Hierarchical transformers (Swin, ConvNeXt-influenced variants) close this gap. If your workflow relies on Transfer Learning in Practice: Fine-Tuning Pretrained Models Effectively, test both a ConvNeXt or EfficientNet and a ViT-B; the winner often depends on whether your target task needs texture (CNNs excel) or shape/context (ViTs excel).

Training Dynamics and Data Hunger: Where CNNs and ViTs Diverge in Practice

Training ResNet, EfficientNet, and ViT feels different in ways that affect hyperparameter choices and compute budgets. ResNets are forgiving. With SGD, momentum 0.9, weight decay 1e-4, and a step or cosine schedule, they converge within 90-120 epochs on ImageNet and tolerate batch sizes from 32 to 1024 with linear LR scaling. EfficientNets are more sensitive: RMSProp with decay 0.9, epsilon 1e-3, and exponential moving average gives best results, and they benefit from siamese augmentations like AutoAugment. Over-regularization hurts small EfficientNets quickly.

ViTs are data-hungry and optimization-sensitive. From scratch on ImageNet-1k, I have only been able to match ResNet accuracy by using AdamW, strong augmentation (RandAugment magnitude 9, Mixup 0.8, CutMix 1.0), stochastic depth 0.1-0.3, label smoothing 0.1, and 300 epochs with 10k warmup steps. Without pretraining, ViT training loss decreases slowly for the first 50 epochs, then drops — a pattern rarely seen with CNNs. Learning rate is critical: ViT-B/16 peaks around 3e-4 to 1e-3 with AdamW, while ResNets prefer 0.1 with SGD. Mixing them up guarantees failure.

Regularization and Augmentation Strategies That Actually Help

For CNNs, basic augmentation (random crop, horizontal flip, color jitter) plus label smoothing is often sufficient up to 10k images per class. For ViTs, heavy augmentation is mandatory to compensate for weak inductive bias. DeiT showed that distillation from a CNN teacher lets a ViT learn locality biases indirectly, improving data efficiency by a large margin. I routinely use a RegNetY-16GF or EfficientNet-B3 teacher to distill into DeiT-S for small datasets — it cuts required data by roughly half. Recent papers on arXiv Machine Learning also confirm that masked autoencoder pretraining (MAE) lets ViTs learn useful representations from unlabeled images, which is valuable when labels are scarce.

Transfer Learning: Fine-Tuning Protocols Compared

Fine-tuning differs sharply. For ResNet/EfficientNet, I freeze the stem and first stage for 5-10 epochs (linear probing), then unfreeze with a 10x lower learning rate for the backbone than the head (e.g., backbone 1e-4, head 1e-3). For ViT, full fine-tuning with a very low LR (1e-5 to 5e-5) and no freezing works better; layer-wise LR decay (0.65-0.75 per layer) preserves pretrained features while adapting deep layers. I also keep weight decay higher for ViTs (0.05) during fine-tuning, whereas CNNs use 1e-4. One consistent finding: ViTs forget less in few-shot settings when using cosine schedule with warmup, while CNNs adapt faster with one-cycle policy.

Latency, Throughput and Memory: Choosing an Architecture for Production Deployment

Accuracy alone rarely decides production architecture. I evaluate three metrics: latency (ms per image, batch=1), throughput (images/sec at max batch), and memory footprint (peak VRAM and model size). The results vary by hardware. On an A100 GPU with TensorRT or mixed precision, EfficientNet-B0 to B3 offer the best accuracy-per-latency ratio for online serving. ResNet-50 is slightly faster at batch=1 due to highly optimized 3x3 kernels and no depthwise ops, which are memory-bound. ViT-B/16 throughput collapses at larger batch sizes because attention's quadratic memory dominates — you hit OOM before saturating compute.

On mobile and edge CPUs, depthwise convolutions in EfficientNet are not always faster. Many ARM kernels are still optimized for dense 3x3 convolutions, so a ResNet-18 or MobileNetV3 can outperform EfficientNet-B0 in wall time despite higher FLOPs. Vision Transformers are generally unsuitable for edge without distillation or hybrid designs (e.g., MobileViT, EdgeViT) because attention is not cache-friendly and layer norm is slower than batch norm on DSPs. For quantized deployment, CNNs are mature: INT8 post-training quantization with PyTorch Documentation workflows loses <1% accuracy for ResNet-50 with proper calibration. ViT INT8 is harder — softmax and GELU cause quantization error amplification, requiring quantization-aware training or FP16 instead.

*ResNet-50 range reflects baseline 76.1% vs modern training recipes (strong augmentation, 300 epochs, label smoothing) reaching ~80.4% without architecture change. This variability is why training protocol matters more than model card numbers.

For deployment planning, start with latency constraints and work backward. If you need <15ms at batch=1 on a T4 GPU, choose ResNet-50 or EfficientNet-B0/B1 with TensorRT FP16. If you need highest accuracy for offline batch processing and have ample VRAM, fine-tune a ViT-L/16 or Swin-B. I've found that hybrid CNN-Transformer architectures (e.g., ConvNeXt, CoAtNet) often provide the best compromise for detection and segmentation — they retain convolutional inductive biases for early feature extraction while using attention for global reasoning, and they quantize better than pure ViTs.

Frequently Asked Questions