Synthetic Training Data with GANs: When It Works
Synthetic data generation is tempting. Instead of collecting and labeling thousands of real images, why not generate them? GANs can produce photorealistic images that look indistinguishable from real photos. The promise is unlimited labeled training data for free.
Reality is more complicated. I've used GAN-generated synthetic data in production training pipelines for three different projects. One worked spectacularly. Two were expensive failures. The difference came down to understanding what GANs actually learn and where the generated distribution diverges from reality.
GAN Training Basics
For training data generation, conditional architectures give you control over what the generator produces. StyleGAN2-ADA is still the strongest starting point for high-resolution outputs. For domain-specific generation, Pix2Pix or CycleGAN variants work better because they preserve structural constraints.
import torch
import torch.nn as nn
class Generator(nn.Module):
def __init__(self, latent_dim=512, img_channels=3):
super().__init__()
self.mapping = nn.Sequential(
nn.Linear(latent_dim, 512), nn.LeakyReLU(0.2),
nn.Linear(512, 512), nn.LeakyReLU(0.2),
nn.Linear(512, 512), nn.LeakyReLU(0.2),
)
self.synthesis = nn.ModuleList([
SynthBlock(512, 512),
SynthBlock(512, 256),
SynthBlock(256, 128),
SynthBlock(128, 64),
SynthBlock(64, 32),
])
self.to_rgb = nn.Conv2d(32, img_channels, 1)
def forward(self, z):
w = self.mapping(z)
x = self.synthesis[0].const
for block in self.synthesis:
x = block(x, w)
return torch.tanh(self.to_rgb(x))Training GANs is notoriously unstable. Mode collapse, training divergence, and checkerboard artifacts are common. Practical tips that helped us:
- Use ADA (Adaptive Discriminator Augmentation) when training on small datasets under 5000 images. It prevents discriminator overfitting.
- Monitor FID during training. FID below 20 indicates good quality; below 10 is excellent. But low FID doesn't guarantee useful training data.
- Train the discriminator more frequently than the generator (2:1 or 3:1 ratio) when the generator is winning too easily.
Where Synthetic Data Succeeds
The project that worked was defect detection for electronics manufacturing. We had 200 real images of PCB solder defects across 8 types. Some defect types had fewer than 10 examples. Training any modern detection model on 10 images per class is hopeless.
We trained a conditional GAN on the real defect images, generating 500 synthetic examples per type. Critically, we used these as augmentation — mixing synthetic and real at a 3:1 ratio in training batches. The model with synthetic augmentation achieved 0.82 mAP versus 0.61 mAP with real data alone.
The key insight: synthetic data works best when it supplements real data for underrepresented classes. Using synthetic data as 100% of training — replacing real data entirely — consistently performs worse than mixing strategies.
See also: Fine-Tuning Language Models with LoRA: Rank Selection and La.
Where Synthetic Data Fails
The first failure was generating synthetic face images for recognition. StyleGAN2 produces incredibly realistic faces. FID scores looked great. But the generated faces lacked real camera capture characteristics — specific noise patterns, motion blur profiles, shadow behavior under fluorescent lighting. The recognition model trained on synthetic faces performed 15% worse on real security camera footage.
The second failure was subtler. We generated synthetic satellite imagery for building segmentation. The images looked realistic and building geometries were plausible. But the GAN learned the wrong shadow angles — shadows consistent with the training data's geographic location and time of year, not the target deployment location. Shadow cues are a strong feature for building segmentation, and the model learned wrong shadow priors.
Evaluation Beyond FID
FID measures distribution-level similarity. It's useful but insufficient. We use three evaluation methods together: FID for distribution quality, Classification Accuracy Score (train on synthetic, test on real) for downstream task utility, and human expert review where a domain expert rates 50 random synthetic images on a 5-point scale.
The Classification Accuracy Score is most informative — it directly measures whether training on synthetic data improves performance on real data. If CAS is below the real-data-only baseline, your synthetic data is hurting.
Our production pipeline: train the GAN on curated real data, generate a candidate pool (10x the needed quantity), filter using a quality classifier, have domain experts review a sample, mix filtered synthetic with real data at experimentally determined ratios, train the downstream model, compare against a real-data-only baseline. It's expensive but when it works, it solves otherwise impossible data scarcity problems.
Conditional Generation for Controlled Output
Unconditional GANs generate random images from the learned distribution. For training data generation, you need control over what gets generated. Class-conditional generation (specifying the output category) is the minimum requirement. Attribute-conditional generation (controlling specific properties like defect size, position, or severity) is even more useful.
For a related perspective, see Data Augmentation Strategies That Actually Improve Model Rob.
For image-to-image tasks, Pix2Pix trains on paired data: input-output image pairs where the input might be a layout map and the output is the realistic image. When paired data isn't available, CycleGAN learns the mapping between two unpaired domains using cycle consistency loss.
# CycleGAN training loop (simplified)
for real_A, real_B in dataloader:
# G_AB generates domain B from domain A
fake_B = G_AB(real_A)
# G_BA reconstructs domain A from fake B
cycled_A = G_BA(fake_B)
# Cycle consistency: reconstructed should match original
cycle_loss_A = F.l1_loss(cycled_A, real_A)
# Same in reverse direction
fake_A = G_BA(real_B)
cycled_B = G_AB(fake_A)
cycle_loss_B = F.l1_loss(cycled_B, real_B)
# Adversarial losses from discriminators
d_loss_B = discriminator_B(fake_B)
d_loss_A = discriminator_A(fake_A)
total_loss = d_loss_A + d_loss_B + 10 * (cycle_loss_A + cycle_loss_B)The cycle consistency weight (lambda, typically 10) balances realism versus faithfulness. Lower lambda produces more realistic but less structurally accurate images. Higher lambda preserves structure but may produce blurry results. We tune this on a per-project basis, starting at 10 and adjusting based on visual inspection of the generated images.
Diffusion Models as an Alternative
Diffusion models (Stable Diffusion, DALL-E) have largely surpassed GANs for general image generation quality. For training data generation specifically, diffusion models offer better mode coverage — they don't suffer from mode collapse the way GANs do, meaning they generate more diverse outputs from the same training data.
The trade-off is generation speed. A GAN generates an image in a single forward pass (milliseconds). A diffusion model requires 20-50 denoising steps (seconds per image). When you need 10,000+ synthetic images, this speed difference matters. We generate overnight in batch rather than on-demand. Quality-wise, diffusion models produce more varied and realistic images for most domains, making them our current default for new synthetic data projects.
Legal and Ethical Considerations
Synthetic training data raises questions about the original data's licensing. If your GAN is trained on copyrighted images, the generated images may inherit legal restrictions. For medical imaging, synthetic data derived from patient scans requires the same IRB approval and data use agreements as the original scans. We consult with legal counsel before any synthetic data project and maintain clear documentation of the training data provenance for every GAN we deploy.
Bias is another concern. A GAN trained on biased data reproduces and potentially amplifies those biases. For face-related applications, this means a GAN trained predominantly on one demographic will generate predominantly that demographic, skewing any downstream model trained on the synthetic data. Auditing the generated data distribution for representativeness is a required step in our pipeline, not an optional nice-to-have.
This connects to the ideas in Anomaly Detection in High-Dimensional Industrial Sensor Data.
Practical Tips for GAN Training
GAN training is fragile. A few practical tips that save significant debugging time:
Start with a small resolution (64x64 or 128x128) and scale up only after the low-resolution model converges. Progressive growing — the technique from ProGAN — adds higher-resolution layers gradually during training. Even if you don't use explicit progressive growing, training at low resolution first and then fine-tuning at high resolution converges faster and more reliably than training at full resolution from scratch.
Use spectral normalization in both the generator and discriminator. It stabilizes training by constraining the Lipschitz constant of each layer, preventing the discriminator from becoming too powerful and collapsing the generator's gradients. It's a one-line change per layer and virtually eliminates training divergence in our experience.
Log images frequently — every 500-1000 training steps. The loss curves for GANs are notoriously uninformative. The generator loss goes down while the discriminator loss goes up, or both oscillate seemingly randomly, and the images might be getting better or worse regardless. Visual inspection of generated samples at regular intervals is the only reliable training monitor.
Fix a set of latent vectors (we use 64) and generate images from those fixed vectors at each logging interval. Watching the same set of latent codes evolve over training reveals mode collapse (all 64 images converge to the same output), training instability (images oscillate between two modes), and progressive quality improvement that the loss curves don't capture.
Dataset Curation for GAN Training
The quality of your GAN's output is bounded by the quality of your training data. Garbage in, garbage out — but for GANs, the effect is amplified because the generator can't produce details that aren't represented in the training distribution.
Remove duplicates and near-duplicates from the training set. A dataset with many similar images biases the GAN toward those specific images, reducing output diversity. We use perceptual hashing to find and remove images that are too similar — specifically, images with a perceptual hash distance below 8 bits get deduplicated.
Ensure consistent image quality across the training set. A few low-quality images (blurry, overexposed, low resolution) contaminate the generated distribution. We filter training images using BRISQUE (a no-reference image quality metric) and remove the bottom 5% before training. This simple filtering step consistently improves FID scores by 5-15%.