Dimensionality Reduction: PCA, t-SNE and UMAP Compared

Dimensionality reduction sits at a critical junction in most ML pipelines: your data lives in hundreds or thousands of dimensions, but your models, your visualizations, and your own intuition do not. In my experience running reduction pipelines on everything from single-cell RNA sequencing data to 2048-dimensional image embeddings, the choice between PCA, t-SNE, and UMAP is rarely about which method is "best" in absolute terms. It's about what structure you need to preserve, what computational constraints you're operating under, and whether the output is destined for a human eye or another model. This article breaks down the mechanics, trade-offs, and practical failure modes I've encountered with all three, so you can make that choice deliberately rather than by default.

The Linear Projection: How PCA Preserves Global Variance Through Eigendecomposition

Principal Component Analysis remains the workhorse for dimensionality reduction for one reason: it is deterministic, mathematically transparent, and optimal for a specific definition of information preservation. PCA finds an orthogonal linear transformation that projects your data onto directions of maximal variance. In practice, you are solving for the eigenvectors of the covariance matrix, or more stably, computing the Singular Value Decomposition (SVD) of your centered data matrix.

Given a centered data matrix X of shape (n_samples, n_features), PCA computes X = U Σ V^T. The columns of V are the principal components, and the diagonal of Σ contains singular values whose squares are proportional to explained variance. When you retain the top k components, you are guaranteeing the minimal reconstruction error in the L2 sense among all possible linear projections to k dimensions. No other linear method can do better.

When Linearity Becomes an Asset

That linearity is PCA's greatest strength and its fundamental limitation. Because it is a simple linear projection, PCA excels at preserving global structure. Distances between far-apart clusters in the original space remain far apart in the projection, and the axes themselves are interpretable as linear combinations of original features. I've found that for downstream model input, this property is invaluable. If you are reducing 500 engineered features to 50 before feeding a gradient boosting model, PCA gives you decorrelated, ranked features where each component's explained variance ratio tells you exactly how much signal you're discarding.

PCA also fails gracefully. It will never invent spurious clusters or tear continuous manifolds apart. What it will do is collapse non-linear structure. A classic Swiss roll manifold, when projected with PCA to 2D, will look like a smeared overlap because a linear plane cannot unroll it. If your data lies on a curved manifold, PCA will show you its shadow, not its shape.

# PCA: stable, reproducible pipeline with proper scaling
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline

# Crucial: PCA is variance-sensitive. Always standardize first
# unless features are already on a comparable scale.
pca_pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('pca', PCA(n_components=0.95, svd_solver='full', random_state=42))
])

# n_components=0.95 retains 95% variance - let the data decide k
X_reduced = pca_pipeline.fit_transform(X_train)

# Inspect the trade-off explicitly
pca = pca_pipeline.named_steps['pca']
print(f"Components retained: {pca.n_components_}")
print(f"Cumulative variance: {pca.explained_variance_ratio_.cumsum()[-1]:.4f}")

# For visualization, fit to 2 components separately - never use the 0.95 model for plotting
pca_2d = PCA(n_components=2).fit_transform(StandardScaler().fit_transform(X_train))

A common mistake I see is fitting PCA on the entire dataset before train/test splitting, which leaks test variance into the projection. Always fit the scaler and PCA on the training set only. Another nuance: for very high-dimensional sparse data (e.g., TF-IDF with 50k features), use TruncatedSVD instead of PCA directly, as it operates without centering and preserves sparsity. For preprocessing workflows that include scaling and selection, see how this step integrates with Feature Engineering for Tabular Data: Encoding, Scaling and Selection Techniques - reduction is most effective after your features are properly encoded and scaled.

t-SNE's Probability Graph: Capturing Local Neighborhood Structure at the Cost of Global Context

t-Distributed Stochastic Neighbor Embedding approaches the problem from an entirely different angle. Instead of preserving variance, it preserves neighborhood probabilities. Introduced by van der Maaten and Hinton, and detailed in the original work on arXiv Machine Learning, t-SNE converts high-dimensional Euclidean distances into conditional probabilities that represent similarities.

For each point x_i, it centers a Gaussian and computes p_j|i, the probability that i would pick j as its neighbor. The variance of that Gaussian (sigma_i) is adapted per-point to achieve a user-specified perplexity. These are symmetrized to p_ij. In the low-dimensional map, it places points y_i and defines a similar probability q_ij but using a Student-t distribution with one degree of freedom (a heavy-tailed distribution). It then minimizes the Kullback-Leibler divergence between P and Q via gradient descent.

That choice of a Student-t distribution in low dimensions is deliberate: it creates a crowding problem solution. By having heavier tails than a Gaussian, moderate distances in high dimensions can be represented by much larger distances in low dimensions, allowing clusters to separate. This is why t-SNE produces such visually compelling, well-separated clusters.

Perplexity, Early Exaggeration, and the Illusion of Clusters

In my experience, t-SNE is the most misinterpreted of the three methods. The hyperparameters dictate the output more than the data itself in some cases. Perplexity, which can be thought of as a smooth measure of the effective number of neighbors (typically 5-50), controls the balance between local and global aspects. I've found that a perplexity of 30 works for many datasets with a few thousand points, but for n=100,000, you often need perplexity 50-100 to avoid fragmenting large clusters.

Early exaggeration (usually factor 12 for 250 iterations) artificially inflates p_ij to force clusters to pull apart early, creating space for them to organize. Learning rate is also critical; a rate that is too low yields dense, collapsed balls, while too high creates noisy, shattered plots. Unlike PCA, t-SNE is stochastic and non-parametric - there is no transform function. You cannot fit on train and transform test. You must run it on the full set you want to visualize, which limits its use in production pipelines.

Most importantly: distances between clusters in a t-SNE plot are meaningless, and cluster sizes are not proportional to original variance. A t-SNE plot will happily create discrete-looking islands from continuous Gaussian noise if you tune perplexity oddly. I never use t-SNE for downstream modeling features, only for exploratory visualization and sanity checking. If you do use its 2D output as a feature, you are feeding your model an embedding optimized for visual crowding, not predictive signal.

# t-SNE: for visualization only - careful with hyperparameters
from sklearn.manifold import TSNE
from sklearn.preprocessing import StandardScaler

X_scaled = StandardScaler().fit_transform(X_train)

# For n < 10k, exact method is okay. For larger, use barnes_hut or openTSNE.
# Always run PCA to 50 dims first as recommended by van der Maaten
from sklearn.decomposition import PCA
X_pca50 = PCA(n_components=50).fit_transform(X_scaled)

tsne = TSNE(
    n_components=2,
    perplexity=30.0,        # increase with dataset size: 5-50 typical
    early_exaggeration=12.0,
    learning_rate='auto',   # 'auto' = n/12, or set 200-1000 manually
    n_iter=1000,
    init='pca',             # more stable than random
    random_state=42,
    n_jobs=-1
)
X_tsne = tsne.fit_transform(X_pca50)

# Pro tip: Run 3 times with different seeds. If cluster membership
# changes dramatically, you don't have stable structure - you have artifacts.

UMAP's Topological Logic: Balancing Local and Global Geometry via Fuzzy Simplicial Sets

Uniform Manifold Approximation and Projection was developed to address t-SNE's computational and theoretical limitations. Described formally in McInnes et al., UMAP is grounded in Riemannian geometry and algebraic topology. It assumes the data is uniformly distributed on a locally connected manifold and attempts to construct a fuzzy topological representation, then optimize a low-dimensional equivalent.

Conceptually, UMAP builds a weighted k-nearest neighbor graph where edge weights represent the likelihood that two points are connected. It defines a local notion of distance by normalizing relative to the distance to the nearest neighbor (rho), ensuring local connectivity. This fuzzy simplicial set is then optimized in low dimensions by minimizing cross-entropy between the high and low-dimensional fuzzy sets, not KL divergence. The optimization uses stochastic gradient descent with negative sampling, which is dramatically faster than t-SNE's approach.

The result is a method that preserves more global structure than t-SNE while still capturing fine local manifold detail. In my work embedding transformer hidden states, I've found that UMAP often retains the hierarchical relationship between clusters - for example, that two sub-clusters of documents are both part of a larger "sports" super-cluster - where t-SNE would place them arbitrarily far apart.

The n_neighbors and min_dist Controls That Define Your Manifold

UMAP's two primary knobs, n_neighbors and min_dist, map directly to the local-global trade-off. n_neighbors (default 15) controls the size of the local neighborhood used for manifold approximation. Low values (5-10) force UMAP to focus on very local structure, producing more, finer clusters. High values (50-200) incorporate broader topology, preserving global continuity at the cost of fine detail. I typically sweep 15, 50, and 100 when exploring a new dataset and compare.

min_dist (default 0.1) controls how tightly points are allowed to pack in the embedding. Low min_dist (0.0-0.1) yields tight, dense clusters excellent for clustering, while high min_dist (0.5-0.99) spreads points out, preserving broader density structure and making the visualization look more like PCA. Unlike t-SNE, UMAP is not inherently normalized to a small ball, so you can see meaningful variation in inter-cluster distances, though you should still not over-interpret them as true high-dimensional distances.

A major practical advantage: UMAP learns a parametric mapping via its underlying graph structure. The umap-learn implementation provides a transform method, so you can fit on training data and transform new test data - a requirement for any feature engineering step that will serve a model. For details on scalable implementations, the PyTorch Documentation and GPU-accelerated libraries like cuml and pytorch-umap are worth benchmarking if you are operating beyond 500k samples.

# UMAP: faster, transformable, and better global structure
import umap

# Fit on training data only for model pipelines
reducer = umap.UMAP(
    n_components=2,
    n_neighbors=30,      # 15 local default, 50+ for global structure
    min_dist=0.1,        # 0.0 tight clusters, 0.5 loose structure
    metric='cosine',     # 'euclidean' for dense tabular, 'cosine' for embeddings/text
    init='spectral',     # stable initialization
    random_state=42
)

X_train_umap = reducer.fit_transform(X_train_scaled)
X_test_umap = reducer.transform(X_test_scaled)  # Unlike t-SNE, this is valid

# For feature generation (not just viz), use n_components=10-50
# and set min_dist=0.0 to retain density for clustering
feature_reducer = umap.UMAP(n_components=15, n_neighbors=30, min_dist=0.0, random_state=42)
X_features = feature_reducer.fit_transform(X_train_scaled)

Computational Footprint and Hyperparameter Sensitivity Across Methods

The runtime and memory characteristics differ by orders of magnitude, which often decides the method before any theoretical argument does. PCA via randomized SVD (svd_solver='randomized') is O(n_samples * n_features * log(n_components)) and can handle millions of samples and thousands of features on a single machine in seconds to minutes. It is deterministic and requires essentially no tuning beyond choosing k via explained variance.

t-SNE with the exact method is O(n²), which becomes intractable beyond ~5,000 samples. The Barnes-Hut approximation brings this to O(n log n) but still struggles beyond 50k-100k points without subsampling. Modern implementations like openTSNE or multicore Barnes-Hut help, but a 100k point t-SNE run can still take 30-60 minutes, while UMAP completes in under 2 minutes for the same data. UMAP's complexity is approximately O(n log n) for graph construction plus SGD optimization, and it scales far better to large n and to moderate n_components > 3. Memory-wise, t-SNE stores a dense distance matrix in intermediate steps; UMAP operates on a sparse k-NN graph, making it far leaner.

Hyperparameter sensitivity inversely correlates with determinism. PCA has one parameter (k). UMAP has two dominant ones (n_neighbors, min_dist) plus metric and initialization. t-SNE has at least four interacting ones (perplexity, learning_rate, early_exaggeration, n_iter). I've found that t-SNE embeddings can change qualitatively with a perplexity shift from 30 to 50, while UMAP and PCA are more robust. For reproducible research, I always fix random_state, log the exact library version, and for t-SNE/UMAP, publish the full embedding with multiple seeds. Without that, your "discovery" of clusters may be irreproducible noise.

Choosing Your Reduction Strategy: A Pipeline Decision Framework

Instead of ranking the methods, I frame the choice as a decision tree based on the downstream consumer of the reduced data. If the consumer is a model, use PCA or a supervised/autoencoder method. If the consumer is a human analyst exploring data, use UMAP or t-SNE. Mixing these roles is where projects get into trouble.

Use PCA when you need an invertible, deterministic transform for preprocessing, denoising, or compression before a supervised learner; when interpretability of variance matters; or when you need to handle new data at inference without retraining the reducer. I also use PCA as a first step before t-SNE/UMAP itself - reducing from 1000 to 50 dimensions with PCA removes noise and speeds up the neighbor search for the non-linear methods by 3-5x without losing meaningful structure.

Use UMAP when you need to visualize both local and global structure, when you need to transform new points (e.g., embedding a live stream of documents and visualizing them), or when you want to use the embedding as a clustering pre-processing step. UMAP's output works well with HDBSCAN for density-based clustering because its min_dist parameter lets you control whether you preserve the density gradients that HDBSCAN needs.

Use t-SNE when your sole goal is high-quality, publication-ready visualization of local cluster structure for a static dataset under ~50k points, and you are willing to tune carefully. In my experience, for pure visual separation of well-defined clusters, a well-tuned t-SNE plot still often looks cleaner than UMAP, but that cleanliness is partly an artifact of its loss function emphasizing separation. Do not use t-SNE to argue about hierarchical relationships or distances between clusters.

A preprocessing detail that applies to all three: handle missing and noisy data before reduction. Distance metrics are extremely sensitive to imputed outliers. A single feature with 10% missing values imputed poorly can dominate nearest neighbor graphs. Establishing a robust imputation strategy first, as outlined in Missing Data Imputation: Statistical, ML-Based and Multiple Imputation Methods, will improve embedding quality far more than hyperparameter tuning.

Embedding Pipelines in Production: From Visualization to Model Input

The way you integrate reduction into a pipeline depends on whether the embedding is an endpoint or a feature. For visualization endpoints, the workflow is offline and exploratory: scale, optionally PCA to 50D, run UMAP/t-SNE to 2D, and plot with interactive tools like Plotly or Datashader for large n. I always color by known labels or by HDBSCAN clusters to validate whether the structure aligns with domain expectations, and I overlay trustworthiness and continuity scores (from scikit-learn's manifold.trustworthiness) to quantify how well neighborhoods are preserved. A trustworthiness below 0.85 for 2D suggests you are distorting heavily.

Avoiding Leakage and Misinterpretation at Inference

For model input, the pipeline must be serializable and leakage-free. That means your reduction step belongs inside a scikit-learn Pipeline or equivalent, fitted only on training folds during cross-validation. I've audited pipelines where UMAP was fit on the full dataset before splitting, inflating validation AUC by 0.05-0.08 because the graph had already seen test neighborhoods. The correct pattern is to fit reducer on X_train, transform X_train and X_val separately within each CV fold.

Another production pitfall: using 2D embeddings as features. Two dimensions rarely retain enough predictive signal. If you want features, generate 15-50 UMAP components or 50-200 PCA components and let a downstream model select. And consider whether you need unsupervised reduction at all. If you have labels, supervised methods like Linear Discriminant Analysis or a trained autoencoder often outperform PCA/UMAP as feature generators. Dimensionality reduction is a preprocessing choice, not a mandatory step - if your model (e.g., XGBoost with 200 features) handles high dimensions well, adding a reduction stage may just discard signal. Evaluate with and without it via rigorous cross-validation, and keep the simpler pipeline if performance is equivalent.

Frequently Asked Questions