Beyond Matrix Factorization
If you're still running a recommendation system based on collaborative filtering with matrix factorization, you're leaving performance on the table. That isn't to say CF doesn't work. It does, and for many use cases it's perfectly adequate. But the two-tower architecture has become the standard approach for large-scale recommendation systems, and for good reason: it decouples candidate generation from scoring in a way that scales to hundreds of millions of items.
The Two-Tower Architecture Explained
The core idea is simple. You build two separate neural networks, one that encodes the user (the query tower) and one that encodes the item (the candidate tower). Each tower maps its input to a fixed-dimensional embedding vector. At serving time, you compute the user embedding, then find the nearest item embeddings using approximate nearest neighbor search.
import torch
import torch.nn as nn
class UserTower(nn.Module):
def __init__(self, num_users, num_features, embed_dim=128):
super().__init__()
self.user_embed = nn.Embedding(num_users, 64)
self.feature_net = nn.Sequential(
nn.Linear(num_features, 128),
nn.ReLU(),
nn.Linear(128, 64),
)
self.projection = nn.Sequential(
nn.Linear(128, embed_dim),
nn.LayerNorm(embed_dim),
)
def forward(self, user_id, user_features):
u_emb = self.user_embed(user_id)
f_emb = self.feature_net(user_features)
combined = torch.cat([u_emb, f_emb], dim=-1)
return self.projection(combined)
class ItemTower(nn.Module):
def __init__(self, num_items, text_dim, embed_dim=128):
super().__init__()
self.item_embed = nn.Embedding(num_items, 64)
self.text_net = nn.Sequential(
nn.Linear(text_dim, 128),
nn.ReLU(),
nn.Linear(128, 64),
)
self.projection = nn.Sequential(
nn.Linear(128, embed_dim),
nn.LayerNorm(embed_dim),
)
def forward(self, item_id, text_features):
i_emb = self.item_embed(item_id)
t_emb = self.text_net(text_features)
combined = torch.cat([i_emb, t_emb], dim=-1)
return self.projection(combined)The training objective is typically contrastive. You want the dot product between a user and their positive items to be higher than the dot product between the user and random negatives. The in-batch negatives trick makes this efficient: every other item in the training batch serves as a negative for each user.
Why Two Towers Instead of One
The key insight is that the item tower output can be precomputed and indexed. You run the item tower once over your entire catalog, store the resulting embeddings in a vector index, and then at serving time you only need to compute the user embedding (one forward pass through a small network) and do an ANN lookup. This brings the latency down to single-digit milliseconds even with catalogs of 100M plus items.
A single-tower model that takes both user and item features as input would need to score every candidate item individually. At 100M items, that's 100M forward passes per request. Not feasible for real-time serving.
Candidate Retrieval Strategy
In production, the two-tower model is usually just the first stage of a multi-stage pipeline. It retrieves 500-2000 candidates from the full catalog, and then a heavier re-ranking model scores those candidates with richer features like cross-attention between user and item representations.
We covered a related topic in Learning Rate Scheduling: Cosine Annealing, Warmup, and Cycl.
The retrieval stage needs to be fast and have high recall. It's better to include a few irrelevant candidates (the re-ranker will filter them) than to miss relevant ones. We typically target 90 percent plus recall at 1000.
Multiple retrieval sources often work better than a single one. In our system, we combine the two-tower model results with a popularity-based retrieval (recent trending items), a collaborative filtering retrieval, and a content-based retrieval (items similar to the user recent history). The union typically gives us 95 percent plus recall.
Training Considerations That Matter
The negative sampling strategy has the biggest single impact on model quality. Random negatives are easy but they make the task too simple. The model learns to distinguish popular items from random ones rather than learning fine-grained preferences. Hard negative mining using items that the user viewed but did not engage with produces much better embeddings, but you need to be careful about the ratio of hard to easy negatives.
Temperature scaling on the contrastive loss also matters more than most papers suggest. We found that a temperature of 0.05-0.1 works best for our domain. Too low and the model becomes overconfident; too high and it doesn't discriminate well between similar items.
One more thing: don't ignore the cold-start problem. New users and new items don't have learned embeddings. For new items, the content features in the item tower handle this reasonably well. For new users, you need a fallback. We use a popularity-weighted default embedding that gets progressively replaced by the learned embedding as engagement data accumulates. The transition should be smooth, not a hard cutover.
This connects to the ideas in Vector Database Architecture: Milvus, Pinecone, and Weaviate.
Embedding Space Visualization and Debugging
When a two-tower model isn't performing well, the first thing I do is visualize the embedding space. UMAP projections of the user and item embeddings tell you a lot about what the model has learned and where it's struggling.
Healthy embedding spaces have clear topical clusters (items of similar categories cluster together) but with smooth boundaries between clusters. If you see very tight, separated clusters with empty space between them, the model has memorized categories but hasn't learned fine-grained preferences within categories. If the space is a uniform blob with no structure, the model hasn't learned anything useful.
Another debugging technique: compute the nearest neighbors for specific items and check if they make sense. If Nike Air Max 90 is closest to Adidas Ultraboost and New Balance 990, the embedding space has learned brand/style similarity. If it's closest to kitchen sponge and car tire, something is very wrong with the training data or the loss function.
Multi-Objective Retrieval
In practice, you rarely optimize for pure relevance. Business constraints layer on top: diversity (don't show 10 sneakers in a row), freshness (promote new items), inventory (don't show out-of-stock items), and fairness (give exposure to new sellers).
These constraints can be applied at the retrieval stage or the re-ranking stage. Retrieval-stage constraints are faster but coarser. Re-ranking constraints are more precise but more expensive. We typically apply hard constraints (inventory, eligibility) at retrieval and soft constraints (diversity, freshness) at re-ranking.
This connects to the ideas in Prompt Engineering as Software Engineering: Version Control .
For diversity specifically, we use a maximal marginal relevance (MMR) approach in the re-ranker. After scoring all candidates by relevance, we greedily select the final list, at each step choosing the candidate that maximizes a weighted combination of relevance and diversity from the already-selected set. The diversity weight is tunable and we adjust it based on the query type. Exploratory queries (broad categories) get more diversity; specific queries (exact product searches) get less.
Evaluation Metrics Beyond Recall
Recall at K is the standard offline metric for retrieval, but it doesn't capture everything that matters in production. We track several additional metrics:
Coverage measures what fraction of the catalog appears in any user recommendations over a time period. Low coverage means the system has a popularity bias. It's showing the same 10,000 items to everyone while the other 990,000 items in the catalog get no exposure. This is a business problem that recall at K can't detect.
Serendipity measures whether the system recommends items the user wouldn't have found through simple browsing or search. High serendipity correlates with user satisfaction and long-term engagement. We approximate it by measuring how different the recommended items are from the user recent browsing history.
NDCG (Normalized Discounted Cumulative Gain) captures ranking quality, not just retrieval quality. Getting the right items into the top 1000 is good; getting them ranked correctly within the top 10 is what actually drives click-through rates. We measure NDCG at 5, 10, and 20.