Feature Engineering for Tabular Data: Encoding, Scaling and Selection Techniques

Most tabular datasets I see in production look nothing like the clean CSVs in tutorials. You get a mix of high-cardinality categoricals, skewed numerics with long tails, inconsistent units, and dozens of columns that add noise rather than signal. For tabular problems, the model you choose matters far less than how you prepare the features you feed it. In my experience, a well-engineered logistic regression or gradient-boosted tree will consistently beat a poorly-preprocessed neural network, and it will be cheaper to train, easier to debug, and more stable in production. This article walks through the three core pillars I use on every tabular project: encoding categorical variables so models can actually learn from them, scaling numeric features without introducing leakage, and selecting the subset of features that improves generalization.

Why Tabular Feature Engineering Still Dictates Model Performance

Unlike images or text where deep learning can learn representations directly from raw pixels or tokens, tabular data has no spatial or sequential structure for a model to exploit. Each column is a distinct semantic field with its own distribution, missingness pattern, and relationship to the target. If you pass a raw string column like payment_method or an unscaled income value ranging from 0 to 2 million into scikit-learn or PyTorch, most algorithms will either error out or learn a heavily biased function.

I've found that spending 60% of project time on data understanding and feature engineering yields the highest ROI. The reason is simple: algorithms are optimization routines. They can only optimize over the representation you give them. When you transform a zip code from a 40,000-value string into a meaningful target-encoded statistic or a clustered geographic embedding, you are injecting domain knowledge that no hyperparameter search can discover.

The Cost of Skipping Feature Engineering

On a recent churn prediction project with 120k rows and 85 raw columns, a baseline XGBoost model with minimal preprocessing achieved 0.71 AUC. After proper encoding, log-transforming skewed monetary features, and removing 31 redundant or leaky features, the same model reached 0.84 AUC without any tuning. The lift came entirely from representation, not model complexity. This pattern repeats across fraud detection, pricing, and lead scoring tasks.

What Happens Before Engineering

Feature engineering assumes your data is already typed and cleaned. That means handling missing values sensibly and understanding whether missingness is informative. For strategies on that step, see Missing Data Imputation: Statistical, ML-Based and Multiple Imputation Methods. Once you have a complete matrix, you can move to transformation.

Turning Categories Into Numbers: One-Hot, Ordinal and Target Encoding Trade-offs

Categorical encoding is where I see the most mistakes. The choice is not just about making strings numeric; it is about preserving the right inductive bias for your model. Linear models, distance-based models like k-NN, and tree-based models react very differently to the same encoding.

One-Hot Encoding for Low-Cardinality Nominals

One-hot encoding creates a binary column per category. It works well when cardinality is low, typically under 10-15 unique values, and when there is no ordinal relationship. For example, contract_type = {month-to-month, one_year, two_year} or region = {north, south, east, west}. It is interpretable and avoids imposing a false ordering. The downside is dimensionality. A column like city with 1,200 unique values would add 1,200 sparse columns, which blows up memory and causes tree models to split inefficiently.

In my experience, always set handle_unknown='ignore' in production. New categories will appear after deployment, and your pipeline should not crash when they do.

Ordinal and Frequency Encoding When Order Matters

If categories have a natural order, like education_level or satisfaction_score = {low, medium, high}, ordinal encoding with a meaningful mapping preserves that order for linear models. For high-cardinality nominals where one-hot is infeasible, frequency or count encoding can be a lightweight alternative. You replace each category with its occurrence count or proportion. It is useful for tree models but can leak target information if computed incorrectly.

Target Encoding for High-Cardinality Features

Target encoding replaces a category with the mean of the target for that category, often with smoothing to handle rare levels. It is powerful for features like merchant_id or zip_code but is also the easiest way to overfit. I have seen validation scores drop 10 points when target encoding was fit on the entire dataset before the train-test split. It must be computed strictly within cross-validation folds.

# Safe categorical encoding with scikit-learn
import pandas as pd
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder
from category_encoders import TargetEncoder
from sklearn.model_selection import StratifiedKFold

# One-hot for low cardinality, ignore unknowns in production
ohe = OneHotEncoder(handle_unknown='ignore', sparse_output=False)
ohe_features = ohe.fit_transform(df[['contract_type', 'payment_method']])

# Target encoding MUST be done inside CV to avoid leakage
# Example using out-of-fold encoding for a high-cardinality column
def oof_target_encode(X, y, col, n_splits=5, smoothing=20):
    oof = pd.Series(index=X.index, dtype=float)
    global_mean = y.mean()
    for train_idx, val_idx in StratifiedKFold(n_splits=n_splits, shuffle=True, random_state=42).split(X, y):
        te = TargetEncoder(cols=[col], smoothing=smoothing)
        te.fit(X.iloc[train_idx][[col]], y.iloc[train_idx])
        oof.iloc[val_idx] = te.transform(X.iloc[val_idx][[col]])[col]
    # Fill rare/unseen with global mean
    return oof.fillna(global_mean)

For a broader discussion on how encoding interacts with class imbalance, where target statistics can be skewed by rare positives, I often refer to Handling Imbalanced Datasets: SMOTE, Cost-Sensitive Learning and Evaluation.

Scaling Numeric Features Without Leaking Information or Distorting Distributions

Not every model needs scaling. Tree-based models like Random Forest, XGBoost, and LightGBM are invariant to monotonic transformations; they only care about split order. But linear models, logistic regression, SVMs, k-NN, and neural networks trained with gradient descent are highly sensitive to feature magnitude. A feature measured in dollars will dominate one measured in years if both are left unscaled.

Choosing Between Standardization, Min-Max and Robust Scaling

Standardization (zero mean, unit variance) is my default for linear and neural models when features are approximately Gaussian. Min-max scaling to [0,1] is useful when you need bounded inputs, such as for neural networks with sigmoid activations or when working with image-like tabular embeddings documented in the PyTorch Documentation. Robust scaling using median and IQR is essential when you have outliers. I've found that income, transaction amount, and tenure features almost always benefit from robust scaling or a log transform prior to standardization.

Do not apply standard scaling to heavily skewed distributions without first correcting skew. A log1p or Box-Cox transform followed by standardization often works far better. Otherwise, most of your standardized values will be crammed into a narrow range with a few extreme outliers stretching the variance.

The Golden Rule: Fit on Train Only

The most common leakage bug I review in pull requests is fitting a scaler on the full dataset before splitting. Your scaler learns mean, variance, and quantiles. Those are data-dependent statistics. If they include the test set, you have given your model information about the future. Always fit the scaler on the training fold and transform train, validation, and test sets with that fitted object. Scikit-learn's Pipeline enforces this automatically.

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, RobustScaler, PowerTransformer
from sklearn.linear_model import LogisticRegression

numeric_skewed = ['monthly_charges', 'total_charges']
numeric_normal = ['tenure_months', 'age']

preprocess = ColumnTransformer(transformers=[
    # Log + robust scale for long-tailed monetary features
    ('skewed', Pipeline([
        ('log', PowerTransformer(method='yeo-johnson', standardize=False)),
        ('scale', RobustScaler())
    ]), numeric_skewed),
    # Standard scale for roughly normal features
    ('normal', StandardScaler(), numeric_normal),
    # One-hot for low-cardinality categoricals
    ('cat', OneHotEncoder(handle_unknown='ignore'), ['contract_type', 'internet_service'])
])

# Full pipeline ensures no leakage during cross-validation
clf = Pipeline(steps=[
    ('preprocess', preprocess),
    ('model', LogisticRegression(max_iter=1000, penalty='l2'))
])

# Correct CV usage: preprocess is fit only on train folds
from sklearn.model_selection import cross_val_score
scores = cross_val_score(clf, X, y, cv=5, scoring='roc_auc')

Creating Interaction and Polynomial Features That Actually Help Tree and Linear Models

Raw columns rarely capture the full story. A model may need to know that high monthly charges combined with short tenure is a stronger churn signal than either feature alone. That is where constructed features help, but they must be constructed with hypothesis and validation, not brute force.

Domain-Informed Interactions

I prioritize interactions that have a business explanation. For telecom churn: charges_per_tenure = monthly_charges / (tenure_months + 1), has_fiber_and_high_charges, or support_calls_per_month. For credit risk: debt_to_income_ratio and utilization * delinquencies. These features are interpretable and often survive feature selection. I avoid generating all pairwise multiplications; with 50 features that creates 1,225 new columns, most of which are noise.

Polynomial Features and When to Use Them

Polynomial features are valuable for linear models that cannot learn non-linear boundaries. Adding x^2 allows a logistic regression to learn a U-shaped relationship, like risk being high for very young and very old customers. For tree models, polynomial features are usually redundant since trees already model non-linearities via splits. I've found that adding polynomial features before a tree model often increases training time with no gain and can even hurt generalization.

Binning and Discretization

Binning a continuous feature into quantiles or business buckets can help linear models capture non-monotonic effects and can make models more robust to outliers. For example, binning age into [18-30, 31-45, 46-60, 60+] lets a linear model assign independent weights per bucket. Use KBinsDiscretizer with strategy='quantile' to avoid empty bins, and always one-hot encode the bins rather than leaving them as ordinal integers if the bins are not strictly ordered in their effect.

Pruning the Noise: Filter, Wrapper and Embedded Methods for Feature Selection

More features is not better. Redundant or noisy features increase variance, slow training, inflate memory, and make explanations harder. After engineering, I typically have 80-150 features and need to prune down to the 20-40 that actually matter. Feature selection is not just about performance; it is about maintainability.

Filter Methods: Fast and Model-Agnostic

Filter methods score each feature independently of any model using correlation, mutual information, or chi-squared tests. They are fast and great for an initial pass. I use mutual information via mutual_info_classif because it captures non-linear relationships unlike Pearson correlation. In practice, I remove features with mutual information close to zero and those with pairwise correlation above 0.95, keeping the one with higher target correlation.

Wrapper Methods: Accurate but Expensive

Wrapper methods like recursive feature elimination (RFE) and forward selection train a model repeatedly with different subsets and keep the best performing set. They are more accurate because they account for feature interactions but can be prohibitively expensive on large datasets. I reserve wrappers for small to medium datasets where I can afford the compute, and I always use cross-validation inside the wrapper to avoid overfitting the selection.

Embedded Methods: Selection During Training

Embedded methods are my go-to for production. L1-regularized logistic regression or Lasso pushes coefficients of weak features to exactly zero, performing selection automatically. Tree-based feature importance and SHAP values are useful, but I treat raw impurity-based importance with caution; it is biased toward high-cardinality features. Regularized gradient boosting with early stopping often reveals which engineered features are truly utilized.

When you have hundreds of features, feature selection overlaps with dimensionality reduction. If you are considering projecting to a lower-dimensional dense space rather than selecting a subset, compare approaches in Dimensionality Reduction: PCA, t-SNE and UMAP Compared.

from sklearn.feature_selection import mutual_info_classif, SelectKBest
from sklearn.linear_model import LassoCV
from sklearn.feature_selection import SelectFromModel
import pandas as pd

# 1. Filter method: mutual information
mi_scores = mutual_info_classif(X_train_transformed, y_train, random_state=42)
mi_series = pd.Series(mi_scores, index=feature_names).sort_values(ascending=False)
keep_mi = mi_series[mi_series > 0.01].index.tolist()

# 2. Embedded method: L1 (Lasso) selection - features with non-zero coef are kept
lasso = LassoCV(cv=5, random_state=42, max_iter=5000).fit(X_train_transformed, y_train)
selector = SelectFromModel(lasso, prefit=True, max_features=40)
X_selected = selector.transform(X_train_transformed)
selected_features = [f for f, mask in zip(feature_names, selector.get_support()) if mask]

# 3. Stability check: re-fit with only selected features and validate
print(f"Selected {len(selected_features)} / {len(feature_names)} features")
print(selected_features[:10])

Detailed benchmarks on filter vs. embedded methods for tabular data are also discussed in recent work on arXiv Machine Learning, which I recommend for staying current on selection stability research.

Putting It Together: A Reproducible Preprocessing Pipeline That Won't Break in Production

The difference between a notebook and a production system is a reproducible pipeline that handles unseen data gracefully. Every transformation you apply must be serializable, versioned, and applied identically at training and inference.

ColumnTransformer and Pipeline as Contracts

Use ColumnTransformer to apply different transformations to different column types and wrap everything in a single Pipeline object. This ensures your encoding, scaling, imputation, and model are fit as one unit inside cross-validation, preventing leakage. Save the entire pipeline with joblib, not just the model. I have debugged too many incidents where the model was loaded but the scaler was refit incorrectly at serving time.

Handling Unseen Categories and Shift

Production data will contain new categories, missing columns, and distribution shift. Build defenses: set handle_unknown='ignore' for one-hot, use smoothing and priors for target encoding, clip outliers after scaling, and log feature distributions in production to detect drift. In my experience, adding a simple data validation layer that checks for cardinality explosion or missing rate spikes catches 90% of preprocessing failures before they reach the model.

Tracking and Documentation

Document why each feature was engineered, not just how. A feature store entry like avg_transaction_last_7d: mean of transaction_amount where event_time > now-7d, winsorized at 99th percentile to handle outliers, robust-scaled is infinitely more maintainable than an undocumented column named feature_27. Version your pipeline with your model. When you retrain, you should be able to reproduce the exact feature matrix from six months ago.

Frequently Asked Questions