Avoiding Validation Leakage in Small ML Experiments

2 min read

Skip to deep dive โ†’

Validation leakage usually shows up when preprocessing learns from the full dataset before the split. In small experiments, that can make a weak model look surprisingly strong.

The safest pattern is simple: split first, fit transforms only on the training fold, and keep the test set untouched until the very end.

from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

model = Pipeline([
    ("scale", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000)),
])

model.fit(X_train, y_train)
score = model.score(X_test, y_test)
print(f"Holdout accuracy: {score:.3f}")

When the dataset is tiny, cross-validation helps, but only if every step lives inside the CV loop. Otherwise the leakage just becomes harder to see.