Classical Machine Learning Still Wins on Tables
Through every hype cycle, one result keeps replicating: on tabular business data — churn, fraud, risk, demand — gradient-boosted trees match or beat deep learning while training in minutes on a laptop and explaining themselves afterward. Classical ML isn't the beginner track; for most structured-data problems it is the destination.
The supervised learning loop
Everything in this part is one loop: split data honestly, fit on train, tune on validation, report on a test set touched exactly once.
from sklearn.model_selection import train_test_split
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import average_precision_score
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=42)
model = HistGradientBoostingClassifier(max_iter=300)
model.fit(X_tr, y_tr)
print(average_precision_score(y_te, model.predict_proba(X_te)[:, 1]))Baseline first, always: a majority-class guesser or single-feature rule. A model that can't visibly beat the dumb baseline is a rumor with a pipeline.
Where models actually fail: leakage
Data leakage — information from the future or the label sneaking into features — is the most expensive bug in applied ML, because it makes offline results spectacular and production results imaginary. The classics:
- A feature computed after the outcome ("account_closed_date" predicting churn — flawlessly).
- Preprocessing fit on the full dataset before splitting (the test set contaminates the scaler). Fix: put every transform inside a scikit-learn
Pipeline, so fitting happens only on training folds. - Random splits on time-ordered data — training on Tuesday to predict Monday. Time-based data gets time-based splits, full stop.
- Duplicate or near-duplicate rows straddling the split.
The smell test: if results look too good, they are. Investigate before celebrating.
Metrics that match reality
Accuracy dies on imbalance — 99.9% accuracy on 0.1% fraud is the model saying "nothing is fraud" with confidence. Use precision/recall and the precision-recall curve for rare classes; pick the operating threshold by the actual cost of each error (a missed fraud vs. a blocked customer are not symmetric). And calibrate: predict_proba is only a probability if you've checked it against reality (sklearn.calibration).
The tabular pecking order
- Logistic/linear regression — the interpretable baseline; sometimes it just wins, and it always explains itself.
- Gradient-boosted trees (scikit-learn's HistGB, XGBoost, LightGBM) — the workhorse. Handles mixed types, missing values, and nonlinearity without ceremony.
- Everything else — earns its place only by beating the boosted baseline in honest cross-validation, by more than the bootstrap noise (Part 4).
For explanations, SHAP values show which features drove each prediction — increasingly a compliance requirement, not a courtesy.
Checkpoint
Pick a real binary-outcome dataset. Build: baseline → logistic regression in a Pipeline → boosted trees, with stratified cross-validation, PR-AUC, a leakage audit written down, and one paragraph on which errors cost what. That artifact is a portfolio piece; most production models are exactly this, done carefully.