Statistics That Earn Their Keep
You don't need a statistics degree. You need enough statistical honesty to stop yourself — and your stakeholders — from believing things the data doesn't say. That's a shorter list than the textbooks suggest, but every item on it gets used weekly.
Look before you summarize
Means and standard deviations describe well-behaved data, and real data is rarely well-behaved. Network traffic, incomes, response times, alert volumes — heavy-tailed, all of them, where the mean is a rumor started by outliers. Plot the distribution first (histogram, or ECDF when tails matter), then choose summaries that fit: medians and percentiles for skewed data, and p95/p99 — not averages — for anything a user waits on.
Uncertainty is part of the answer
"Conversion rose from 3.1% to 3.4%" is an incomplete sentence — on how many observations? The working tools:
- Standard error intuition: uncertainty shrinks with √n. Quadrupling data halves the noise; this single fact calibrates most "is this real?" questions.
- The bootstrap: resample your data 10,000 times, recompute the statistic, read the spread. No formulas to misremember, works for medians and weird metrics, ten lines of NumPy. When in doubt, bootstrap.
Hypothesis tests, without the cargo cult
A p-value answers one narrow question: if there were truly no effect, how surprising is this data? It is not the probability you're right, and p < 0.05 is a social convention, not a law of nature. The failure modes that actually burn people:
- Peeking: checking an experiment daily and stopping when it hits significance manufactures false wins. Fix the sample size (or use a sequential method) before starting.
- Multiple comparisons: test twenty metrics and one will "win" by chance. Correct for it, or pre-register the one metric that matters.
- Significance ≠ size: with enough data, a 0.01% difference is "significant." Report the effect size and its interval; let the business judge whether it's worth anything.
The Bayesian posture
You don't have to convert. But two Bayesian habits repay learning them: state your prior (extraordinary claims — "this one weird trick doubled retention" — need extraordinary evidence), and think in updated beliefs rather than binary verdicts. For security work, the base-rate lesson is mandatory: a 99%-accurate detector on events that are 1-in-10,000 produces ~99% false positives. That arithmetic, not model quality, is why alert fatigue exists (Part 9 builds on this).
Correlation, causation, and the confounder you didn't measure
Observational data can whisper causality but rarely prove it. Ice cream doesn't cause drownings; summer causes both. Before shipping a causal claim, ask: what third factor could drive both? Could the arrow point backward? Randomized experiments (A/B tests) exist because they answer these by construction — which is why companies run thousands of them a year rather than trusting dashboards.
Checkpoint
Take a real metric you care about. Plot its distribution, report the honest summary (median + p95, if skewed), bootstrap a confidence interval, and write one paragraph a non-statistician could read without being misled. That paragraph is the deliverable statistics exists for.