Data Wrangling: Polars, Pandas, and Parquet
The dirty secret of the profession: most working hours go to reshaping data, not modeling it. Joins, filters, aggregations, timestamps that lie, categories spelled four ways, and the eternal question of what NULL meant to whoever produced the file. Master this and you are useful anywhere; skip it and no model can save you.
Think in split-apply-combine
Nearly every transformation decomposes into: split rows into groups, apply a computation per group, combine results. Once you see it, you stop writing loops and start writing declarations:
import polars as pl
df = pl.read_parquet("data/flows.parquet")
top_talkers = (
df.filter(pl.col("bytes") > 0)
.group_by("src_ip")
.agg(
pl.col("bytes").sum().alias("total_bytes"),
pl.col("dst_ip").n_unique().alias("unique_dests"),
)
.sort("total_bytes", descending=True)
.head(20)
)Declarative chains aren't just cleaner — Polars' lazy mode (pl.scan_parquet) reads them as a query plan and optimizes the whole pipeline before touching a byte, which is how a laptop chews through files that used to demand a cluster.
Polars or pandas?
Learn to read both, and default to Polars for new work: multi-threaded, memory-lean, and strict — it raises errors where pandas silently guesses, which converts three future debugging sessions into one present-tense fix. You'll still meet pandas everywhere (tutorials, legacy code, libraries), and interop is one call in each direction: df.to_pandas() / pl.from_pandas(df).
Files: Parquet in, CSV only at the edges
CSV has no types — everything is a string with ambitions — no compression, and no schema. Accept it from the outside world, then immediately convert:
pl.scan_csv("export.csv").sink_parquet("data/export.parquet")Parquet is columnar, typed, compressed, and fast to filter. Reading two columns from a 50-million-row Parquet file touches only those columns; the same query against CSV parses everything. For datasets that outgrow memory, DuckDB queries Parquet in place with plain SQL — duckdb.sql("SELECT ... FROM 'data/*.parquet'") — and covers the awkward middle ground below "buy a warehouse."
The cleaning discipline
- Never edit raw data.
data/raw/is immutable history; transformations produce new files indata/processed/. Reproducibility is a lineage property, not a virtue. - Interrogate NULLs. Missing-at-random, not-collected, and zero-meaning-absent are three different facts wearing one mask. Ask the producer; encode the answer explicitly.
- Parse timestamps to UTC on arrival, keep the original string until validated. Timezones cause more silent corruption than any other single feature of reality.
- Normalize categories early —
.str.strip_chars().str.to_lowercase()before the first group-by, or "Miami", "miami ", and "MIAMI" are three cities. - Validate at boundaries. Row counts before and after every join; a schema check (Pandera or plain asserts) at pipeline entry. Joins that silently drop or multiply rows are the field's most expensive bug.
Checkpoint
Take any real dataset over a million rows, land it as Parquet, and answer five questions about it with lazy Polars chains — including one join and one time-based aggregation — with row counts asserted along the way. When that feels routine, you own the 80% of the job nobody puts on posters.