Working with Large Language Models
LLMs inverted the old workflow: the model arrives already trained, and the engineering is everything around it — feeding it the right context, constraining its output, measuring whether it's actually good, and paying for it sanely. Treat an LLM as a very capable, occasionally overconfident text function, and build systems that assume both halves of that sentence.
Prompting is interface design
A production prompt is a spec, not a wish: role and constraints stated plainly, the output format pinned down (JSON with a schema — modern APIs enforce structured output natively), two or three worked examples for anything subtle, and an explicit escape hatch — "if the answer isn't in the provided context, say so." That last line, plus context in the prompt, removes most hallucination in practice. Version prompts in git like the code they are; a one-word change can move quality double digits.
RAG: give the model your facts
Models know the public internet as of their training cutoff. They do not know your runbooks, tickets, or configs. Retrieval-augmented generation closes the gap without training anything:
- Index: chunk documents, compute embeddings (vectors where similar meaning lands nearby), store them — pgvector, Chroma, or SQLite variants; you rarely need exotic infrastructure.
- Retrieve: embed the user's question, fetch the most similar chunks; hybrid (keyword + vector) search beats either alone on technical text full of acronyms and hostnames.
- Generate: answer from the retrieved context, citing which chunk supports which claim.
When RAG disappoints, the retrieval is almost always the culprit, not the model — debug what got fetched before blaming what got written.
Fine-tune, or just write a better prompt?
In order of preference: better prompt → RAG → fine-tuning. Fine-tune (LoRA on an open model, cheaply) to change a model's behavior — house style, a niche format, a specialized classification. Don't fine-tune to add knowledge; facts belong in retrieval, where updating them is a re-index instead of a training run. Local open-weight models via Ollama or vLLM make sense for privacy-sensitive pipelines, high-volume cost control, and air-gapped environments; frontier APIs still win on the hardest reasoning.
Evaluation, or it didn't happen
The demo always works. The discipline that separates AI engineering from AI theater:
- Build a golden set — 50–200 real inputs with known-good answers, growing every time production surprises you.
- Score every prompt/model/pipeline change against it — exact checks where possible, an LLM judge with a rubric where not (spot-check the judge; it drifts toward generosity).
- Log production inputs and outputs; sample and review weekly. Regressions arrive silently, wearing an API version number.
Security is part of the job now
Anything an LLM reads can instruct it — a webpage, a retrieved document, a pasted email can carry "ignore your instructions and…" Prompt injection is unsolved; engineer accordingly: least-privilege tools, no secrets in context, human confirmation for consequential actions, and treat model output as untrusted input to downstream systems — the same posture you'd take with any user-supplied data.
Checkpoint
Build a small RAG assistant over your own notes or docs: index, hybrid retrieval, cited answers, a 50-question golden set, and a measured before/after for one prompt change. Under a hundred lines with modern libraries — and it exercises every concept above.