From notebook to production: the checklist I run before launch

A model that works in a notebook has proven exactly one thing: it works, once, on your machine, on that data. Everything that decides whether it survives contact with production is still ahead of you.

Here is the checklist I work through. It is deliberately boring.

Reproducibility

• Training data snapshot is versioned and addressable, not "the export from March".
• Training is a script with pinned dependencies, runnable end to end by someone else.
• Every run logs parameters, metrics, and artifacts to a tracking server.
• The exact artifact serving traffic can be traced back to the run that produced it.

If you cannot rebuild the current production model from scratch, you do not have a model — you have a file.

Serving

• Inference is behind a versioned API with a schema and validation on input.
• Preprocessing code is shared between training and serving, not reimplemented. Training/serving skew is the most expensive bug in this field.
• Latency and throughput are measured under realistic concurrency, not one request at a time.
• Batch and real-time paths, if both exist, produce identical results on the same input. Verify this with a test.

Monitoring

• Input distribution monitoring, so drift is detected before accuracy visibly drops.
• Prediction distribution monitoring, which catches silent failures a health check never will.
• Business metric tracking alongside model metrics — the question is always whether the model is helping, not whether it is running.
• Alerting that names a human and a runbook.

Rollback and release

• Previous model version is kept warm and can be restored in minutes.
• New versions go out behind a shadow deployment or a traffic split, never a hard cutover.
• A documented rollback trigger: which metric, what threshold, who decides.

Security and governance

• No secrets in the training or serving code. Model artifacts live in access-controlled storage.
• PII handling reviewed for both training data and logged inference inputs — inference logs are the leak people forget.
• Prompt-injection and output-handling review for anything LLM-based touching external content.
• Record of what data trained the model and who approved its deployment.

Handover

• A runbook a teammate can follow at 3am.
• Retraining trigger and procedure written down.
• Known limitations documented honestly, including the inputs where the model is unreliable.

None of this is glamorous, and all of it is the difference between a model that runs for two years and one that gets quietly turned off after a bad month.