Treating Pipelines Like Production Systems
Lab / Educational
The premise
A genomics pipeline is a production system. It takes untrusted input, runs a chain of tools, and emits results that people make decisions on. Yet pipelines are routinely run with floating tool versions, unverified inputs, and no record of what actually executed. The Biocompute Object is one answer to this; the underlying discipline is borrowed wholesale from software operations.
Pin every version
“It worked last month” is not reproducibility. A pipeline that silently
pulls latest for its aligner or caller can produce different results run to
run. Pin exact versions — the same reason production deployments pin dependencies
rather than tracking a moving tag. Record them where the run records them, not in
someone's memory.
# Instead of whatever is on PATH, pin it:
bwa 0.7.17-r1188
samtools 1.9
# ...and capture these in the run's manifest (e.g. a BCO).Verify every input
Inputs get corrupted, truncated, or swapped. Checksum them before use — the same mindset behind the VCF Integrity Checker and the backup restore drill: don't assume a file is what it claims to be, verify it.
sha256sum -c inputs.sha256 # fail loudly before the pipeline runsRecord provenance
Provenance answers “what produced this result, with what inputs, using what tools?” — the lab equivalent of an audit log. When a result is questioned months later, provenance is the difference between re-deriving it in minutes and shrugging.
Closing the loop
Pinned versions, verified inputs, and recorded provenance together make a pipeline run reproducible, auditable, and defensible. None of this is exotic; it is the same operational hygiene any production stack expects. Applying it to the dry lab is simply refusing to hold scientific computing to a lower standard than we hold a web server.