Learn / Infrastructure / Lesson 09
Deploys and versioning
Most outages start with a change. How to ship so that old and new code never mix, and why a bad release should reach a few machines before it reaches all of them.
Last updated: 2026-09-19
What it is
A deploy replaces running code with new code. For a web app that means the server process, and also every file the browser downloads: HTML, CSS, JavaScript. For a moment, old and new exist side by side. Most deploy failures happen in that moment.
Why it is a rule
- Knight Capital, 1 August 2012. A new trading release was copied to seven of eight servers. The eighth still had old, retired code, and the new release reused a flag that woke it up. That server sent millions of unintended orders. The firm lost about $440 million in 45 minutes and had to be rescued.
- CrowdStrike, 19 July 2024. A faulty content update for a security product was pushed to every customer at once. It crashed about 8.5 million Windows machines, grounding flights and stopping hospitals and banks. The follow-up report committed to staged rollouts for that kind of update.
Neither was an attack. Both were one release reaching every machine, unchecked, at once.
How to do it
One build, every machine
Build once and ship the same artifact everywhere. A server that was "updated by hand" is how Knight's eighth server kept its old code.
Version every asset with the deploy
<script src="/app.js?v=9f3c2e1"></script>Rewrite every local asset URL to carry the deploy id. Then:
- A new deploy is a new URL, so browsers and CDNs fetch it fresh, with no purge needed for assets.
- The old URL can be cached forever (
immutable), because its content never changes. - A page and the assets it names ship together. An old page never loads new JavaScript, or the reverse.
Tell open tabs a new version exists
A tab left open for a day is running yesterday's code against today's server. Push the current version id over a socket; when it changes, offer a reload instead of letting the old code keep sending requests the new server may not understand.
Roll out in stages
- Ship to a small slice first (one server, one region, staff only), watch errors, then widen.
- Keep the previous build ready so rolling back is one command.
- A feature flag lets code ship switched off and be turned on later without a deploy.
Check at boot, fail loudly
The new process checks that its secrets, database and required files exist before it takes traffic. A process that boots with a missing secret should refuse to start, not serve errors quietly.
How we do it here
Every local asset URL on every page carries the deploy id, and page composition happens once per deploy. The server announces its version over a socket so open tabs know when to reload. Boot checks for missing secrets, and the edge cache is purged when the new process comes up.
Benefits
- No mixed old and new code in a user's browser.
- Assets can be cached forever, so repeat visits are almost free.
- Staged rollouts turn a total outage into a small, reversible one.
Disadvantages
- Every deploy invalidates every asset, so the first visit after a release downloads everything again, even files that did not change. Content hashes per file avoid this but need a build step.
- Staged rollouts are slower, and need monitoring good enough to spot a problem in the first slice.
- A reload prompt interrupts someone mid-task if it is not timed with care.
Checklist
- One build artifact for every machine.
- Asset URLs carry a version; versioned assets are
immutable. - Open tabs learn about new versions.
- Rollback is one step and has been tried.
Sources
- US SEC — Order against Knight Capital Americas LLC, 2013
- MDN — Cache-Control: immutable
- CrowdStrike, Preliminary Post Incident Review, July 2024.