---
title: Deploys and versioning
summary: Most outages start with a change. How to ship so that old and new code never mix, and why a bad release should reach a few machines before it reaches all of them.
topic: infrastructure
order: 2
updated: 2026-09-19
---

## What it is

A **deploy** replaces running code with new code. For a web app that means the server
process, and also every file the browser downloads: HTML, CSS, JavaScript. For a moment,
old and new exist side by side. Most deploy failures happen in that moment.

## Why it is a rule

- **Knight Capital, 1 August 2012.** A new trading release was copied to seven of eight
  servers. The eighth still had old, retired code, and the new release reused a flag that
  woke it up. That server sent millions of unintended orders. The firm lost about $440
  million in 45 minutes and had to be rescued.
- **CrowdStrike, 19 July 2024.** A faulty content update for a security product was pushed to
  every customer at once. It crashed about 8.5 million Windows machines, grounding flights and
  stopping hospitals and banks. The follow-up report committed to staged rollouts for that
  kind of update.

Neither was an attack. Both were one release reaching every machine, unchecked, at once.

## How to do it

### One build, every machine

Build once and ship the same artifact everywhere. A server that was "updated by hand" is how
Knight's eighth server kept its old code.

### Version every asset with the deploy

```html
<script src="/app.js?v=9f3c2e1"></script>
```

Rewrite every local asset URL to carry the deploy id. Then:

- A new deploy is a new URL, so browsers and CDNs fetch it fresh, with no purge needed for
  assets.
- The old URL can be cached forever (`immutable`), because its content never changes.
- A page and the assets it names ship together. An old page never loads new JavaScript, or
  the reverse.

### Tell open tabs a new version exists

A tab left open for a day is running yesterday's code against today's server. Push the
current version id over a socket; when it changes, offer a reload instead of letting the old
code keep sending requests the new server may not understand.

### Roll out in stages

- Ship to a small slice first (one server, one region, staff only), watch errors, then widen.
- Keep the previous build ready so rolling back is one command.
- A feature flag lets code ship switched off and be turned on later without a deploy.

### Check at boot, fail loudly

The new process checks that its secrets, database and required files exist before it takes
traffic. A process that boots with a missing secret should refuse to start, not serve errors
quietly.

## How we do it here

Every local asset URL on every page carries the deploy id, and page composition happens once
per deploy. The server announces its version over a socket so open tabs know when to reload.
Boot checks for missing secrets, and the edge cache is purged when the new process comes up.

## Benefits

- No mixed old and new code in a user's browser.
- Assets can be cached forever, so repeat visits are almost free.
- Staged rollouts turn a total outage into a small, reversible one.

## Disadvantages

- Every deploy invalidates every asset, so the first visit after a release downloads
  everything again, even files that did not change. Content hashes per file avoid this but
  need a build step.
- Staged rollouts are slower, and need monitoring good enough to spot a problem in the first
  slice.
- A reload prompt interrupts someone mid-task if it is not timed with care.

## Checklist

- One build artifact for every machine.
- Asset URLs carry a version; versioned assets are `immutable`.
- Open tabs learn about new versions.
- Rollback is one step and has been tried.

## Sources

- [US SEC — Order against Knight Capital Americas LLC, 2013](https://www.sec.gov/litigation/admin/2013/34-70694.pdf)
- [MDN — Cache-Control: immutable](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#immutable)
- CrowdStrike, *Preliminary Post Incident Review*, July 2024.