---
title: Backups and secrets
summary: A backup you have never restored is a hope, and a password in a git repository is public. One company that nearly lost everything, one that did, and one that hid a breach for a year.
topic: infrastructure
order: 4
updated: 2026-09-19
---

## What it is

**Backups** are copies of your data you can bring back after a mistake, a failure or an
attack. **Secrets** are the keys to everything else: database passwords, API tokens, signing
keys. One protects you from losing data; the other protects you from someone else taking it.

## Why it is a rule

### Backups

- **GitLab, 31 January 2017.** While fighting a spam attack late at night, an engineer ran a
  delete command on the wrong database server. When the team went to restore, they found that
  of five backup methods, none was working as intended. They recovered from a copy that
  happened to be six hours old, and published the whole recovery live.
- **Code Spaces, 2014.** An attacker got into the company's cloud account and, when refused a
  ransom, deleted the servers **and the backups**, which lived in the same account. The
  company shut down within days.

### Secrets

- **Uber, 2016.** Attackers found cloud credentials in a private code repository, used them
  to reach a storage bucket, and took the data of 57 million riders and drivers. Uber paid
  the attackers to stay quiet and did not disclose it for a year.

## How to do it

### Backups: 3-2-1, and practise the restore

- **3** copies of the data, on **2** different kinds of storage, **1** of them away from the
  main account and provider.
- At least one copy the production credentials **cannot delete**. Code Spaces lost
  everything because one login could reach both.
- **Restore on a schedule.** Bring a backup up on a scratch server and check the row counts.
  The only proof a backup works is a restore.
- Before any risky migration, take a fresh copy, and keep it out of the code repository.

### Secrets: never in git

```gitignore
.env
.env.*
!.env.example
*.pem
*.key
```

- Secrets live in environment variables or a secret manager, never in the code.
- `.gitignore` blocks the usual secret files from the very first commit. A secret that was
  committed once is in the history forever, even if the file is deleted later. Rotate it.
- A hard-coded fallback such as `process.env.KEY || 'dev-key'` quietly becomes the
  production key the day someone forgets to set the variable. The server should refuse to
  start instead.

### Least privilege

Each key can do only its job. The web app's database user cannot drop tables. A deploy token
cannot read customer data. A leaked key then opens one door, not all of them.

### Rotate

Change secrets on a schedule and immediately when someone leaves or a key may have leaked.
Build it so rotating is a routine step, not an emergency.

## How we do it here

Secrets live only in the hosting provider's environment, and the repository's ignore file has
a secrets block at the top that is never shortened. The server lists any missing secret at
boot. Database dumps taken before a migration are kept out of git because they hold real
client rows.

## Benefits

- A bad command or a ransomware attack becomes a bad afternoon, not the end of the business.
- A leaked key opens as little as possible, for as short a time as possible.
- Restore drills turn a panicked night into a checklist.

## Disadvantages

- Off-site, undeletable backups cost money and set-up time.
- Restore drills take hours that nobody schedules unless they are required.
- More keys with less power each means more keys to manage and rotate.
- Backups clash with deletion requests: a deleted customer still exists in old backups until
  they age out, and the privacy policy must say so.

## Checklist

- 3-2-1 backups, one copy the app's own credentials cannot delete.
- A restore has been tested recently.
- No secret in the repository or its history.
- Missing secrets stop the server from starting.
- Each key has the least power it needs.

## Sources

- [GitLab — Postmortem of database outage of January 31](https://about.gitlab.com/blog/2017/02/10/postmortem-of-database-outage-of-january-31/)
- [OWASP — Secrets Management Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html)
- US Federal Trade Commission, revised settlement with Uber over the 2016 breach, 2018.