Learn / Infrastructure / Lesson 11
Backups and secrets
A backup you have never restored is a hope, and a password in a git repository is public. One company that nearly lost everything, one that did, and one that hid a breach for a year.
Last updated: 2026-09-19
What it is
Backups are copies of your data you can bring back after a mistake, a failure or an attack. Secrets are the keys to everything else: database passwords, API tokens, signing keys. One protects you from losing data; the other protects you from someone else taking it.
Why it is a rule
Backups
- GitLab, 31 January 2017. While fighting a spam attack late at night, an engineer ran a delete command on the wrong database server. When the team went to restore, they found that of five backup methods, none was working as intended. They recovered from a copy that happened to be six hours old, and published the whole recovery live.
- Code Spaces, 2014. An attacker got into the company's cloud account and, when refused a ransom, deleted the servers and the backups, which lived in the same account. The company shut down within days.
Secrets
- Uber, 2016. Attackers found cloud credentials in a private code repository, used them to reach a storage bucket, and took the data of 57 million riders and drivers. Uber paid the attackers to stay quiet and did not disclose it for a year.
How to do it
Backups: 3-2-1, and practise the restore
- 3 copies of the data, on 2 different kinds of storage, 1 of them away from the main account and provider.
- At least one copy the production credentials cannot delete. Code Spaces lost everything because one login could reach both.
- Restore on a schedule. Bring a backup up on a scratch server and check the row counts. The only proof a backup works is a restore.
- Before any risky migration, take a fresh copy, and keep it out of the code repository.
Secrets: never in git
.env
.env.*
!.env.example
*.pem
*.key- Secrets live in environment variables or a secret manager, never in the code.
.gitignoreblocks the usual secret files from the very first commit. A secret that was committed once is in the history forever, even if the file is deleted later. Rotate it.- A hard-coded fallback such as
process.env.KEY || 'dev-key'quietly becomes the production key the day someone forgets to set the variable. The server should refuse to start instead.
Least privilege
Each key can do only its job. The web app's database user cannot drop tables. A deploy token cannot read customer data. A leaked key then opens one door, not all of them.
Rotate
Change secrets on a schedule and immediately when someone leaves or a key may have leaked. Build it so rotating is a routine step, not an emergency.
How we do it here
Secrets live only in the hosting provider's environment, and the repository's ignore file has a secrets block at the top that is never shortened. The server lists any missing secret at boot. Database dumps taken before a migration are kept out of git because they hold real client rows.
Benefits
- A bad command or a ransomware attack becomes a bad afternoon, not the end of the business.
- A leaked key opens as little as possible, for as short a time as possible.
- Restore drills turn a panicked night into a checklist.
Disadvantages
- Off-site, undeletable backups cost money and set-up time.
- Restore drills take hours that nobody schedules unless they are required.
- More keys with less power each means more keys to manage and rotate.
- Backups clash with deletion requests: a deleted customer still exists in old backups until they age out, and the privacy policy must say so.
Checklist
- 3-2-1 backups, one copy the app's own credentials cannot delete.
- A restore has been tested recently.
- No secret in the repository or its history.
- Missing secrets stop the server from starting.
- Each key has the least power it needs.
Sources
- GitLab — Postmortem of database outage of January 31
- OWASP — Secrets Management Cheat Sheet
- US Federal Trade Commission, revised settlement with Uber over the 2016 breach, 2018.