A backup you have not restored is only a hope
Backups fail silently. The only way to know that yours works is to rebuild the system from it, on purpose, before you need to.
Les articles sont publiés en anglais.
Almost every company has backups. Far fewer have ever rebuilt a system from one. The difference is discovered on the worst possible day, when a job that has been reporting success for months turns out to have been saving an empty file.
The 3-2-1 rule
Keep at least three copies of your data, on two different kinds of storage, with one copy in another location. The rule is old and still sound. It protects against disk failure, against a fire or a provider outage, and against the mistake that deletes both the data and the copy next to it.
Ransomware changes the requirements
Ransomware operators look for backups first. If your backup server can be reached with the same administrator password as everything else, it will be encrypted along with everything else.
At least one copy must be out of reach: stored offline, or in object storage with an immutability lock that even an administrator cannot shorten. Use separate credentials for the backup system, and do not join it to your main directory.
Two numbers to agree on
Recovery point objective (RPO): how much recent data you can afford to lose. A nightly backup means up to a day of work.
Recovery time objective (RTO): how long you can afford to be down. Restoring several terabytes from another country may take much longer than anyone assumed.
These are business decisions, and shorter times cost more. Agree them with whoever will have to live with the result.
What people forget to back up
The database is rarely the problem. The missing piece is usually something else:
- configuration files and environment settings;
- encryption keys, which make the backup useless if they are lost;
- uploaded files stored apart from the database;
- DNS records and TLS certificates;
- the instructions for putting it all back together.
Test by restoring
Schedule a restore drill, at least quarterly for anything important. Take a clean machine and rebuild the service from backups only, following the written procedure. Time it. Have someone other than the author do it, because the author remembers the steps that were never written down.
Check that the result works: sign in, open a recent record, compare record counts against production.
Between drills, automate what you can. Verify that each backup completed, that its size is plausible, and that a sample restore can be read. Send an alert when a backup does not run, since silence is how these systems fail.
After a drill
Write down what went wrong and fix the procedure. The first drill always finds something, and that is the point of holding it on a day when nothing is on fire.
Rédigé par l'équipe d'ingénierie d'Avtim. Ce sont des informations générales ; vérifiez-les sur votre propre système avant de vous y fier.