A backup is not a backup until you have restored it

Every operations team has a backup job, and most of those jobs report success every night. A green tick in a scheduler says one thing: a process ran and exited without an error. It does not say the copy is complete. It does not say the copy can be read, or that the app will start on top of it, or how long any of that takes.

The only way to know a backup works is to restore it. Until you have done that, what you have is a file and a hope. This post is about the gap between the two, and about why we put restore tests on a calendar instead of leaving them for the day something breaks.

What a successful backup job actually proves

A backup job reports on its own behaviour. It read some data, wrote some bytes to a destination and did not crash. That is useful, but it is a narrow claim, and the failures that hurt most sit outside it.

A few common ways a nightly job succeeds while the backup is useless:

• The wrong thing is backed up. An app moves its uploads to a new directory after an upgrade. The job keeps copying the old, now empty, path. Every run succeeds.
• The database is copied as files while it is running. A raw copy of a live database directory can be internally inconsistent. It looks complete and may refuse to start, or start with silent corruption.
• The encryption key lives on the machine being protected. The backup is encrypted, as it should be. The key sits on the server that just failed. The copy is intact and unreadable.
• The app needs something the backup does not hold. Environment variables, a secret, a licence file, an OAuth client registered in an identity provider. The data restores and the app still will not start.
• The restore depends on a version that no longer exists. The dump came from one major version of a database, and the only image you can pull today is two majors later.

None of these show up in a job log. All of them show up in a restore.

Why teams do not restore

If restore tests are so revealing, why do so few teams run them? The honest reasons are practical, not technical.

A restore needs somewhere to go. Restoring over production is out of the question, so you need an isolated environment with enough disk, the right versions and no route to real users or real outbound email. Building that each time is work.

A restore takes time and produces nothing visible when it goes well. It is the first task dropped in a busy month.

And a restore test can fail, which creates work nobody planned. It is tempting not to look.

That last reason is exactly why the test matters. A failure in a scheduled test costs an afternoon. The same failure during an incident costs the data.

What a real restore test checks

“We restored the file and it was there” is not a restore test. A useful test answers four questions.

Is the backup complete? Restore the database, the files and the configuration together, from the same point in time. An app is the sum of those parts. A database without its uploaded files, or files without the database rows that reference them, is not a working app.

Does the app start? Bring the application up on the restored data, in an environment with no route to production. Many failures only appear here: a missing secret, a changed schema, a dependency that was never captured.

Does it answer with its data intact? Log in. Open a record you know exists. Check that recent entries are present, which tells you how far back the restore actually reaches. That is your real recovery point, as opposed to the one written on a slide.

How long did it take? Time the whole thing, from “we need to restore” to “users can work”. That is your real recovery time. It is almost always longer than people guess, because decryption, download and database index rebuilds take time.

Then write it down: date, app, backup used, checks, duration and result. A test with no record helps only the person who ran it.

Recovery objectives are a contract, not a mood

Two numbers describe what you can lose. The recovery point objective (RPO) is how much recent data you can afford to lose: a day, an hour, nothing. The recovery time objective (RTO) is how long you can do without the app while it comes back.

These numbers differ by app. A team wiki can lose a day and be down for an afternoon without real harm. An invoicing system or a clinical record probably cannot. Setting one number for everything either overspends on the wiki or underprotects the finance system.

A restore test is how you find out whether your objectives are true. If your contract says four hours and the last test took nine, the contract is wrong, or the setup is. Either way you want to find out on a Tuesday afternoon, not during an outage.

If you run your own backups

You do not need us to start doing this properly. If you operate your own apps, a short list gets you most of the way:

1. List what each app needs to run. Database, files, configuration, secrets, external registrations. Back up all of it, or document where it lives.
2. Keep the key away from the data. Store encryption keys and restore credentials somewhere that survives the loss of the server, and make sure more than one person can reach them.
3. Use application-consistent dumps. Use the database’s own dump or backup tool rather than copying its directory while it runs.
4. Put a restore test on the calendar. Monthly is a sensible default. Put a name next to it.
5. Restore into isolation. No production credentials, no outbound email, no route to real users.
6. Record the result. Date, scope, duration, outcome. Keep the failures as well as the passes.

The first test usually finds something missing. That is the point.

The record is the backup policy

Auditors, insurers and regulators are asking the same question more and more: not “do you have backups?” but “show me the last time you restored one”. Frameworks such as NIS2 and DORA put resilience and recovery testing at the centre. The NIS2 Directive names backup management and disaster recovery among the measures it requires (article 21). DORA goes further for financial entities and requires their backup, restoration and recovery procedures to be tested periodically (article 12). A backup policy that exists only as a document does not answer that question.

A dated log of restore tests does. It also tells you, when something breaks, which backup to use and how long the restore will take.

How Pilae fits

At Pilae, backups and recovery are part of every plan rather than an add-on, for the open-source apps we operate on your premises or on dedicated machines in Pilae Cloud.

• Consistent copies, encrypted before they leave. Every app is backed up daily: its database, its files and its configuration, taken so that they are consistent with each other rather than as a raw disk snapshot. Each backup is encrypted on your machine before it leaves and stored offsite in the country you choose (data residency). The provider holding the copy holds data it cannot read.
• A verified backup before every change. Before every change, the Pilae Agent takes a fresh backup and verifies it. If that backup fails, the change does not start. An update that cannot be undone is not applied.
• Restore tests on the calendar. On the Enterprise plan, an engineer restores your apps every month into an isolated environment and checks that they start and answer with their data intact. The date, the scope and the result are recorded in the control plane, where you and your auditors can read them without taking our word for it.
• Objectives the tests keep honest. Retention follows your plan, with longer retention available. On Enterprise, the RPO and RTO for each app are set in your contract, and the monthly tests are what keep those numbers honest (pricing).
• An engineer when you need a restore. Managed operations covers the engineers who run a restore when you ask for one.

If you want to see how that looks on your own apps, talk to an engineer. We go through the apps you run, the data each one holds and how long you can do without it, and set the recovery objectives for each.