Everyone makes backups. Almost nobody restores them

A backup that’s never been tested

What I often find at new clients: there’s a backup procedure in place, it’s running, it’s dutifully producing files. So far, so good. What’s usually missing next: a testing procedure. No documentation on how to actually perform a restore. No schedule for periodic tests. Nobody who’s ever checked whether that backup can actually be put back.

And then comes the moment you need it. That’s not the moment to find out it doesn’t work.

The tape drive that was three months out of sync

I went through this myself, early in my career. We were still working with tapes, and at some point it turned out the recorder’s head was out of sync. Not slightly, completely. Not a single tape was still usable.

The worst part: it had been like that for three to four months. For three to four months, we thought we had backups. We didn’t. We eventually had to bring in an engineer to restore everything, and it took a long time. The entire time, we had no safety net.

If we’d been testing regularly, we would have caught it within a week instead of after three months. That’s a lesson you only need to learn once.

Matching frequency to your rate of change

The question every organization needs to ask itself: how much data can you afford to lose? If your backup runs once a day, a crash costs you at most a day of data. Run it once an hour, and it’s an hour. For some organizations, that’s perfectly fine. For an environment processing transactions continuously, it’s far too much. Match your backup frequency to your rate of change, and to what your organization considers an acceptable amount of data loss. In most cases, we recommend intermediate transaction log backups, which capture only the changes since the last backup. How often you run those (every fifteen minutes, every five minutes, continuously) depends on your situation.

The full backup stays on its regular schedule, and the intermediate backups fill in the gaps. That way, you cap your maximum possible data loss at exactly the interval you chose yourself. That’s a deliberate choice you make up front, not something you discover after the fact once you realize how much you’ve actually lost.

How to test restores without touching production

Ideally, you test restores on a dedicated machine set up specifically for that purpose. That keeps your production environment clean and lets you verify everything calmly.

There’s also a smarter variant I see more and more: organizations that run a separate, read-only query environment. They populate it with production backups. Every night, all production databases get backed up, and those backups get loaded automatically into the query environment. The next day, staff use that environment for reporting.

The beauty of it: you test your backup and restore process every single night, without anyone having to think about it separately. If the restore fails, you find out the next morning, because the query environment isn’t available. That’s a lot less painful than finding out during an actual outage.

On top of that, you can occasionally run a data compare: checking the data in your query environment against production. Doesn’t need to happen daily, but a periodic check that what you restored actually matches what’s there is worthwhile. This entire process can be automated with scripts and a scheduler.

Five out of six systems down, backups saved the day

A client of ours recently had five out of six production systems crash due to an infrastructure failure. Fortunately, it happened over the weekend, so few users were active and the data loss stayed limited.

All production systems were restored using the daily backups. The restore process was documented, automated, and tested. It worked. The data that fell into the gap between the last backup and the crash could partly be reconstructed from linked systems.

Looking back, they were enormously relieved that the whole backup and restore process was in order. We had helped build and document it. Without that groundwork, the damage would have been many times worse.

IT’s neglected stepchild

The backup process is a neglected stepchild. Everyone knows it needs to be there, but there’s no urgency as long as it’s working. And in our world, it usually is working. So it slides down the priority list: “Oh right, we’ll get to that.” Or: “We’ll set that up properly later.”

Larger organizations generally have it in order. At smaller companies, it varies a lot more. There, it only becomes a priority once something goes wrong. And then you learn the lesson the hard way.

That’s not a disgrace. I think you should see it as a lesson. It happens to you once, and after that, you set it up properly. What I do have trouble with is when it happens a second time.

Advice for the IT manager

If I were an IT manager myself, here’s what I’d want. Regular spot checks: prove that a restore actually works, not just on paper but in practice. Make sure the documentation is accurate and current, because nothing changes as fast as IT. It only takes a few things shifting in your process, and if that isn’t documented, you’re scrambling during an outage.

Schedule failover tests more than once a year. A lot of organizations run an annual failover test, simulating a data center outage and checking whether the failover works. That’s good, but for your backup and restore process, it’s not enough. Do it every quarter, or at least every six months.

Automate where you can. Testing restores doesn’t have to be a manual chore. With the right scripts and scheduling, it runs in the background and flags you the moment something’s wrong. Talk about it with your team. Make it a standing item on the agenda, not something that happens only if there’s time left over.

How concrete this can get was proven by the fire at NorthC Datacenters in Almere this past May. Organizations like NAK saw their entire service grind to a halt because they couldn’t reach their data. No certificates, no fieldwork, no lab work. For weeks. Recovery took weeks.

You can rebuild an application. Not your data.

You hear it more and more: the database is “just a commodity.” Developers saying, “Just spin up a database, dump my data in there.” That attitude underestimates what’s actually at stake. You can rebuild an application. You can’t rebuild your data. And the only way to protect your data is a backup process that doesn’t just run, one you’ve proven actually works.

Want to know where your backup and restore process stands? Get in touch for a no-obligation conversation. We’d be glad to take a look with you.