Direct naar content

From Highway to Hell to Smooth Operations: Lessons from a Complex HADR Implementation

Harry Splinter, Senior Database Reliability Engineer at OptimaData, is no stranger to complex database landscapes. When a customer approached us to help implement SAP HADR (High Availability Disaster Recovery), it initially appeared to be a fairly straightforward assignment. But as is often the case in the database world, things quickly became more complicated than anticipated. In this blog, Harry reflects on an implementation that started as a “nice journey” but turned into a real “Highway to Hell”. And how expertise and perseverance ultimately turned it into a success.

Harry Splinter

DBA Consultant en Senior Database Reliability Engineer
Harry Splinter - DBA Consultant en Senior Database Reliability Engineer
Van Highway to Hell naar succes ofwel hoe we een complexe HADR-implementatie tot een goed einde brachten

What Is HADR?

Let me start with an important nuance: SAP HADR is not what most DBAs think it is. It’s not SQL Server Always On, even though they originated from the same roots decades ago.

HADR is actually a collection of SAP software components and tools, including Adaptive Server Enterprise, SAP Replication Server, SAP Host Control, Replication Agent Manager, and SAP Fault Manager. It was originally developed for Linux and only later ported to Windows.

And that is where we ran into our very first obstacle.

Windows Was Not the Right Choice

For this project, the customer chose Windows as the operating system, for reasons that made sense to them at the time. In hindsight, however, it turned out not to be the most practical decision. Instead of Linux, the native platform for this functionality, they opted for an environment that simply wasn’t fully ready for the challenge.

It didn’t take long before we discovered that both the documentation and the software itself were… incomplete. At that point, I could almost hear Emperor Palpatine’s voice from Star Wars echoing in the back of my mind: Your journey towards the dark side is complete.”

Documentation? What Documentation?

The documentation lacked Windows-specific commands and configuration examples altogether. We had to reach out directly to SAP to get the information we needed — but most of what we received was clearly written with Linux in mind. Translating that guidance to Windows was left entirely up to us.

To be fair, SAP was very helpful, full credit where it’s due, but this was not what we had expected going into the project.

Gradually, we started to notice changes appearing in the online documentation. A clear sign that things were still very much a work in progress. Commands that SAP initially shared with us directly were nowhere to be found in the SAP4Me documentation at first… only to suddenly appear there at a later stage. “Could it be magic?“. As Barry Manilow once famously sang.

The Software Did What It Was Supposed to Do — But…

While the software delivered on what it promised, it had clearly been designed with a very different mindset than the environment we were dealing with. HADR was primarily built for single-database environments, not for the multi-database setups we encountered in this project.

When we started working with replication, it became clear that the existing code was not fit for purpose. Certain configuration settings had to be introduced to get things working properly again, and a significant number of procedures had to be rewritten. This wasn’t a limitation of HADR itself, but rather the result of application code that had been developed without replication in mind. Don’t you forget about me“, as Simple Minds would say… but someone seems to have missed that memo.

Materialization: The Fast Way Is the Slow Way

When setting up HADR, every database needs to be materialized to the companion server. The straightforward approach is also the slowest one. It can be accelerated, but only with manual intervention and custom scripting.

Uppercase or Lowercase? Pick One

Windows is case-insensitive by default. SAP HADR is not. The conclusion was simple: everything has to be either uppercase or lowercase… and when I say everything, I really mean everything.
sql.ini, interfaces, configuration files, profile files… you name it.

As Yazz sang back in 1988: The only way is up” – In this case, quite literally, all the way to uppercase (or lowercase, for that matter).

The sapadm Problem

And then there was the sapadm account. Not quite what Salt-N-Pepa were singing about back in the ’80s, but definitely something that needed to be discussed.

This critical local Windows account requires administrator privileges and must use the same password across all involved servers. Not exactly a security officer’s dream, but then again… we do want the system to actually work, don’t we?

And Active Directory? My advice: don’t. If the AD\sapadm account gets locked for any reason, you’re in trouble. HADR will simply stop functioning, as this is the account used to make changes throughout the environment.

The Linux Test: When Most Pieces Finally Came Together

“However” (with the voice of Jeremy Clarkson from Top Gear), we also tested this HADR solution on Linux, and almost all of the challenges described above disappeared overnight. This reinforced our long-standing advice: always run software on its native operating system, using proven technology.

From highway to hell to success

Despite all the obstacles, we successfully brought this project to completion. It required a significant amount of additional effort, creative problem-solving, and direct collaboration with SAP — but in the end, the HADR solution is running stable and reliable.

This project perfectly illustrates what sets OptimaData apart. We tackle even the most complex database challenges, especially when standard documentation and best practices fall short. Through craftsmanship, perseverance, and a healthy dose of creativity, we turn every “highway to hell” into a success story.

Author’s Note

While writing this article, my brain naturally started playing songs and summoning familiar voices. That always happens when I’m working on something I’m truly passionate about — and database challenges definitely fall into that category.

Complete Playlist of Referenced Songs

Playlist opent in nieuw tabblad

Want to Learn More?

Curious about complex database implementations and how we tackle them? At OptimaData, we have the expertise to solve even the toughest challenges. Feel free to get in touch with us.

Secret Link