🥇Experience of changing SAP hosting: how to migrate systems painlessly

Or can it be done? Of course, migrating SAP systems is a complex and meticulous process, requiring coordinated efforts from all participants for success. If migration is to be completed under tight deadlines, the task becomes exponentially more complicated. Not everyone dares to take it on. There can be several reasons for this. For instance, the process itself is lengthy and organizationally complicated. Additionally, there is a risk of unplanned system downtimes. Or clients may doubt that going through such an operation will yield benefits commensurate with the effort expended. However, there are exceptions.

Below, we will discuss the challenges faced by clients during the migration and maintenance of SAP systems, explain why stereotypes do not always reflect reality, and share a case study on how we managed to migrate a client's systems to a new infrastructure in just over three months.

SAP Systems Hosting

Just five years ago, it was hard to imagine that clients would start massively using hosting resources for SAP applications. In most cases, they were implemented on-premise. However, with the development of outsourcing models and the cloud services market, clients' perspectives began to change. What are the arguments influencing the choice of cloud for SAP?

For beginners who have only just planned the implementation of SAP, cloud infrastructure has practically become the standard choice – scalability of resources to meet the current needs of the system and a desire not to divert resources to develop non-core competencies.
In companies with a large system landscape, hosting SAP systems allows CIOs to achieve a qualitatively different level of risk management, as the partner is responsible for the SLA.
The third most frequently cited argument is the high cost of building infrastructure for implementing high availability and DR scenarios.
Factor 2027 – the vendor's announced discontinuation of support for outdated systems in 2027. This means transitioning the database to HANA, which entails expenses for modernization and the purchase of new computing resources.

The SAP hosting market in Russia can now be considered quite mature. This offers significant opportunities for clients looking to change their hosting platforms. However, such projects understandably raise concerns among businesses due to the complexity of the migration process. This compels customers to impose higher requirements on service providers, who must not only possess exceptional competencies in hosting and supporting SAP systems but also have a successful track record in migration.

What are the challenges of changing SAP hosting?

There are different types of hosting. A mismatch with the declared level of service, numerous 'buts' and asterisks with fine print caveats, limited resources and capabilities of the hosting provider, lack of flexibility in communication with the client, bureaucracy, technical limitations, low competence of technical support specialists, as well as many other nuances — this is just a small part of the pitfalls that clients may encounter when operating their business systems in outsourced infrastructures. Often, all of this remains hidden from the client, buried deep in the multi-page contract, and only surfaces during the use of the services.

At some point, it becomes clear to the client that the level of service they are receiving is far from their expectations. This serves as a catalyst for seeking solutions to rectify the situation, and in cases where the issues accumulate to a breaking point and become unbearable, they move towards actively exploring alternative options for changing the service provider.

Why do they wait until the last moment? The reason is simple — the process of migrating systems is not always transparent or understandable for clients. It is challenging for the client to assess the actual risks associated with the migration process. One could say that migration for clients is a sort of black box: it's unclear about the costs, system downtime, risks and how to mitigate them, and overall it feels dark and frightening. After all, if things don’t work out, heads will roll among both the executives and the implementers.

SAP is an enterprise-level system, complex and frankly not cheap. Considerable budgets are spent on implementation, modification, and maintenance, and the availability and proper functioning of these systems are critical for the survival of an enterprise. Now imagine the consequences of halting a major manufacturing operation. These are financial losses that can be measured in figures with a lot of zeros, along with reputational and other equally significant risks.

Let's explore the challenges that may arise at each stage by examining the migration case of SAP systems for one of our clients.

Preparation and Design

Migration is a formula with many different components. One of the most crucial stages is the design and preparation of the target (new) infrastructure.

We needed to delve into the existing implementations of the systems and their architecture. In the target infrastructure, we replicated some existing solutions, supplemented and improved others, redesigned certain aspects, and chose solutions for ensuring fault tolerance and availability, as well as maximally consolidating all resources.

During the design process, numerous exercises were carried out that ultimately allowed us to prepare thoroughly for the migration and account for all the nuances and pitfalls (which we will discuss later).

What we achieved as a result is a uniquely designed private cloud infrastructure based on our data center:

dedicated physical servers for SAP HANA;
VMware virtualization platform for application servers and infrastructure services;
duplicated communication channels between data centers for L2; VPN;
two primary storage systems to segregate production from 'everything else';
backup system based on Veritas Netbackup with a separate server, disk shelf, and tape library.

Here's how we implemented all of this from a technical standpoint.

SAP

For efficient use of storage for productive HANA, we used shared disks without database system replication through SAP. All of this was wrapped into an Active-Standby cluster SUSE HAE based on Pacemaker. Yes, the recovery time is a bit longer than with replication, but we achieve a doubling of storage space savings and, consequently, a budget saving for the client.
In the pre-production environments, HANA clusters were rejected, but the production configuration was technically replicated.
Test environments and development environments were distributed across several servers without clusters in the MCOS configuration.
All application servers were virtualized and hosted in VMware.

Networks

Physically separated the management and production network segments with stacks of switches, directing production towards the client's data center.
Provided enough network interfaces to avoid mixing large traffic flows.
For data transfer from the storage system, classic FC SAN fabrics were established.

NAS

The SAP production and pre-production load was placed on an all-flash array.
The developer test environments and infrastructure services were placed on a separate hybrid array.

SRK

Built on Veritas NetBackup.
Slightly edited the built-in scripts to back up MCOS configurations.
Operational copies were placed on a disk shelf for quick recovery, and we use tapes for long-term storage.

Monitoring

All hardware, OS, and SAP were brought under Zabbix.
Collected numerous useful dashboards in Grafana.
When an alert occurs, Zabbix is capable of creating a ticket in the incident management system, which is implemented on Jira in our case. Information is also duplicated in a Telegram channel.

Overall HANA health

SAP application server status:

Infrastructure services

To service internal namespaces, we set up a cluster of DNS servers that synchronize with the client's servers.
Created a separate file server for data exchange.
To store various configurations, we added GitLab.
For various sensitive information, we used HashiCorp Vault.

Migration process

In general, the migration process consists of the following stages:

preparing all necessary project documentation;
negotiating with the current provider — resolving organizational issues;
purchasing, delivering, and installing new equipment for the project;
test migration and process debugging;
system transfer, live migration.

At the end of October 2019, we signed a contract, then designed the architecture, and after its approval with the client, ordered the necessary equipment.

The first thing to pay attention to is the delivery time of the equipment. On average, the delivery of certified hardware for SAP NAHA, compliant with the software vendor's hardware platform requirements, takes 10-12 weeks. Considering the seasonal aspect (the project implementation coincided precisely with the New Year), this period could extend by another month. Therefore, it was necessary to expedite the process as much as possible: we worked with the distributor-supplier, negotiating for expedited delivery by air (instead of land and sea routes).

November and December were dedicated to preparing for the migration and receiving part of the equipment. We conducted the preparation on a test stand in our public cloud, where we practiced all the main steps and identified potential complexities and issues:

we prepared a detailed interaction plan for project team members with minute-by-minute timings;
we built a test stand for databases and application servers similarly to how it would be in the target infrastructure;
we configured the necessary communication channels and infrastructure services to test integration functionality;
we worked through cutover scenarios;
the cloud also helped us create pre-configured templates for virtual machines, which we later simply imported and deployed in the target landscape.

Shortly before the New Year holidays, the first batch of equipment arrived. This allowed us to deploy part of the systems on real hardware. Since not everything arrived, we connected substitute equipment, which we managed to arrange with the vendor and distributors. The remaining components of the target infrastructure were received only at the final stage.
To meet the deadline, our engineers had to sacrifice their New Year holidays and start preparing the target infrastructure on January 2, right in the midst of the celebrations. Yes, this sometimes happens when there is no alternative. The operational capacity of systems, which is critical for the enterprise's viability, was at stake.

The overall migration process looked like this: first, the least critical systems (development landscape, testing landscape), then the production systems. The final stage of migration took place at the end of January to early February.

The migration process was detailed down to the minute. This is a cutover plan with a list of all tasks, execution times, and responsible parties. All steps had already been practiced in a test migration, so the live migration simply required following the plan and coordinating the process.

The migration was conducted systematically in several stages. Each stage included two systems.

The outcome of the three-month sprint was a fully operational system in the CROK data center. Overall, the positive result was achieved thanks to the collaborative efforts, with maximum contribution and dedication from all participants in the process.

The role of the client in the project

Communicating with the provider that our client was leaving was challenging. Understandably, they were last on the list of parties interested in the successful completion of the project. The client took on the tasks of escalation and handling all communication issues and managed this task exceptionally well. For this, they deserve special thanks. Without such active participation in the process, the project's result could have been very different.

Due to the formalized processes at the 'former' provider, infrastructure support was handled by specialists who were, in a sense, far removed from the problems of their client at that time. For instance, the process of exporting the same database could take anywhere from one hour to five. At that moment, it felt like some kind of magic, a secret that was never revealed to us. Perhaps the technical support engineers were, in their spare time, meditating, forgetting that somewhere in distant Russia, deadlines were looming, engineers were without New Year's salads, and the client was crying and suffering...

Project Results

The final touch of the migration was the transfer of systems to support.

We now provide a one-stop service for client inquiries and handle the entire volume of tasks related to the maintenance of infrastructure components and SAP basis in collaboration with our partner - itelligence. The client has been operating in a private cloud for six months. Here is the statistics on service requests during this time: