HighLoad++, Evgeny Kuzovlev (EcommPay IT): what to do when a minute of downtime costs $100,000

Everyone talks about development and testing processes, staff training, and increasing motivation, but these processes seem inadequate when a minute of service downtime costs astronomical amounts. What should you do when handling financial transactions under a strict SLA? How can you enhance the reliability and resilience of your systems, excluding development and testing?

The next HighLoad++ conference will take place on April 6 and 7, 2020, in Saint Petersburg. Details and tickets at this link. November 9, 6:00 PM. HighLoad++ Moscow 2018, Delhi + Kolkata hall. Abstracts and presentation.

Evgeny Kuzovlev (hereinafter – EK): – Friends, hello! My name is Evgeny Kuzovlev. I am from EcommPay, specifically from the EcommPay IT division, the IT branch of the group. Today we will talk about downtimes – how to avoid them and how to minimize their consequences if avoidance is not possible. The topic is titled: “What to do when a minute of downtime costs $100,000”? To give you a sneak peek, our numbers are comparable.

What does EcommPay IT do?

Who are we? Why am I standing here before you? Why am I qualified to share something with you here? And what will we discuss in more detail?

The EcommPay group of companies is an international acquirer. We process payments globally – in Russia, Europe, and Southeast Asia (All Around the World). We have 9 offices, a total of 500 employees, and approximately just under half of them are IT specialists. Everything we do, everything we earn money from, has been developed in-house.

All our products (and we have quite a few – around 16 different components in our range of large IT products) are created by us; we develop them ourselves. Currently, we conduct around a million transactions per day (millions, is probably the better phrase). We are still a young company – we are just about six years old.

Six years ago, it was a startup when a group of folks came together with a business. They were united by an idea (nothing but the idea), and we took off. Like any startup, we were running faster… For us, speed was more important than quality.

At some point, we stopped: we realized that we could no longer live with the same speed and quality, and we needed to focus on quality first. At that moment, we decided to write a new platform that would be correct, scalable, and reliable. We began developing this platform (investing in it, developing, testing), but at some point, we understood that development and testing alone would not allow us to reach a new level of service quality.

You're creating a new product, you're launching it in production, but something is bound to go wrong somewhere. Today, we will discuss how to reach a new quality level (our experience and how we achieved it), putting development and testing aside; we will talk about what operations can do on their own and what they can offer to testing to influence quality.

Downtime. The commandments of operations.

The cornerstone of today's discussion is downtime. A scary word. When we experience downtime, everything goes wrong. We rush to fix it; the administrators hold the server – hopefully, it won't go down, as the song goes. This is what we will talk about today.

When we started changing our approaches, we formulated four commandments. They are presented in my slides:

These commandments are quite simple:

• Quickly identify the problem.
• Get rid of it even faster.
• Help understand the cause (later, for the developers).
• And standardize the approaches.

I want to draw your attention to point No. 2. We eliminate the problem rather than solve it. Solving it is secondary. For us, the primary concern is that the user is shielded from this problem. It will exist in an isolated environment, but that environment will not interact with them. We will go through these four groups of problems (in some detail and in less detail), and I will share what we use, what our relevant experience in solutions is.

Troubleshooting: when they happen and what to do about them?

But we won’t start in order; we’ll begin with point number 2 – how to quickly eliminate the problem? We have a problem – we need to solve it. "What should we do about this?" – is the main question. And when we started thinking about how to solve the problem, we developed some requirements that problem-solving should follow.

To formulate these requirements, we asked ourselves the question: "When do we encounter problems?" And, as it turned out, problems typically occur in four cases:

• Hardware failure.
• External service outages.
• Software version changes (the deployment).
• Explosive growth in load.

We won’t discuss the first two. Hardware failures can be resolved quite simply: everything should be duplicated. If it’s disks – they should be configured in RAID; if it’s a server – the server should be duplicated; if you have a network infrastructure – you should have a second copy of the network infrastructure, meaning you duplicate everything. And if something fails, you switch to backup resources. It’s hard to say much more about this.

The second issue is external service outages. For the majority of systems, this is not a problem at all, but not for us. Since we process payments, we are an aggregator that stands between the user (who enters their card details) and the banks and payment systems (like Visa, MasterCard, Mir, etc.). Our external services (payment systems, banks) are prone to outages. Neither we nor you (if you have such services) can influence this.

So, what should we do then? There are two options here. Firstly, if you can, you should duplicate that service in some way. For example, if we can, we shift traffic from one service to another: if we were processing cards through Sberbank and Sberbank has issues, we redirect the traffic [conditionally] to Raiffeisen. Secondly, what we can do is quickly detect external service outages, which is why we will discuss response speed in the next part of the report.

In fact, out of these four, we can specifically influence the software version changes – take actions that will improve the situation in terms of deployments and in the context of explosive load growth. Essentially, that's what we did. Here, again, a small remark…

Of these four issues, several can be resolved immediately if you have cloud infrastructure. If you are using Microsoft Azure, Ozon, our clouds from Yandex or Mail, then at least hardware failure becomes their responsibility, and everything becomes good in terms of hardware failures.

We are a somewhat unconventional company. While everyone talks about Kubernetes and clouds, we don't have any Kubernetes or clouds. Instead, we have racks with hardware in numerous data centers, and we are compelled to live on that hardware; we have to take responsibility for all of it. So, in this context, let's discuss the problems. We've set the first two aside.

Software Version Change. Foundations

Our developers do not have access to production. Why is that? Simply put, we are PCI DSS certified, and our developers are simply not allowed to access production. That's it. Point. Fully. Therefore, the responsibility of development ends precisely at the moment the build is handed over to release.

Our second foundation, which also helps us significantly, is the absence of unique undocumented knowledge. I hope you have the same situation. Because if not, you will run into problems. Problems arise when this unique undocumented knowledge is not present at the right time and place. For example, if one person knows how to deploy a specific component – and that person is unavailable due to vacation or illness – you will face problems.

And the third foundation we arrived at. We came to it through pain, blood, and tears – we concluded that any of our builds contain errors, even if they appear to be error-free. We've decided that whenever we deploy something, whenever we push something into production – our build has errors. We have established the requirements that our system must meet.

Requirements for Software Version Change

There are three such requirements:

• We must be able to quickly roll back a deployment.
• We must minimize the impact of an unsuccessful deployment.
• And we must be able to deploy quickly in parallel.
  Exactly in that order! Why? Because, first of all, during the deployment of a new version, speed is not the priority, but if something goes wrong, it's essential to roll back quickly and minimize the impact. However, if you have a set of versions in production that contain a bug (like snow coming out of nowhere, with no deployment but an error being present) – the speed of deploying subsequent versions becomes crucial. What have we done to meet these requirements? We've adopted a methodology:

  
  
  It's quite well-known; we didn't invent it – it's Blue/Green deployment. What does it mean? For each group of servers running your applications, you should have a copy. A 'warm' copy: there is no traffic on it, but at any moment, that traffic can be switched to this copy. This copy contains the previous version. At the moment of deployment, you roll out the code to the inactive copy. Then you switch a portion of the traffic (or all of it) to the new version. Thus, to change the traffic flow from the old version to the new one, you need to perform just one action: you need to change the load balancer in upstream, redirecting it from one upstream to another. This method is very convenient and addresses the problem of quick switching and rollback.

  Here’s the solution to the second question – minimization: you can direct only a portion of your traffic (say, for example, 2%) to the new line with the new code. And these 2% are not 100%! If you've lost 100% of traffic during an unsuccessful deployment – that's alarming; but losing 2% of traffic – that's unpleasant, but not catastrophic. Moreover, users are likely not to even notice it because, in some cases (not all), the same user, by hitting F5, will land on another, functioning version.

  Blue/Green deployment. Routing

  However, it's not all that simple with 'Blue/Green deployment'... All of our components can be divided into three groups:

  • the frontend (payment pages that our clients see);
  • the core processing;
  • an adapter for working with payment systems (banks, MasterCard, Visa, etc.).

  There's a nuance here – the nuance lies in the routing between lines. If you simply switch 100% of the traffic, you won't have these problems. But if you want to switch 2%, you start to encounter questions: "How do you do that?" The simplest way, straightforward: you can set up a random selection, Round Robin in nginx, and you'll have 2% going left, 98% going right. But this doesn't always fit.

  For us, for instance, a user interacts with the system not with a single request. This is normal: 2, 3, 4, 5 requests – your systems may have the same. And if it's important for you that all user requests come to the same line as the first request, or (second point) that all user requests go to a new line after switching (they might have started working with the system earlier, before the switch) – then random distribution doesn't work for you. In that case, there are the following options:

  
  
  The first option, the simplest – based on the client's basic parameters (IP Hash). You have an IP, and you separate left and right based on the IP address. Then you'll have the second case I described, when deployment occurred, and the user could already have started working with your system, and from the time of deployment, all requests will go to the new line (to the same one, say).

  If for some reason this doesn't suit you and you must send requests to the line where the initial, first user request came, then you have two options...
  The first option: you can use paid nginx+. There is a mechanism called Sticky sessions, which, upon the initial user request, assigns a session to the user and ties it to a specific upstream. All subsequent user requests within the session's lifespan will be sent to the same upstream where the session was set.

  This didn't fit for us because we already had regular nginx. Switching to nginx+ – it wasn't that it was expensive, but it was somewhat painful and not quite right for us. For example, "Sticky sessions" didn’t work for us for the simple reason that they don't allow routing based on an "Either-or" basis. You can set up, for instance, Sticky sessions based on the IP address or on the IP address and cookies or on post parameters, but for "Either-or" – that's more complicated.

  Therefore, we arrived at the fourth option. We took nginx 'on steroids' (that's openresty) – it's the same nginx, but additionally supports embedding last-scripts. You can write a last-script, pass it to this 'openresty', and that last-script will run when a user request comes in.

  We actually wrote such a script, set up 'openresty', and in this script, we iterate over 6 different parameters through 'Or' concatenation. Depending on the presence of certain parameters, we can tell whether the user came to one page or another, or to one line or another.

  Blue/Green deployment. Pros and cons.

  Of course, it might have been simpler (using 'Sticky sessions', for instance), but we have another nuance: not only the user interacts with us within one transaction processing... Payment systems also interact with us: after we process the transaction (sending a request to the payment system), we receive a callback.
  And let's say, within our environment, we can pass the user's IP address in all requests and based on the users' IP addresses separate them, we can't say to 'Visa': 'Hey guys, we're this retro company, we're kind of international (on the site and in Russia)... Could you please pass us the user's IP address in an additional field, your standardized protocol!' Obviously, they won't agree.

  
  
  Therefore, that approach didn't work for us – we went with openresty. Accordingly, here’s how we managed routing:

  The 'Blue/Green deployment' has its advantages, as I mentioned, and disadvantages.

  There are two main disadvantages:

  • you need to deal with routing;
  • the second main drawback is costs.

  You need twice as many servers, you need twice as much operational resources, and you have to spend twice as much effort to maintain this whole setup.

  By the way, among the advantages, there's one more thing I haven't mentioned before: you have a backup in case of increased load. If you experience a sudden surge in traffic, with a large number of users hitting your site, you simply enable a second line for a 50/50 distribution – and you immediately have two servers in your cluster, until you resolve the server availability issue.

  How to achieve a quick deployment?

  We've discussed how to solve the problem of minimizing and rolling back quickly, but the question remains: "How can we deploy quickly?"

  
  
  Here's a brief and straightforward answer.

  • You must have a Continuous Delivery (CD) system in place – it's essential. If you have one server, you can deploy manually. We have about one thousand five hundred servers, and obviously, we can't employ manual deployment for all of them; that would require a team the size of this hall just for deploying.
  • Deployment should be parallel. If your deployment is sequential, that's problematic. One server is fine, but with one thousand five hundred servers, you’ll be deploying all day.
  • Again, to speed things up, this might not be necessary, probably. During deployment, the project is usually built. You may have a web project with a frontend (you’re doing a web pack, running npm – something like that), and this process is generally not long – about 5 minutes, but those 5 minutes can be critical. That's why, for example, we don't do that: we've eliminated those 5 minutes; we deploy the artifacts instead.

    What is an artifact? An artifact is a built package where all the assembly has already been completed. We store this artifact in an artifact repository. At one time, we used two repositories – Nexus and now jFrog Artifactory. We initially used Nexus because we started practicing this approach in Java applications (it suited well for this purpose). Then, we also included some PHP applications there; and since Nexus was no longer suitable, we chose jFrog Artifactory, which can handle practically everything. We've even come to the point where we store our own binary packages, which we compile for servers, in this artifact repository.

  Sudden traffic surge

  We talked about upgrading software versions. Next, we have a sudden traffic surge. Here, I probably understand by sudden surge something not entirely correct...

  We have written a new system – it is service-oriented, stylish, and beautiful, with workers everywhere, queues everywhere, and asynchronicity everywhere. In such systems, data can flow through different paths. For the first transaction, the 1st, 3rd, and 10th workers may be involved, and for the second transaction – the 2nd, 4th, and 5th. Today, for example, in the morning, you have a data stream that involves the first three workers, and in the evening, it abruptly changes, activating three different workers.

  Here, it turns out that you need to somehow scale the workers, you need to scale your services in a way that avoids resource bloat.

  
  
  We have defined our requirements. These requirements are quite simple: there should be service discovery, parameterization – all standard for building such scalable systems, with the exception of one point – resource amortization. We stated that we are not ready to amortize resources just to keep servers running. We opted for Consul and Nomad, which manages our workers.

  Why is this a problem for us? Let's rewind a bit. Currently, we have about 70 payment systems behind us. In the morning, traffic goes through Sberbank, then Sberbank fails, for example, and we switch to another payment system. We had 100 workers operating with Sberbank, and after that, we need to urgently deploy 100 workers for a different payment system. Ideally, this should happen without human intervention. Because if human intervention is required, an engineer must be on duty 24/7, solely dedicated to this issue, since such failures occur regularly with 70 systems relying on us.

  Therefore, we looked into Nomad, which has an open IP, and created our tool Scale-Nomad – ScaleNo, which does the following: it monitors the growth of the queue and increases or decreases the number of workers based on queue dynamics. When we completed it, we thought: 'Maybe we should open-source it?' Later, we looked at it – it’s as simple as can be.

  So far, we haven't open-sourced it, but if after this presentation, you realize you need such a tool, my contact information is on the last slide – please reach out to me. If at least 3-5 people express interest, we will consider open-sourcing it.

  
  
  How does it work? Let’s take a look! To give you a preview: on the left side is a piece of our monitoring: one line at the top represents the event processing time, in the middle is the number of transactions, and at the bottom is the number of workers.

  If we look at this picture, there is a failure. On the upper graph, one of the charts peaked at 45 seconds – one of the payment systems went down. Immediately, we can see the traffic for 2 minutes and a queue increase on another payment system, where there were no workers (we did not utilize resources – on the contrary, we utilized the resource correctly). We did not want to overload – there was a minimal number, about 5-10 workers, but they couldn't handle it.

  The last graph shows a ‘hump’ that indicates that ‘Scaled’ raised this number twofold. And then, when the graph dropped a bit, it slightly reduced – the number of workers was automatically adjusted. This is how this thing works. We discussed point #2 – ‘How to quickly eliminate causes.’

  Monitoring. How to quickly identify a problem?

  Now, the first point – ‘How to quickly identify a problem?’ Monitoring! We must quickly understand certain things. What things should we quickly grasp?

  
  
  Three things!

  • We must quickly understand and assess the performance of our own resources.
  • We need to quickly identify outages, monitor the performance of systems that are external to us.
  • The third point – identifying logical errors. This is when the system operates normally according to all indicators, but something goes wrong.

  Here, I probably won’t share anything too groundbreaking. I’ll be Captain Obvious. We were looking at what the market has to offer. We have formed a ‘fun zoo’. Here is the zoo we currently have:

  
  
  We use ‘Zabbix’ for monitoring ‘hardware’, for monitoring the main server indicators. We use ‘Okmeter’ for databases. ‘Grafana’ and ‘Prometheus’ are used for all other metrics that didn’t fit the first two, with some parts – ‘Grafana’ with ‘Influx’ and Telegraf.

  A year ago, we wanted to use New Relic. It's a great tool, it can do everything. But as much as it can do, it's equally expensive. When we grew to 1,500 servers, a vendor came to us and said, 'Let's sign a contract for next year.' We looked at the price and said no, we won't do that. Now we're discontinuing New Relic; we have about 15 servers left under its monitoring. The price turned out to be completely outrageous.

  And there is one tool that we developed ourselves – it's the Debugger. At first, we called it 'Bagger,' but then an English teacher of ours came by, laughed hard, and we renamed it to 'Debugger.' What is it? It's a tool that, in fact, runs tests on the overall operability of each component, like a 'black box' of the system, within 15-30 seconds.

  For instance, if it's an external page (payment page) – it simply opens it and checks how it should look. If it's processing, it sends a test 'transaction' – ensuring that this 'transaction' goes through. If it's a connection with payment systems, we respectively send a test request where we can, and check that everything is fine.

  What metrics are important for monitoring?

  What do we mainly monitor? Which metrics are important to us?

  
  

  • Response time / RPS on the fronts is a very important metric. It immediately indicates that something is wrong.
  • The number of processed messages in all queues.
  • The number of workers.
  • The main correctness metrics.

  The last point is a 'business' metric. If you want to monitor the same thing, you need to define one or two metrics that are your primary indicators. For us, such a metric is throughput (the ratio of successful transactions to the total flow of transactions). If anything changes in it over a period of 5-10-15 minutes – it means we have problems (if it changes drastically).

  Here’s what it looks like for us – an example of one of our dashboards:

  
  
  On the left side, there are 6 graphs, representing the number of workers and messages in queues. On the right side, we have RPS and RTS. At the bottom is the key business metric. This metric immediately shows us that something went wrong with the two middle graphs... It turns out that another system, which supports us, has crashed.

  Secondly, we needed to monitor the failures of external payment systems. Here we implemented OpenTracing—a mechanism, standard, paradigm that allows for tracing distributed systems; and we made some modifications. The standard OpenTracing paradigm suggests building a trace for each individual request. We didn’t need that, so we wrapped it into a cumulative, aggregative trace. We created a tool that enables us to track the performance of the systems supporting us.

  
  
  The graph indicates that one of the payment systems started responding in 3 seconds—we encountered problems. Additionally, this tool reacts when issues arise within a 20-30 second interval.

  The third class of monitoring errors we have is logical monitoring.

  Honestly, I didn’t know what to draw on this slide because we spent a long time looking for a solution on the market that would suit us. We found nothing, so we had to create it ourselves.

  
  
  What do I mean by logical monitoring? Imagine you create a system (for example, a clone of Tinder); you build it, launch it. A successful manager, Vasya Pupkin, installs it on his phone, sees a girl, likes her... but the like doesn’t go to the girl—it goes to the security guard, Mikhailovich, in the same business center. The manager goes downstairs and then wonders, 'Why is this guard Mikhailovich smiling at me so pleasantly?'

  In such situations… For us, this situation sounds a bit different because (as I mentioned) it's a reputational loss that indirectly leads to financial losses. We have the opposite situation: we may incur direct financial losses – for example, if we conducted a transaction as successful but it was unsuccessful (or vice versa). We had to build our own tool that tracks the number of successful transactions by business metrics over time. We didn’t find anything on the market! That was the point I wanted to convey. There’s nothing on the market for solving such tasks.

  This was about how to quickly identify the problem.

  How to determine the causes of the deployment

  The third group of tasks we tackle is, after identifying the problem and resolving it, it would be great to understand the cause for development, for testing, and to do something about it. Accordingly, we need to investigate, we need to gather logs.

  
  
  If we talk about logs (the main reason is logs), most of our logs are in ELK Stack – this is true for almost everyone. Some may not use ELK, but if you’re writing logs in gigabytes, sooner or later you’ll end up with ELK. We write them in terabytes.

  
  
  There's a problem here. We fixed a bug for the user, began digging into what the issue was, went into Kibana, entered the transaction ID, and got this long output (shows a lot). And in this output, nothing is clear at all. Why? Because it’s unclear which part relates to which worker, which part relates to which component. At that moment, we realized we needed tracing – the famous OpenTracing I mentioned.

  We thought about this a year ago, turned our gaze towards the market, and found two tools – Zipkin and Jaeger. Jaeger is actually an ideological successor, an ideological continuation of Zipkin. Zipkin has everything good, except that it cannot aggregate, it cannot include logs in the tracing, only time tracing. Jaeger supports this.

  We looked at 'Eger' and realized we could instrument applications and write to the API (the API standard for PHP at that time, however, had not been approved – that was a year ago, but now it has been approved), yet there was absolutely no client. 'Okay,' we thought, and wrote our own client. Here's what we ended up with:

  
  
  In 'Eger', spans are created for each message. So when a user opens the system, they see one or two blocks for each incoming request (1-2-3 – the number of blocks corresponds to the incoming requests from the user). To make it easier for users, we added tags to the logs and the time tracing. Accordingly, in the event of an error, our application will mark the log with the corresponding Error tag. You can filter by the Error tag, and it will display only the spans containing that block with the error. Here's what it looks like when we expand the span:

  
  
  Inside the span, there is a set of traces. In this case, there are three test traces, and the third trace tells us that an error occurred. At the same time, we can see the time tracing: at the top, there's a timeline, and we can see the time interval during which each log was recorded.

  Therefore, we achieved great results. We wrote our own extension and open-sourced it. If you want to work with tracing, if you want to use 'Eger' in PHP – here's our extension, welcome to use it, as they say:

  
  
  Our extension is a client for working with the OpenTracing API, made as a PHP extension, meaning you will need to build it and integrate it into your system. A year ago there was nothing else. Now other clients have emerged that are available as components. It's up to you: either you can download components using Composer, or you can use the extension.

  Corporate Standards

  We discussed the three commandments. The fourth commandment is to standardize approaches. What is this about? It's roughly about this:

  
  
  Why the word 'corporate' here? Not because we are a large or bureaucratic company, no! I wanted to use the word 'corporate' in the context that each company, each product should have its own standards, and so should you. What standards do we have?

  
  

  • We have a deployment regulation. Without it, we cannot move forward. We deploy about 60 times a week, meaning our deployments happen almost constantly. However, there is a restriction in our deployment regulations against deploying on Fridays—we simply do not deploy.
  • We require documentation. No new component goes live unless there is documentation for it, even if it was created by our R&D teams. We require them to provide a deployment guide, a monitoring map, and a general description (as programmers can write) of how this component works and how to troubleshoot it.
  • We focus not on the root cause of the problem, but on the problem itself—that's what I've mentioned before. It is important for us to shield the user from issues.
  • We have tolerances. For example, we do not consider downtime if we lose 2% of traffic for two minutes. This does not count in our statistics. If it’s more in percentage or time, then we do consider it.
  • And we always write postmortems. Whatever happens, any situation that behaves unusually in production will be reflected in the postmortem. A postmortem is a document where you detail what occurred, including a timeline of what you did to fix it, and (this is a mandatory section!) what you will do to prevent it from happening again in the future. This is essential for subsequent analysis.

  What constitutes downtime?

  
  
  What did all this lead to?

  This led to certain stability issues that were unacceptable to both our clients and us, resulting in a stability rate of 99.97% over the past 6 months. One could say that this is not much. Yes, we have room for improvement. About half of that figure involves stability not entirely under our control, but from our web application firewall, which is utilized as a service, yet clients do not differentiate.

  We have learned to sleep at night. Finally! Six months ago, we could not. And on this note, I'd like to make one remark. Last night there was a wonderful presentation about nuclear reactor management systems. If I have people who wrote this system listening to me—please forget what I said about "2% not being downtime." For you, 2% is downtime, even if it is just for two minutes!

  That's all! Your questions.

  
  

  About load balancers and database migration

  Audience question (hereinafter – Q): – Good evening. Thank you very much for such an administrative report! A brief question on your load balancers. You mentioned that you have a WAF, so as I understand it, you use some external service as a load balancer...

  EK: – No, we use our own services as a load balancer. In this case, WAF is solely a tool for protecting against DDoS attacks.

  Q: – Can you say a few words about load balancers?

  EK: – As I mentioned, this is a group of servers in openresty. Currently, we have 5 reserved groups that are dedicated exclusively… that is, the server running openresty solely proxies traffic. For your understanding, our standard traffic flow is several hundred megabits. They manage well, they’re doing fine, and not even pushing themselves.

  Q: – A simple question as well. There is Blue/Green deployment. What do you do, for instance, with database migrations?

  EK: – Good question! In our Blue/Green deployment, we have separate queues for each line. That is, if we’re talking about event queues that transfer from worker to worker, there are separate queues for the blue line and the green line. When it comes to the database itself, we deliberately narrowed it down as much as possible, moving almost everything into queues; the database only stores the transaction stack. And this transaction stack is common for all lines. Regarding the database in this context: we do not separate it into blue and green because both versions of the code need to know what is happening with the transaction.

  Friends, I have one more small prize to motivate you – a book. And I need to award it for the best question.

  Q: – Hello. Thank you for the report. Here's the question. You monitor payments, you monitor the services you interact with… But how do you monitor that a person somehow reached your payment page, made a payment, and the project credited the money to them? That is, how do you monitor that the merchant is available and accepted your callback?

  EK: – The 'merchant' for us in this case is exactly the same external service as the payment system. We monitor the response time of the 'merchant'.

  On database encryption

  Q: – Hello. I have a quick question. You have sensitive data according to PCI DSS. I wanted to know how you store PANs in queues that you need to transmit. Do you use any encryption? And the follow-up question: PCI DSS requires regular re-encryption of the database in case of changes (like admin dismissals, etc.) – how does availability work in this case?

  
  
  EK: – That's a great question! First of all, we do not store PANs in queues. We are not allowed to store PANs in plaintext anywhere, so we use a special service (we call it 'Keydemon') – this service does only one thing: it takes incoming messages and returns them encrypted. And we store everything in that encrypted form. Accordingly, our key length is under a kilobyte to ensure it's solid and reliable.

  Q: – Is it really 2 kilobytes now?

  EK: – It seems just yesterday it was 256… How much more do we need?!

  So, that's the first point. Secondly, the solution we have supports the re-encryption procedure – it has two pairs of 'keys' (KEKs) that produce 'DEKs' that encrypt (key is the key, dek is derived from the keys and performs encryption). In case we initiate the procedure (which happens regularly, from every 3 months to ± some point), we load a new pair of KEKs, and we perform data re-encryption. We have separate services that extract all the data, re-encrypt it; alongside the data, we store the key identifier with which they have been encrypted. So, as soon as we have the data encrypted with the new keys, we remove the old keys.

  Sometimes payments need to be processed manually…

  Q: – So, if a refund comes through for a transaction, you decrypt it using the old key?

  EK: – Yes.

  Q: – Then, one more small question. When there is a failure, crash, or incident, it is necessary to push the transaction in manual mode. Such situations occur.

  EK: – Yes, they do.

  Q: – Where do you get this data? Or do you manually access that storage?

  EK: – Well, of course, we have a back-office system that contains an interface for our support. If we don’t know the status of the transaction (for example, while the payment system has not responded due to a timeout), we inherently don’t know, which means we only assign the final status when we are fully confident. In this case, we place the transaction in a special status for manual processing. In the morning, the next day, as soon as support receives information that there are such transactions remaining in the payment system, they manually process them in this interface.

  
  
  Q: – I have a couple of questions. One of them is about the continuation of the PCI DSS zone: how do you export logs from their environment? This question arises because the developer could have placed anything in the logs! The second question: how do you deploy hotfixes? Manually in the database is one option, but there might be free hotfixes – what is the procedure for that? And the third question is probably related to RTO and RPO. Your availability was 99.97, almost four nines, but I understand that you have a second data center, a third data center, and a fifth data center… How do you handle their synchronization, replication, and everything else?

  EK: – Let’s start with the first. Was the first question about logs? When logs are written, there is a layer that masks all sensitive data. It looks at the mask and additional fields. Consequently, our logs are produced with already masked data and conform to PCI DSS. This is one of the regular tasks assigned to the testing department. They are required to check every task for the logs they produce, and this is one of the regular tasks during code review to ensure that the developer hasn’t recorded anything inappropriately. Subsequent checking is carried out regularly by the information security department about once a week: logs from the last day are sampled and processed through a special scanner-analyzer from test servers to verify everything.
  About hot-fixes. This is included in our deployment regulations. We have a separate section about hot-fixes. We believe that we deploy hot-fixes around the clock whenever we need to. As soon as the version is built, as soon as it has been tested, as soon as we have the artifact, the on-call system administrator is alerted by support, and they deploy it at the moment it is necessary.

  About the 'four nines.' The current figure we have was genuinely achieved, and we aimed for it yet on another data center. Now we have a second data center, and we're starting to route between them, and the issue of cross-data center replication is indeed a non-trivial one. We tried to solve it in various ways: we attempted to use the same 'Taranto' – it didn’t work out, I can tell you immediately. Therefore, we have come to the conclusion that we manually order 'Sensa.' Each application actually runs the necessary 'change-done' synchronization in asynchronous mode between the data centers.

  Q: – If you have a second one, then why not a third? Because Split-brain hasn’t been solved by anyone…

  EK: – We don’t have a 'Split-brain.' Because each application runs on a multi-master, we don't care which center the request comes to. We are prepared for the case where one data center fails (we plan for this) and during the user’s request we switch to the second data center, we are ready to lose that user, indeed; but it will be just a few, absolute few.

  Q: – Good evening. Thank you for the presentation. You talked about your debugger that runs some test transactions in production. Could you tell us about those test transactions? How deep does it go?

  EK: – It goes through the entire cycle of the component. There are no differences for the component between a test transaction and a live one. In terms of logic, it’s just a separate project in the system, where only test transactions are run.

  Q: – Where do you cut it off? Core sent…

  EK: – We are behind 'Core' in this case for test transactions… We have a concept of routing: 'Core' knows which payment system to send to – we send it to a fake payment system that just provides an http response and that’s it.

  Q: – Could you please tell me if your application is written as one huge monolith, or have you broken it down into some services or even microservices?

  EK: – We are definitely not a monolith; we have a service-oriented application. We joke that our service consists of monoliths – they are indeed quite large. It wouldn't be accurate to call them microservices, but they function as services, within which distributed machine workers operate.

  If a service on the server is compromised…

  Q: – Then I have the next question. Even if it were a monolith, you still mentioned that you have many of these instant servers, all of which process data. The question is: 'In the event of a compromise of one of the instant servers or the application, a particular link, do they have some access control? What can each of them do? Who can they contact for what data?'

  
  
  EK: – Yes, undoubtedly. The security requirements are quite stringent. Firstly, we have open data movements, and only those ports are available that we anticipate traffic flow through. If a component communicates with a database (let's say, MySQL) through ports 5-4-3-2, only ports 5-4-3-2 will be open, and other ports, other traffic directions will not be accessible. Additionally, it should be understood that in production we have about 10 different security contours. And even if the application is somehow compromised, god forbid, the attacker will not be able to access the server management console, as it is in a different security network zone.

  Q: – In this context, I am more interested in the fact that you have certain contracts with the services – what they can do, which 'actions' they can call on each other… And in a normal flow, specific services request a certain range of 'actions' from others. They do not generally call upon others under normal circumstances, and they have different areas of responsibility. If one of them is compromised, will it be able to execute the 'actions' of that service?

  EK: I understand. If communication was generally allowed with a different server under normal circumstances, then yes. According to the SLA contract, we do not monitor that you are only allowed the first 3 "actions," and the 4th "action" is not permitted. This is probably excessive for us since we have a 4-level protection system in place for the contours. We prefer to defend ourselves at the contour level rather than at the internals level.

  How Visa, MasterCard, and Sberbank Work

  Q: I want to clarify the moment regarding the switching of a user from one data center to another. As far as I know, Visa and MasterCard operate on the binary synchronous protocol 8583, which has mixes. I wanted to know if the switching here refers directly to Visa and MasterCard or to the payment systems, to the processors?

  EK: This is before the mixes. Our mixes are located in one data center.

  Q: So you have one connection point, roughly speaking?

  EK: For Visa and MasterCard, yes. Simply because Visa and MasterCard require significant investments in infrastructure to conclude separate contracts for obtaining a second pair of mixes, for example. They are reserved within one data center, but if, God forbid, our data center where the mixes for connecting to Visa and MasterCard are located dies, we will lose connection with Visa and MasterCard...

  Q: How can they be reserved? I know that Visa only allows one connection, in principle!

  EK: They supply the equipment themselves. In any case, the equipment we received is internally redundantly reserved.

  Q: So the rack is from their Connects Orange?

  EK: – Yes.

  Q: So what happens in this case: if your data center goes down, how do you continue using it? Or does traffic simply stop?

  EK: No. In this case, we will simply switch traffic to another channel, which, of course, will be more expensive for us and more costly for clients. But traffic will not go through our direct connection to Visa and MasterCard, but through a conditional Sberbank (very exaggerated).

  I sincerely apologize if I offended Sberbank employees. But according to our statistics, among Russian banks, Sberbank fails most often. There isn't a month when Sberbank doesn't experience some outage.

  
  
  

  

  A little advertisement 🙂

  Thank you for staying with us. Do you enjoy our articles? Want to see more interesting content? Support us by placing an order or recommending us to your friends, cloud VPS for developers starting at $4.99, a unique entry-level server alternative that we have created for you: The whole truth about VPS (KVM) E5-2697 v3 (6 Cores) 10GB DDR4 480GB SSD 1Gbps from $19 or how to properly share a server? (options available with RAID1 and RAID10, up to 24 cores and up to 40GB DDR4).

  Dell R730xd at half the price in the Equinix Tier IV data center in Amsterdam? Only with us 2 x Intel TetraDeca-Core Xeon 2x E5-2697v3 2.6GHz 14C 64GB DDR4 4x960GB SSD 1Gbps 100TB starting at $199 in the Netherlands! Dell R420 — 2x E5-2430 2.2GHz 6C 128GB DDR3 2x960GB SSD 1Gbps 100TB — from $99! Read about how To build a corporate-class infrastructure using Dell R730xd E5-2650 v4 servers costing 9000 euros for peanuts?

Source: habr.com