{"id":35092,"date":"2019-10-31T22:02:18","date_gmt":"2019-10-31T19:02:18","guid":{"rendered":"https:\/\/prohoster.info\/blog\/bitriks24-bystro-podnyatoe-ne-schitaetsya-upavshim\/"},"modified":"2019-10-31T22:02:18","modified_gmt":"2019-10-31T19:02:18","slug":"bitriks24-bystro-podnyatoe-ne-schitaetsya-upavshim","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/bitriks24-bystro-podnyatoe-ne-schitaetsya-upavshim","title":{"rendered":"Bitrix24: \"What rises quickly is not considered to have fallen\"","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

As of today, the Bitrix24 service does not have hundreds of gigabits of traffic, and it does not have a huge server park (though there are, of course, quite a few existing). However, for many clients, it is the main work tool in their company; it is a truly business-critical application. Therefore, it cannot afford to go down at all. But what if a failure does occur, and the service \"raises\" itself so quickly that no one even notices? How is this failover implemented without losing quality of service or customer count? Alexander Demidov, the director of cloud services at Bitrix24, shared with our blog how the backup system has evolved over the seven years of the product's existence.<\/p>\n

\"Bitrix24:
\n<\/a><\/noindex>
\nWe launched Bitrix24 as a SaaS 7 years ago. The main challenge was probably this: before launching it publicly as a SaaS, this product existed only as a boxed solution. Clients purchased it from us, hosted it on their servers, and created a corporate portal \u2014 a unified solution for employee communication, file storage, task management, CRM \u2014 all that. By 2012, we decided we wanted to launch it as SaaS, administering it ourselves and ensuring fault tolerance and reliability. We gained experience in the process, because until then we simply didn't have it \u2014 we were just software manufacturers, not service providers. <\/p>\n

When launching the service, we understood that the most important thing was to ensure fault tolerance, reliability, and constant service availability, because if you have a simple regular site, like a store for example, and it goes down for an hour \u2014 you\u2019re the only one who suffers, losing orders and clients, but for your customer, that\u2019s not very critical. They might be upset, of course, but they\u2019ll just go and buy from another site. However, if it\u2019s an application on which all internal company work and communication depend, the key aspect is to gain user trust, meaning not letting them down and not going down. Because all work may come to a standstill if something inside stops working.<\/p>\n

Bitrix24 as SaaS<\/h4>\n

\nWe built the first prototype a year before the public launch, in 2011. It was assembled in about a week; we checked it out and even had a working version. This meant we could enter a portal name into the form, a new portal would be created, and a user database would be established. We took a look at it, evaluated the product, and shelved it for a whole year for further development. Our major goal was clear: we didn't want to maintain two different code bases. We aimed to avoid separately supporting a packaged product and cloud solutions \u2014 we wanted to do everything within a single codebase. <\/p>\n

\"Bitrix24:
\n
\nAt that time, a typical web application was just one server running some PHP code, a MySQL database, files uploaded, documents, and images stored in an upload folder \u2014 and that was the extent of it. Unfortunately, it was impossible to launch a critically resilient web service on that setup. It lacked support for distributed caching and database replication. <\/p>\n

We formulated the requirements: the ability to be deployed in various locations, support for replication, ideally distributed across different geographically located data centers. We needed to separate product logic from data storage. The system had to dynamically scale according to load, and static content needed to be offloaded completely. This formed the foundation of the requirements for the product, which we worked on refining for a year. During that time, we developed a unified platform \u2014 for both packaged solutions and our own service \u2014 supporting the necessary features. We implemented MySQL replication support at the product level: developers writing code do not need to worry about how their requests will be distributed; they simply use our API, and we handle the correct distribution of read and write requests between masters and slaves. <\/p>\n

We implemented support at the product level for various cloud object storage solutions: Google Cloud Storage, Amazon S3, in addition to OpenStack Swift. This was convenient for us as a service provider and for developers working with the packaged solutions: if they simply use our API for their operations, they do not need to worry about whether the file will be saved locally on the file system or in an object storage.<\/p>\n

As a result, we immediately decided to reserve resources at the level of an entire data center. In 2012, we fully launched on Amazon AWS because we already had experience with this platform \u2014 our own website was hosted there. We were attracted by the fact that in every region on Amazon there are several availability zones \u2014 essentially, (in their terminology) several data centers that are more or less independent from each other, allowing us to reserve at the level of an entire data center: if one of them fails, the databases are replicated master-master, web application servers are reserved, and static files are stored in S3 object storage. The load is balanced \u2014 at that time it was Amazon's ELB, but later we moved to our own load balancers because we needed more complex logic. <\/p>\n

We got exactly what we wanted\u2026<\/h4>\n

\nAll the basic things we wanted to ensure \u2014 the fault tolerance of the servers, web applications, and databases \u2014 all worked well. The simplest scenario: if any of our web applications fails, it\u2019s straightforward \u2014 they are removed from the load balancing. <\/p>\n

\"Bitrix24:
\n
\nThe failed machines were automatically marked as unhealthy by the load balancer (which at that time was Amazon's ELB) and removed from the load distribution. Amazon's auto-scaling worked: when the load increased, new machines were added to the auto-scaling group, and the load was distributed to them \u2014 everything was fine. With our load balancers, the logic is roughly the same: if something happens to the application server, we stop sending requests to it, remove those machines, start new ones, and continue working. The scheme has changed slightly over the years, but it continues to work: it is simple, clear, and there are no complications. <\/p>\n

We operate worldwide, and our clients experience varying load peaks. Ideally, we should be able to perform service work on any component of our system at any time \u2014 without clients noticing. Therefore, we have the ability to temporarily shut down the database, redistributing the load to the second data center. <\/p>\n

How does it all work? - We redirect traffic to the operational data center - if there is a failure at the data center, then completely; if it is our scheduled maintenance with a specific database, we redirect part of the traffic servicing those clients to the second data center, and replication is paused. If new machines are needed for web applications due to increased load at the second data center, they automatically start. We finish the work, replication is restored, and we return all the load back. If we need to perform mirrored work at the second DC, for example, installing system updates or changing settings in the second database, we generally repeat the same process, just in the opposite direction. And if it\u2019s a failure, we handle it quite straightforwardly: we use the event-handlers mechanism in the monitoring system. If several checks are triggered and the status goes to critical, this handler is activated, which can execute specific logic. We have designated for each database which server acts as its failover, and where to redirect traffic in case of its unavailability. Historically, we have been using Nagios or some of its forks in one form or another. Essentially, similar mechanisms are present in almost any monitoring system; we are not currently using anything more complex, but that may change in the future. Right now, monitoring triggers on unavailability and has the capability to redirect something.<\/p>\n

Have we reserved everything?<\/h4>\n

\nWe have many clients from the USA, many clients from Europe, and many clients closer to the East \u2014 Japan, Singapore, and so on. Of course, a significant portion of our clients is in Russia. This means we operate across multiple regions. Users desire fast response times and there are requirements for compliance with various local laws. Within each region, we secure two data centers, plus there are additional services that are convenient to deploy within a region for clients operating there. REST handlers and authorization servers are less critical for overall client operations; switching between them can be done with a slight acceptable delay, but we don\u2019t want to reinvent the wheel regarding monitoring and managing them. Therefore, we try to use existing solutions as much as possible instead of developing competencies in additional products. In some instances, we simply use DNS-level switching, determining service availability through the same DNS. Amazon has the Route 53 service, but it's not just a DNS where you can enter records and that's it \u2014 it's much more flexible and convenient. You can build geo-distributed services with geolocations through it, allowing you to determine where the client came from and provide them with particular records \u2014 it enables the construction of failover architectures. The same health checks can be configured in Route 53; you specify the endpoints to be monitored, set metrics, and specify which protocols to use to determine the service's 'liveness' \u2014 tcp, http, https; you define the frequency of checks that determine whether the service is live or not. And in the DNS, you specify which will be primary, which will be secondary, and where to switch if a health check is triggered within Route 53. This can all be done with other tools as well, but the convenience is that you configure it once and then don\u2019t think at all about how the checks are performed and how the switching occurs: everything works automatically.<\/p>\n

The first 'but'<\/b>: But how and with what should we back up Route 53? Just in case something happens with it? Fortunately, we have never run into this issue, but again, I will share why we thought it necessary to have a backup. Here we're laying down some precautionary measures in advance. Several times a day, we perform a complete export of all zones registered in Route 53. Amazon's API allows us to export this data in JSON format, and we have set up several backup servers where we convert, export this as configurations, and essentially have a backup configuration. In case of any issues, we can quickly redeploy it manually and will not lose DNS settings data.<\/p>\n

The second 'but'<\/b>: what else is not backed up in this picture? The load balancer! We distribute clients across regions very simply. We have the domains bitrix24.ru, bitrix24.com, .de \u2014 currently there are about 13 different ones that operate in various zones. We came to the following conclusion: each region has its own load balancers. This makes it easier to distribute traffic by region, depending on where the peak network load occurs. If there's a failure at the level of a single load balancer, it is simply taken out of operation and removed from DNS. If there\u2019s a problem with a group of load balancers, they can be backed up at other locations, and the switch between them is done using Route 53, because due to the short TTL, the switching happens in a maximum of 2, 3, or 5 minutes. <\/p>\n

The third 'but'<\/b>: what else is not backed up? S3, right. When we store files that we keep for users in S3, we genuinely believed it to be foolproof and that there was no need to back it up. But history shows that things happen differently. In general, Amazon describes S3 as a fundamental service because Amazon itself uses S3 to store machine images, configurations, AMI images, snapshots\u2026 And if S3 goes down, as it did once in these 7 years that we have been operating Bitrix24, it causes a ripple effect \u2014 making virtual machines unavailable, causing API failures, and so on. <\/p>\n

An S3 failure can happen \u2014 it has happened before. Therefore, we arrived at the following scheme: several years ago, there were no serious public object storage solutions in Russia, and we considered creating our own... Fortunately, we did not proceed with that, as we would have gotten mired in expertise that we lacked and would have likely made mistakes. Now, S3-compatible storage solutions exist with Mail.ru, Yandex, and several other providers. Ultimately, we concluded that we want to have, first of all, redundancy, and second, the ability to work with local copies. For the specific Russian region, we use the Mail.ru Hotbox service, which is compatible with S3 via API. We did not require any significant modifications to the application code, and we implemented the following mechanism: there are triggers in S3 that activate upon the creation\/deletion of objects; Amazon has a service called Lambda \u2014 this is serverless code execution that runs precisely when these triggers are activated.<\/p>\n

\"Bitrix24:
\n
\nWe kept it very simple: when a trigger activates, we execute code that copies the object to Mail.ru storage. To fully initiate work with local data copies, we also need reverse synchronization so that clients within the Russian segment can work with the storage that is closer to them. Mail is about to finalize its triggers in its storage \u2014 soon it will be possible to execute reverse synchronization at the infrastructure level, but for now, we are handling it at the level of our own code. If we see that a client has uploaded a file, we place an event in a queue at our code level, process it, and perform reverse replication. The downside of this is that if any work is done with our objects outside our product, that is, through some external means, we will not account for it. Therefore, we are waiting until the triggers at the storage level are available so that regardless of where we executed the code, the object that came to us would be copied in the opposite direction. <\/p>\n

At the code level, we define two storage options for each client: one is considered primary, and the other as backup. When everything is functioning properly, we operate with the storage that is closer to us: our clients in Amazon work with S3, while those in Russia use Hotbox. If a flag is triggered, we should switch to failover, and we redirect clients to the alternate storage. We can set this flag independently by region and toggle them as needed. In practice, we haven't used this yet, but we have planned for this mechanism, believing that at some point this switching will be necessary. It has happened once before. <\/p>\n

Oh, has your Amazon run away...<\/h4>\n

\nThis April marks the anniversary of the beginning of Telegram blockages in Russia. The provider that suffered the most from this is Amazon. Unfortunately, it was mainly Russian companies that operated globally that were affected. <\/p>\n

If a company is global and Russia represents a tiny segment for it, say 3-5%, then they may have to sacrifice it, one way or another. <\/p>\n

If it's strictly a Russian company, I'm sure it should host locally\u2014it's simply more convenient for users, and it reduces risks. <\/p>\n

But if it's a company that operates globally, with about equal numbers of clients in Russia and elsewhere around the world? The interconnection of segments is important, and they must work together in some way. <\/p>\n

Back in late March 2018, Roskomnadzor sent a letter to the largest operators announcing their plans to block several million Amazon IPs to block... the Zello messenger. Thanks to those providers, they successfully leaked the letter to everyone, leading to the understanding that the connectivity with Amazon might collapse. It was a Friday, and we panicked and rushed to our colleagues from servers.ru, saying: 'Friends, we need several servers that are not in Russia, not in Amazon, but perhaps somewhere in Amsterdam,' so we could at least have some means of establishing our own presence there. vpn<\/a> And a proxy for certain endpoints that we cannot influence, for example, the endpoints of S3 \u2014 we can't try to launch a new service and get a different IP; we still need to connect to it. Within a few days, we configured and deployed these servers, and by the time the blocking began, we were prepared. Interestingly, the Russian Communications Oversight (Roskomnadzor), seeing the commotion and the panic that ensued, said: 'No, we will not block anything at this moment.' (But that was right up until they started blocking Telegram.) After setting up bypass capabilities and realizing that no blocking was implemented, we nevertheless decided not to dismantle everything just in case. <\/p>\n

\"Bitrix24:
\n
\nSo, in 2019, we are indeed living under conditions of blockage. Just last night I was looking: about a million IP addresses continue to be blocked. However, Amazon was mostly unblocked, peaking at 20 million addresses... In general, the reality is that connectivity, good connectivity \u2014 may not exist. Suddenly. It may be absent for technical reasons \u2014 fires, excavators, all that. Or, as we have seen, for not entirely technical reasons. Therefore, someone large and significant, with their own AS numbers, can probably manage this in other ways \u2014 direct connect and other solutions already at the L2 level. But in a simpler scenario, like us or even smaller ones, it is wise to have redundancy at the server level, set up elsewhere, pre-configured VPNs, proxies, with the ability to quickly switch configurations in those segments that are critical to your connectivity. This has helped us more than once when Amazon blocking began; we routed S3 traffic through them in the worst case, but gradually everything settled down.<\/p>\n

And how to reserve... an entire provider?<\/h4>\n

\nCurrently, we do not have a contingency plan for a total outage of Amazon. We have a similar plan for Russia. In Russia, we partnered with a provider where we selected multiple locations. A year ago, we faced a problem: despite having two data centers, there can be network configuration issues at the provider level that might affect both data centers. Consequently, we could experience downtime at both locations. And that's exactly what happened. We ultimately reassessed our architecture internally. It hasn't changed significantly, but for Russia, we now have two locations that are with two different providers. If one fails, we can switch to the other.<\/p>\n

Hypothetically, we are considering reserving at another provider for Amazon; maybe Google, or someone else... But so far, we've observed in practice that when Amazon has incidents at the level of one availability zone, incidents at the level of an entire region are relatively rare. Therefore, we theoretically have an idea that we might do a reservation of 'Amazon - non-Amazon', but in practice, that doesn't exist yet. <\/p>\n

A few words about automation<\/h4>\n

\nIs automation always necessary? Here it's appropriate to recall the Dunning-Kruger effect. On the x-axis, we have our knowledge and experience that we accumulate, and on the y-axis, our confidence in our actions. Initially, we know nothing and are not confident at all. Then we know a little and become overly confident \u2014 this is the so-called 'peak of ignorance', well illustrated by the image of 'incompetence and courage'. Afterward, we learn a bit more and are ready to go into battle. Then we step on some serious rakes, finding ourselves in the valley of despair, when it seems we know something, but in reality, we don't know much. As we gain experience, we become more confident.<\/p>\n

\"Bitrix24:
\n
\nOur logic regarding different switches automatically in response to various incidents is well illustrated by this graph. We started with no experience, performing almost all tasks manually. Then we realized that we could automate everything and feel at ease. Suddenly, we hit a major snag: a false positive triggers, and we switch traffic back and forth when we really shouldn't. Consequently, replication fails or something else goes wrong \u2014 that's the infamous valley of despair. Eventually, we come to understand that we need to approach everything thoughtfully. It makes sense to rely on automation while anticipating the possibility of false alarms. However, if the consequences could be catastrophic, it's better to delegate this to the on-duty shift, to the engineers on call, who will verify that an incident has indeed occurred and take the necessary actions manually\u2026<\/p>\n

Conclusion<\/h4>\n

\nIn 7 years, we have transitioned from panic whenever something went down to understanding that there are no problems, only tasks that need to be addressed \u2014 and can be solved. When building a service, take an overarching view, evaluate all potential risks. If you can see the risks upfront, make provisions for redundancy and the ability to construct a fault-tolerant infrastructure, because any point that could fail and lead to service outages will inevitably do so. Even if you believe certain infrastructure elements won't fail \u2014 like s3, keep in mind that they might. At the very least, have a theoretical plan for what you will do if something happens. Have a risk management strategy. When considering whether to automate everything or handle tasks manually, assess the risks: what will happen if the automation starts switching things \u2014 will that lead to an even worse outcome than the incident itself? Perhaps there's a need to strike a reasonable balance between using automation and having the on-duty engineer react, who can evaluate the real situation and decide whether something needs to be switched immediately or if it can wait.<\/p>\n

A reasonable compromise between perfectionism and the real forces, time, and money you can spend on the scheme you will ultimately have.<\/p>\n

This text is an enhanced and expanded version of Alexander Demidov's report at the conference. Uptime day 4<\/a><\/noindex>.<\/i>
\n
Source: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

\u041d\u0430 \u0441\u0435\u0433\u043e\u0434\u043d\u044f\u0448\u043d\u0438\u0439 \u0434\u0435\u043d\u044c \u0443 \u0441\u0435\u0440\u0432\u0438\u0441\u0430 \u00ab\u0411\u0438\u0442\u0440\u0438\u043a\u044124\u00bb \u043d\u0435\u0442 \u0441\u043e\u0442\u0435\u043d \u0433\u0438\u0433\u0430\u0431\u0438\u0442 \u0442\u0440\u0430\u0444\u0438\u043a\u0430, \u043d\u0435\u0442 \u043e\u0433\u0440\u043e\u043c\u043d\u043e\u0433\u043e \u043f\u0430\u0440\u043a\u0430 \u0441\u0435\u0440\u0432\u0435\u0440\u043e\u0432 (\u0445\u043e\u0442\u044f \u0438 \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u044e\u0449\u0438\u0445, \u043a\u043e\u043d\u0435\u0447\u043d\u043e, \u043d\u0435\u043c\u0430\u043b\u043e). \u041d\u043e \u0434\u043b\u044f \u043c\u043d\u043e\u0433\u0438\u0445 \u043a\u043b\u0438\u0435\u043d\u0442\u043e\u0432 \u043e\u043d \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u043e\u0441\u043d\u043e\u0432\u043d\u044b\u043c \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u043e\u043c \u0440\u0430\u0431\u043e\u0442\u044b \u0432 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438, \u044d\u0442\u043e \u043d\u0430\u0441\u0442\u043e\u044f\u0449\u0435\u0435 business-critical \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0435. \u041f\u043e\u044d\u0442\u043e\u043c\u0443 \u043f\u0430\u0434\u0430\u0442\u044c \u2014 \u043d\u0443, \u043d\u0438\u043a\u0430\u043a \u043d\u0435\u043b\u044c\u0437\u044f. \u0410 \u0447\u0442\u043e \u0435\u0441\u043b\u0438 \u043f\u0430\u0434\u0435\u043d\u0438\u0435 \u0432\u0441\u0435-\u0442\u0430\u043a\u0438 \u0441\u043b\u0443\u0447\u0438\u043b\u043e\u0441\u044c, \u043d\u043e \u00ab\u0432\u043e\u0441\u0441\u0442\u0430\u043b\u00bb \u0441\u0435\u0440\u0432\u0438\u0441 \u0442\u0430\u043a \u0431\u044b\u0441\u0442\u0440\u043e, \u0447\u0442\u043e \u043d\u0438\u043a\u0442\u043e \u043d\u0438\u0447\u0435\u0433\u043e \u0438 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":26389,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-35092","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47'Bitrix24': 'What is quickly raised does not count as fallen' | ProHoster","description":"As of today, the 'Bitrix24' service does not have hundreds of gigabits of traffic, nor a vast server park (although there are, of course, quite a few existing).","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/bitriks24-bystro-podnyatoe-ne-schitaetsya-upavshim","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u00ab\u0411\u0438\u0442\u0440\u0438\u043a\u044124\u00bb: \u00ab\u0411\u044b\u0441\u0442\u0440\u043e \u043f\u043e\u0434\u043d\u044f\u0442\u043e\u0435 \u043d\u0435 \u0441\u0447\u0438\u0442\u0430\u0435\u0442\u0441\u044f \u0443\u043f\u0430\u0432\u0448\u0438\u043c\u00bb | ProHoster","og:description":"\u041d\u0430 \u0441\u0435\u0433\u043e\u0434\u043d\u044f\u0448\u043d\u0438\u0439 \u0434\u0435\u043d\u044c \u0443 \u0441\u0435\u0440\u0432\u0438\u0441\u0430 \u00ab\u0411\u0438\u0442\u0440\u0438\u043a\u044124\u00bb \u043d\u0435\u0442 \u0441\u043e\u0442\u0435\u043d \u0433\u0438\u0433\u0430\u0431\u0438\u0442 \u0442\u0440\u0430\u0444\u0438\u043a\u0430, \u043d\u0435\u0442 \u043e\u0433\u0440\u043e\u043c\u043d\u043e\u0433\u043e \u043f\u0430\u0440\u043a\u0430 \u0441\u0435\u0440\u0432\u0435\u0440\u043e\u0432 (\u0445\u043e\u0442\u044f \u0438 \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u044e\u0449\u0438\u0445, \u043a\u043e\u043d\u0435\u0447\u043d\u043e, \u043d\u0435\u043c\u0430\u043b\u043e).","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/bitriks24-bystro-podnyatoe-ne-schitaetsya-upavshim","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:02:18+00:00","article:modified_time":"2019-10-31T19:02:18+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"35092","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-02-04 14:51:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 02:10:27","updated":"2026-02-04 14:51:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/35092","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=35092"}],"version-history":[{"count":1,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/35092\/revisions"}],"predecessor-version":[{"id":156668,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/35092\/revisions\/156668"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/26389"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=35092"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=35092"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=35092"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}