Legitimate traffic in the DDoS-Guard network recently exceeded one hundred gigabits per second. Now 50% of all our traffic is generated by client web services. These are many tens of thousands of domains, very different and in most cases requiring an individual approach.
Under the cut - how we manage front nodes and issue SSL certificates for hundreds of thousands of sites.
Setting up the front for one site, even a very large one, is easy. We take nginx or haproxy or lighttpd, set it up according to guides and forget it. If you need to change something, do reload and forget again.
Everything changes when you process large volumes of traffic on the fly, evaluate the legitimacy of requests, compress and cache user-generated content, and at the same time change parameters several times per second. The user wants to see the result on all external nodes immediately after he has changed the settings in his personal account. And the user can also download several thousand (and sometimes tens of thousands) domains via the API with individual traffic processing parameters. All this should also work immediately in America, and in Europe, and in Asia - the task is not the most trivial, given that in Moscow alone there are several physically separated filtration nodes.
Why many large reliable nodes around the world?
-
Quality of service for client traffic - requests from the US must be processed in the US (including for attacks, parsing and other anomalies), and not pulled to Moscow or Europe, unpredictably increasing the processing delay.
-
Attacking traffic must be localized - transit operators can degrade during attacks, the volume of which often exceeds 1Tbps. Transporting attack traffic over trans-Atlantic or trans-Asian links is not a good idea. We had real cases when Tier-1 operators said: "The volume of attacks that you accept is dangerous for us." That is why we take incoming streams as close as possible to their sources.
-
Strict requirements for continuity of service - cleaning centers should not depend on each other, nor on the local events of our rapidly changing world. Were all 11 floors of MMTS-9 de-energized for a week? - no problem. Not a single client that does not have a physical connection in this particular location will be affected, and web services will not be affected under any circumstances.
How to manage all this?
Service configurations should be distributed as quickly as possible (ideally instantly) to all front nodes. You canβt just take and rebuild text configs and restart daemons on each change - the same nginx keeps the processes ending (worker shutting down) for a few more minutes (or maybe hours if there are long websocket sessions).
When reloading the nginx configuration, the following picture is quite normal:
For memory recycling:
Old workers eat memory, including those that do not depend linearly on the number of connections - this is normal. When client connections are closed, this memory will be freed.
Why wasn't this a problem when nginx was just getting started? There was no HTTP/2, no WebSocket, no massive long keep-alive connections. 70% of our web traffic is HTTP/2, which means very long connections.
The solution is simple - do not use nginx, do not manage fronts based on text files, and certainly do not drive zip-packed text configurations across trans-Pacific channels. Channels, of course, guaranteed and reserved, but no less transcontinental.
We have our own front-server-balancer, the innards of which I will talk about in the following articles. The main thing that he can do is to apply thousands of configuration changes per second on the fly, without restarts, reloads, spasmodic growth in memory consumption, and all that. This is very similar to Hot Code Reload, for example in Erlang. The data is stored in a geo-distributed key-value database and read immediately by the fronts' executive mechanisms. Those. you uploaded an SSL certificate via a web interface or API in Moscow, and in a few seconds it is ready to work in our cleaning center in Los Angeles. If suddenly a world war happens and the Internet is lost all over the world, our nodes will continue to work autonomously and fix the split-brain as soon as one of the dedicated channels Los Angeles-Amsterdam-Moscow, Moscow-Amsterdam-Hong Kong-Los becomes available. Angeles or at least one of the backup GRE overlays.
This same mechanism allows us to instantly issue and renew Let's Encrypt certificates. Very simplified, it works like this:
-
As soon as we see at least one HTTPS request for our client's domain without a certificate (or with an expired certificate), the external node that accepted the request reports this to the internal certificate authority.
-
If the user has not prohibited the issuance of Let's Encrypt, the certification authority generates a CSR, receives a confirmation token from LE and sends it to all fronts via an encrypted channel. Now any node can confirm the validation request from LE.
-
In a few moments, we will receive the correct certificate and private key and send it to the fronts in the same way. Again without reloading the daemons
-
7 days before the date of expiration, the procedure for re-obtaining the certificate is initiated
Right now, we are rotating 350k certificates in real time, completely transparent to users.
In the following articles of the cycle, I will talk about other features of real-time processing of large web traffic - for example, about analyzing RTT using incomplete data to improve the quality of service for transit customers and, in general, about protecting transit traffic from terabit attacks, about delivering and aggregating information about traffic, about WAF, almost unlimited CDN and many mechanisms for optimizing the return of content.
Only registered users can participate in the survey. , you are welcome.
What would you like to know first?
-
Present in several = 14,3%Algorithms for clustering and analyzing the quality of web traffic<3
-
Present in several = 33,3%Internals of DDoS-Guard7 balancers
-
Present in several = 9,5%Protection of transit L3/L4 traffic2
-
Present in several = 0,0%Protecting websites on transit traffic0
-
Present in several = 14,3%Web Application Firewall3
-
Present in several = 28,6%Protection against parsing and clicks6
21 users voted. 6 users abstained.
Source: habr.com