Let's tell you a cool story about how "third parties" tried to interfere with our customers, and how this problem was solved.
How it all began
It all started on the morning of October 31, the last day of the month, when many desperately need to have time to close urgent and important issues.
One of the partners, who keeps several virtual machines of the clients he serves in our cloud, reported that from 9:10 to 9:20, several Windows servers running on our Ukrainian site at once did not accept connections to the remote access service , users could not access their desktops, but after a few minutes the problem seemed to disappear by itself.
We raised the statistics of the communication channels, but did not find any bursts of traffic or failures. We looked into the statistics of the load on computing resources - no anomalies. And what was that?
Then another partner, who hosts another hundred servers in our cloud, reported the same problems that some of their clients noted, while it turned out that in general the servers are available (correctly respond to a ping test and other requests), but the service remote access on these servers either accepts new connections or rejects them, while we were talking about servers at different sites, the traffic to which comes from different data transmission channels.
Let's take a look at this traffic. A packet with a request to establish a connection arrives at the server:
xx:xx:xx.xxxxxx IP xxx.xxx.xxx.xxx.58355 > 192.168.xxx.xxx.3389: Flags [S], seq 467744439, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
The server receives this packet, but the connection is rejected:
xx:xx:xx.xxxxxx IP 192.168.xxx.xxx.3389 > xxx.xxx.xxx.xxx.58355: Flags [R.], seq 0, ack 467744440, win 0, length 0
This means that the problem is clearly not caused by any malfunctions in the infrastructure, but by something else. Maybe all users have problems with remote desktop licensing? Maybe some kind of malware managed to infiltrate their systems, and today it is activated, as it was with a couple of years ago. XData и Petya?
While sorting it out, we received similar requests from several more clients and partners.
What actually happens on these machines?
The event logs are full of messages about trying to guess a password:
Typically, such attempts are logged on all servers where the standard port (3389) is used for the remote access service and access from anywhere is allowed. The Internet is full of bots that constantly scan all available connection points and try to guess the password (this is the reason why we strongly recommend using complex passwords instead of "123"). However, the intensity of these attempts that day was too high.
How to proceed?
Recommend that customers spend a lot of time changing the settings of a huge number of end users in order to switch to a different port? Not a good idea, customers won't be happy. Recommend allowing access only through VPN? In a hurry and panic, raising IPSec connections for those who have not raised them - perhaps such happiness does not smile at customers either. Although, I must say, this is a charitable matter in any case, we always recommend hiding the server in a private network and are ready to help with the settings, and for those who like to figure it out on their own, we share instructions for setting up IPSec / L2TP in our cloud in site-to-site or road mode -warrior, and if anyone wants to set up a VPN service on their own Windows server, they are always ready to share tips on how to set up a standard RAS or OpenVPN. But as cool as we were, it wasn't the best time to do customer education because we needed to fix the problem as quickly as possible with as little disruption to users as possible.
The solution we implemented was as follows. We set up the analysis of passing traffic in such a way as to track all attempts to establish a TCP connection to port 3389 and select addresses from it that try to establish connections with more than 150 different servers in our network for 16 seconds - these are the sources of attack ( of course, if one of the clients or partners has a real need to establish connections with so many servers from the same source, you can always add such sources to the “white list.” Moreover, if in one class C network for these 150 seconds, more than 32 addresses have been identified, it makes sense to block the entire network. The block is set for 3 days, and if no attacks were made from this source during this time, this source is removed from the "black list" automatically. The list of blocked sources is updated every 300 seconds.
This list is available here: , you can build your ACLs based on it.
We are ready to share the source code of such a system, there is nothing super complicated in it (these are a few simple scripts compiled in just a couple of hours “on the knee”), and at the same time it can be adapted and used not only to protect against such an attack, but also to detect and blocking any attempts to scan the network:
In addition, we made some changes to the settings of the monitoring system, which now monitors more closely the reaction of the control group of virtual servers in our cloud to an attempt to establish an RDP connection: if the reaction does not follow within a second, this is a reason to pay attention.
The solution turned out to be quite effective: there are no more complaints from both clients and partners, and from the monitoring system. New addresses and entire networks regularly get blacklisted, which indicates that the attack continues, but no longer affects the work of our clients.
One in the field is not a warrior
Today we learned that other operators faced a similar problem. Someone still believes that Microsoft made some changes to the code of the remote access service (if you remember, we suspected the same thing on the first day, but we rejected this version very soon) and promises to do everything possible to rather find a solution. Someone simply ignores the problem and advises clients to defend themselves (change the connection port, hide the server in a private network, and so on). And on the very first day, we not only solved this problem, but also created some groundwork for a more global threat detection system that we plan to develop.
Special thanks to customers and partners who did not remain silent and did not sit on the banks of the river waiting for the corpse of the enemy to float along it one day, but immediately drew our attention to the problem, which gave us the opportunity to eliminate it on the same day.
Source: habr.com