We will share an intriguing story about how 'third parties' attempted to disrupt the operations of our clients and how this issue was resolved.
How It All Began
It all started on the morning of October 31, the last day of the month, when many urgently needed to resolve pressing and important matters.
One of our partners, who hosts several clients' virtual machines in our cloud, reported that from 9:10 to 9:20 several Windows servers operating in our Ukrainian data center were not accepting connections from the remote access service, and users were unable to access their desktops. However, a few minutes later, the issue seemed to resolve itself.
We checked the statistics for network performance but found no spikes in traffic or outages. Looking into the load statistics for computing resources showed no anomalies. So what was going on?
Then another partner, hosting nearly a hundred servers in our cloud, reported similar issues noted by some of their clients. It turned out that the servers were generally accessible (responding normally to ping tests and other queries), but the remote access service on these servers was intermittently accepting and rejecting new connections, despite the fact that the servers were located in different data centers with traffic incoming from various data transmission channels.
Let's take a look at that traffic. A packet with a connection request arrives at the server:
xx:xx:xx.xxxxxx IP xxx.xxx.xxx.xxx.58355 > 192.168.xxx.xxx.3389: Flags [S], seq 467744439, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
The server receives this packet but rejects the connection:
xx:xx:xx.xxxxxx IP 192.168.xxx.xxx.3389 > xxx.xxx.xxx.xxx.58355: Flags [R.], seq 0, ack 467744440, win 0, length 0
This means that the problem is clearly not related to any infrastructure failures, but something else. Could all users be having issues with remote desktop licensing? Could some malware have infiltrated their systems, and today it activated, similar to what happened a couple of years ago with XData and Petya?
While we were sorting this out, we received similar inquiries from several other clients and partners.
So what is actually happening on these machines?
The event registration logs are full of messages about password cracking attempts:
Typically, such attempts are logged on all servers where the default port (3389) is used for remote access and access is granted from everywhere. There are numerous bots on the Internet that continuously scan all available connection points and attempt to crack passwords (this is precisely why we strongly recommend using complex passwords instead of '123'). Nevertheless, the intensity of these attempts was excessively high that day.
What to do?
Is it a good idea to recommend clients spend a lot of time changing settings for a vast number of end users to switch to another port? Probably not a good idea; clients won't be happy. Should we recommend allowing access only via VPN? Rushing and panicking to set up IPSec connections for those who haven't done so might not be a cheerful prospect for clients either. Although, it's worth mentioning that it's always a good practice to hide the server in a private network, and we are ready to assist with the configurations. For those who prefer to handle it themselves, we share instructions for setting up IPSec/L2TP in our cloud in site-to-site or road-warrior mode. If someone wants to set up a VPN service on their own Windows server, we're always ready to provide tips on how to deploy a standard RAS or OpenVPN. However, regardless of how great we might be, this was not the best time for educational outreach among clients, as the priority was to resolve the issue as quickly as possible with minimal stress for users.
The solution we implemented was as follows. We set up traffic analysis to monitor all attempts to establish TCP connections to port 3389, selecting addresses that attempt to connect to more than 16 different servers in our network within 150 seconds; these are the sources of the attack (of course, if any of our clients or partners have a legitimate need to connect with such a number of servers from the same source, such sources can always be whitelisted. If more than 32 addresses from a single Class C network are detected during these 150 seconds, it makes sense to block the entire network. The block is set for 3 days, and if no attacks from this source occur during that time, the source is automatically removed from the blacklist. The list of blocked sources is updated every 300 seconds.
This list is available at the following address: , you can build your ACLs based on it.
We are ready to share the source code of such a system; there is nothing overly complex in it (just a few simple scripts, put together in literally a couple of hours on the fly), and it can be adapted and used not only to protect against such attacks but also to identify and block any scanning attempts in the network:
Additionally, we made some changes to the monitoring system settings, which now closely monitors the response of the control group of virtual servers in our cloud to attempts to establish RDP connections: if the response doesn't follow within a second, that is a reason to pay attention.
The solution turned out to be quite effective: there have been no complaints from clients, partners, or the monitoring system anymore. New addresses and entire networks regularly enter the blacklist, indicating that the attack is ongoing but no longer affects our clients' operations.
One in the field is not a warrior.
Today we learned that other operators have encountered a similar issue. Some still believe that Microsoft made changes to the remote access service code (if you remember, we suspected the same on the first day, but we quickly dismissed this theory) and promise to do everything possible to find a solution as soon as possible. Some simply ignore the problem and advise clients to protect themselves (changing the connection port, hiding the server in a private network, and so on). On the very first day, we not only resolved this issue but also laid the groundwork for a more global threat detection system that we plan to develop.
A special thanks to our clients and partners who did not remain silent and did not sit by the river waiting for a dead enemy to float by, but immediately brought the problem to our attention, allowing us to resolve it the same day.
Source: habr.com
