A system for analyzing traffic without decrypting it. This method is simply called 'machine learning.' It turns out that if a large volume of various traffic is fed into a special classifier, the system can detect malicious code actions within encrypted traffic with a very high degree of certainty.
Network threats have evolved and become 'smarter.' Recently, the concept of attack and defense has changed. The number of events in the network has increased significantly. Attacks have become more sophisticated, and hackers now have a broader range of attacks.
According to Cisco statistics, in the past year, attackers have tripled the number of malicious programs used for their activities, specifically for concealment via encryption. It is known that a 'secure' encryption algorithm cannot be cracked. To understand what is hidden within encrypted traffic, one must either decrypt it knowing the key, attempt to decrypt it with various tricks, use brute force, or exploit vulnerabilities in cryptographic protocols.
The Landscape of Network Threats Today
Machine Learning
Know the Technology! Before discussing how the decryption technology works based on machine learning, it's essential to break down how neural network technology functions.
Machine Learning (Machine Learning) is a vast subfield of artificial intelligence studying methods for building algorithms capable of learning. This science focuses on creating mathematical models for 'teaching' computers. The aim of this learning process is to make predictions about something. In human terms, we call this process 'wisdom.' wisdomWisdom manifests in people who have lived many years (a 2-year-old child cannot be wise). When seeking advice from older peers, we provide them with some information about the event (input data) and ask for help. They, in turn, recall all the situations from their lives that relate to your problem (knowledge base) and based on that knowledge (data) give us a kind of prediction (advice). This type of advice has been called a prediction because the person giving it cannot know for sure what will happen but can only speculate. Life experience shows that a person may be right or may make a mistake.
It is not appropriate to compare neural networks with a branching algorithm (if-else). These are different things, and there are key differences between them. The branching algorithm has a clear 'understanding' of what to do. Let me demonstrate with examples.
Task. Determine the braking distance of a car based on its make and year of manufacture.
Example of branching algorithm operation. If the car is make 1 and was manufactured in 2012 – its braking distance is 10 meters; otherwise, if the car is make 2 and was manufactured in 2011, and so on.
Example of neural network operation. We collect data on the braking distances of cars over the past 20 years. By make and year, we create a table in the form of 'make-year of manufacture-braking distance'. We provide this table to the neural network and start training it. The training is carried out as follows: we input data into the neural network, but without the braking distance. The neural network attempts to predict what the braking distance will be based on the table loaded into it. It makes a prediction and asks the user, 'Am I right?'. Before the question, it creates a fourth column – the guessing column. If it is correct, it writes 1 in the fourth column, if it is wrong, it writes 0. The neural network moves to the next event (even if it was wrong). This is how the network learns, and when the training is complete (a certain convergence criterion is reached), we input data about the car we are interested in and finally get an answer.
To clarify the question about the convergence criterion, it is a mathematically derived formula for statistics. A striking example of two different convergence formulas. The red line represents binary convergence, and the blue line represents normal convergence.
Binomial and normal probability distributions
To clarify, ask the question, "What is the probability of encountering a dinosaur?" There are 2 possible answers. Option 1 – very low (blue graph). Option 2 – either encounter or not (red graph).
Of course, a computer is not a human and learns in a different way. There are 2 types of learning for a machine – precedent-based learning and deductive learning.
Precedent-based learning is a method of learning based on mathematical laws. Mathematicians gather statistical tables, draw conclusions, and upload the result – a formula for calculation – into the neural network.
Deductive learning occurs entirely within the neural network (from data collection to analysis). A table is formed here without a formula, but with statistics.
A comprehensive overview of the technology would take a few more dozen articles. For now, this is enough for a general understanding.
Neuroplasticity
In biology, there is a concept called neuroplasticity. Neuroplasticity is the ability of neurons (brain cells) to act "according to the situation." For example, a person who has lost their eyesight hears sounds better, senses odors, and feels objects. This happens because part of the brain (part of the neurons) responsible for vision reallocates its work to other functionalities.
A vivid example of neuroplasticity in life is the BrainPort candy.
In 2009, the University of Wisconsin-Madison announced the release of a new device that developed the ideas of a "language display" – it was named BrainPort. BrainPort operates according to the following algorithm: the video signal from the camera is sent to a processor that controls the zoom, brightness, and other picture parameters. It also converts digital signals into electrical impulses, effectively taking on the functions of the retina.
BrainPort candy with glasses and a camera
BrainPort in action
The same goes for computers. If a neural network detects a change in the process, it adapts to it. This is the key advantage of neural networks compared to other algorithms – autonomy. A sort of humanity.
Encrypted Traffic Analytics
Encrypted Traffic Analytics is part of the Stealthwatch system. Stealthwatch is a solution developed by Cisco in the field of security monitoring and analytics, utilizing telemetry data from the existing network infrastructure.
Stealthwatch Enterprise is based on the Flow Rate License, Flow Collector, Management Console, and Flow Sensor tools.
Cisco Stealthwatch Interface
The issue of encryption has become quite pressing due to the significant increase in traffic being encrypted. Previously, only code was mainly encrypted; now, all traffic is encrypted, making it significantly harder to separate 'clean' data from viruses. A stark example is WannaCry, which used Tor to conceal its presence in the network.
Visualization of the growth in encrypted traffic on the network
Encryption in macroeconomics
The Encrypted Traffic Analytics (ETA) system is essential for working with encrypted traffic without decryption. Malicious actors are clever and use cryptographically strong encryption algorithms, making it not only a challenge to crack them but also incurs enormous costs for organizations.
The system operates as follows. Traffic comes into the company. It enters TLS (Transport Layer Security — a transport layer protection protocol). Let's say the traffic is encrypted. We try to answer a series of questions about the nature of the established connection.
Principle of operation of the Encrypted Traffic Analytics (ETA) system
To answer these questions, we utilize machine learning in this system. Research from Cisco is used to create a table with two outcomes — malicious and 'good' traffic. Of course, we cannot know for certain what traffic has entered the system at any given moment, but we can track the history of traffic both within and outside the company by leveraging global data. At the output of this stage, we obtain a large dataset.
The study reveals characteristic signs — specific rules that can be expressed in mathematical form. These rules will vary greatly across different criteria, such as the size of transmitted files, type of connection, the country from which the traffic originates, etc. As a result, an enormous table was transformed into a set of numerous formulas. They have become fewer, but this is still insufficient for comfortable operation.
Next, machine learning technology is applied — convergence of the formula, and as a result of this convergence, we obtain a trigger — a switch that indicates whether the output data has triggered a flag in an up or down position.
The resulting stage involves obtaining a set of triggers that cover 99% of the traffic.
Traffic verification stages in ETA
As a result, another issue is resolved — internal attacks. There is no longer a need for intermediaries to manually filter traffic (in this regard, I am undermining myself). Firstly, there's no longer a need to spend a lot of money on a skilled system administrator (I continue to undermine myself). Secondly, there is reduced risk of internal hacking (at least partially).
Outdated concept of Man-in-the-Middle
Now, let's clarify what the system is based on.
The system operates on four communication protocols: TCP/IP — data transmission protocol on the internet, DNS — domain name server, TLS — transport layer security protocol, SPLT (SpaceWire Physical Layer Tester) — physical layer communication tester.
Protocols working with ETA
The comparison is made by matching data. The reputation of websites is checked via TCP/IP protocols (history of visits, purpose of the website, etc.); thanks to the DNS protocol, we can filter out 'bad' website addresses. The TLS protocol works with website 'fingerprints' and verifies the site against a database of computer emergency response teams (CERT). The final stage of connection verification is done at the physical level. The details of this stage are not specified, but the essence lies in checking the sine and cosine waves of data transmission curves on oscillographic setups; thus, through the structure of the request at the physical level, we determine the purpose of the connection.
As a result of the system's operation, we can extract data from encrypted traffic. By studying the packets, we can read maximum information from the unencrypted fields within the packet itself. Through packet inspection at the physical level, we can determine the characteristics of the packet (partially or completely). We should also remember the reputation of the websites. If the request came from a .onion source, it is not advisable to trust it. To simplify working with such data, a risk map has been created.
ETA Performance Result
And everything seems fine, but let's talk about network deployment.
Physical Implementation of ETA
Here, a number of nuances and subtleties arise. Firstly, when creating such a
network with high-level software, data collection is necessary. Collecting data manually is quite
wild, and implementing a response system is already more interesting. Secondly, the data
must be plentiful, which means that the installed network sensors must operate
not only autonomously but also in a finely tuned mode, which presents several challenges.
Sensors and the Stealthwatch System
Installing a sensor is one thing, but configuring it is a completely different task. For sensor configuration, there is a complex that works on the following topology – ISR = Cisco Integrated Services Router; ASR = Cisco Aggregation Services Router; CSR = Cisco Cloud Services Router; WLC = Cisco Wireless LAN Controller; IE = Cisco Industrial Ethernet; ASA = Cisco Adaptive Security Appliance; FTD = Cisco Firepower Threat Defense; WSA = Web Security Appliance; ISE = Identity Services Engine.
Comprehensive Monitoring Considering Any Telemetric Data
Network administrators feel a sense of arrhythmia at the number of times "Cisco" appeared in the previous paragraph. The price of this marvel is substantial, but that’s not what we’re discussing today...
The modeling of hacker behavior will occur as follows. Stealthwatch meticulously monitors the activity of each device on the network and is capable of creating a pattern of normal behavior. Moreover, this solution provides a deep understanding of known malicious behavior. This solution employs about 100 different analysis algorithms or heuristic rules that pertain to various types of traffic behavior, such as scanning, transmitting emergency frames from nodes, brute-force login attempts, suspected data capture, suspected data leakage, etc. The listed security events fall under the category of high-level logical alerts. Some security events may also trigger alarms on their own. Thus, the system can correlate numerous isolated anomalous incidents and assemble them to determine the possible type of attack, as well as tie it to a specific device and user (see Figure 2). Subsequently, the incident can be examined in dynamics and with consideration of related telemetry data. This forms contextual information at its best. Doctors examining a patient to understand what happened do not look at symptoms in isolation. They study the overall picture to make a diagnosis. Similarly, Stealthwatch captures every anomalous activity on the network and analyzes it holistically to send alerts in context, thereby assisting security professionals in prioritizing risks.
Anomaly detection through behavior modeling
The physical network deployment looks like this:
Branch network deployment option (simplified)
Branch network deployment option
The network is deployed, but there remains an open question about the neural network. A data transmission network has been organized, sensors have been installed at thresholds, and a data collection system has been initiated, but the neural network has not participated in the process. For now.
Multi-layer neural network
The system analyzes user and device behavior to identify malware infections, communications with command servers, data leaks, and potentially unwanted applications operating within the organization’s infrastructure. There are several levels of data processing, where a combination of artificial intelligence, machine learning, and mathematical statistics helps the network to self-learn its normal activity, allowing it to detect malicious activities.
The network security analysis pipeline, which collects telemetry data from all parts of the extended network, including encrypted traffic, is a unique feature of Stealthwatch. It gradually builds an understanding of what constitutes 'anomalous' behavior, then categorizes specific elements of 'threat activity', and finally renders a conclusive decision on whether a device or user has actually been compromised. The ability to piece together small fragments, collectively forming evidence for making a final decision about a compromise, is ensured through meticulous analysis and correlation.
This capability is crucial, as a standard enterprise may receive an enormous number of alarm signals daily, making it impossible to investigate each one—security experts' resources are limited. The machine learning module processes a vast amount of information almost in real-time to identify critical incidents with a high level of confidence, and it can also suggest a clear course of action for rapid resolution.
Let’s delve deeper into the numerous machine learning methods used by Stealthwatch. When an incident is sent to the Stealthwatch machine learning module, it undergoes a security analysis funnel, within which a combination of supervised and unsupervised machine learning methods is utilized.
Multilevel machine learning capabilities
Level 1. Anomaly detection and trust modeling
At this level, 99% of traffic is filtered out using statistical anomaly detectors. These sensors collectively form complex models of what is considered normal and what is deemed anomalous. However, not all anomalies are necessarily malicious. Much of what occurs in your network is not related to a threat—it's just unusual. It's important to classify such processes regardless of threatening behavior. For this reason, the results from such detectors undergo further analysis to identify strange behavior that can be explained and trusted. Ultimately, only a small fraction of the most critical flows and requests move to levels 2 and 3. Without such machine learning methods, operational costs for separating the signal from the noise would be prohibitively high.
Anomaly detection. In the initial stage of anomaly detection, statistical machine learning methods are used to separate statistically normal traffic from anomalous traffic. More than 70 individual detectors process telemetry data collected by Stealthwatch about the traffic passing through your network perimeter, isolating internal traffic of the domain name system (DNS) and proxy server data when available. Each request is processed by over 70 detectors, each employing its own statistical algorithm, generating an anomaly score. These scores are combined, resulting in a single assessment for each individual request through the application of multiple statistical methods. This aggregate score is then used to distinguish between normal and anomalous traffic.
Trust modeling. Then similar requests are grouped, and the aggregate anomaly assessment for such groups is determined as a long-term average. Over time, more requests are analyzed, followed by the determination of a long-term average, which helps reduce false positives and false negatives. The trust modeling results are used to select a subset of traffic whose anomaly assessment exceeds a dynamically defined threshold value for its advancement to the next processing level.
Level 2. Event Classification and Object Modeling
At this level, the classification of the results obtained in previous stages occurs, attributing them to specific malicious events. The classification of events is performed according to the values assigned by machine learning classifiers to ensure a constant accuracy rate above 90%. These include:
- Linear models based on the Neyman-Pearson lemma (the normal distribution law from the chart at the beginning of the article)
- Support vector machines using multivariate learning
- Neural networks and the 'random forest' algorithm.
Such isolated security events are then linked over time to a single endpoint. It is at this stage that a threat description is formed, based on which a complete picture is created of how the respective attacker managed to achieve certain outcomes.
Event Classification. The statistically anomalous subset from the previous level is distributed across 100 or more categories using classifiers. Most classifiers are based on individual behavior, group relationships, or behavior on a global or local scale, while others can be quite specific. For example, a classifier might indicate traffic from command servers, suspicious extensions, or unauthorized software updates. As a result of this stage, a set of anomalous events in the security system is formed, categorized accordingly.
Object Modeling. If the amount of evidence supporting the hypothesis of the harmfulness of a certain object exceeds the threshold of significance, a threat is determined. Relevant events that influenced the threat definition are linked to this threat and become part of the discrete long-term model of the object. As evidence accumulates over time, the system determines new threats when the significance threshold is reached. This threshold is dynamic and undergoes intelligent adjustment based on the level of risk of the threat occurring and other factors. After this, the threat appears on the web interface dashboard and escalates to the next level.
Level 3. Relationship Modeling
The purpose of relationship modeling is to synthesize the results obtained at previous levels from a global perspective, taking into account not only the local but also the global context of the relevant incident. It is at this stage that you can determine how many organizations have faced such an attack, to understand if it was specifically targeted at you or if it is part of a global campaign, and you simply got caught in the crossfire.
Incidents are either confirmed or detected. A confirmed incident implies 99 to 100% confidence, as the relevant methods and tools have been previously observed in action on a larger (global) scale. Detected incidents are unique to you and are part of a narrow-focused campaign. Past results are provided with a known course of action, saving you time and resources in responding. They are supplied along with the investigation tools you will need to understand who attacked you and how targeted the campaign is against your digital business. As you can imagine, the number of confirmed incidents far exceeds the number of detected incidents, for the simple reason that confirmed incidents do not entail significant costs for the attackers, whereas detected incidents do.
expensive, as they must be new and customized. By creating the opportunity to identify confirmed incidents, the game's economy has finally shifted in favor of defenders, giving them a certain advantage.
Multilevel learning of the neural network system based on ETA
Global Risk Map
The Global Risk Map is created as a result of analyses applied by machine learning algorithms to one of the largest datasets in the industry. It provides extensive statistical data on server behavior online, even for unknown servers. These servers may be linked to attacks or can be engaged or used in an attack in the future. This is not a 'blacklist', but a comprehensive picture of the considered server from a security perspective. Such information about the activity of specified servers, contextualized, allows Stealthwatch's machine learning detectors and classifiers to accurately predict the risk level associated with communications with such servers.
Available maps can be viewed .
Now the network is learning and stands ready to protect your network.
Finally, is the panacea found?
Unfortunately, none. From my experience with the system, I can say that there are 2 global problems.
Problem 1. Price. The entire network is deployed on a Cisco system. This is both good and bad. The good part is that you don't have to worry about installing a lot of plug-ins like D-Link, MikroTik, etc. The downside is the enormous cost of the system. Given the current economic state of Russian businesses, currently only a wealthy owner of a large firm or bank can afford this marvel.
Problem 2. Training. I did not mention in the article the training period of the neural network, not because there isn’t one, but because it is constantly learning, and we cannot predict when it will learn. Of course, there are tools of mathematical statistics (take, for instance, the formulation of Pearson's convergence criterion), but these are half-measures. We obtain the probability of traffic filtering, and that only under the condition that the attack has already been mastered and is known.
Despite these two issues, we have made significant progress in the field of information security as a whole, and network protection in particular. This fact can serve as motivation to explore network technologies and neural networks, which are currently a very promising direction.
Source: habr.com
