95% of cybersecurity threats are known and can be defended against using traditional means like antivirus software, firewalls, IDS, and WAF. The remaining 5% of threats are unknown and the most dangerous. They account for 70% of the risk to a company because they are very difficult to detect and even harder to defend against. Examples include such as the WannaCry and NotPetya/ExPetr ransomware outbreaks, cryptominers, and the Stuxnet "cyber weapon" (which affected Iran's nuclear facilities), along with many others (does anyone still remember Kido/Conficker?) that are challenging to defend against using classic protection methods. We want to discuss how to combat these 5% of threats using Threat Hunting technology.
The continuous evolution of cyberattacks necessitates constant detection and response, ultimately leading us to the idea of an endless arms race between attackers and defenders. Classic defense systems can no longer provide an acceptable level of security where risk doesn't impact key business metrics (economic, political, reputation) without customization for specific infrastructures, but they do cover some risks. Even during the deployment and configuration process, modern protection systems find themselves in a catch-up role and must respond to the challenges of the new era.
One response to the challenges of modernity for information security specialists may be Threat Hunting technology. The term Threat Hunting (hereafter referred to as TH) emerged a few years ago. The technology itself is quite interesting but still lacks any accepted standards and regulations. The diversity of information sources and the scarcity of Russian-language resources on this topic complicates matters. Therefore, we at 'LANIT-Integration' decided to write an overview of this technology.
Relevance
The TH technology relies on infrastructure monitoring processes.Alerting (similar to MSSP services) is a traditional method that involves searching for previously developed signatures and indicators of attacks and responding to them. This scenario is successfully handled by traditional signature-based protection tools. Hunting (an MDR-type service) is a monitoring method that addresses the question, 'Where do signatures and rules come from?'. It is a process of creating correlation rules by analyzing hidden or previously unknown indicators and signs of an attack. Threat Hunting refers to this type of monitoring.
Only by combining both types of monitoring can we achieve protection that is close to ideal, yet there always remains a certain level of residual risk.
Protection using two types of monitoring
Here’s why TH (and hunting in general!) will become increasingly relevant:
Threats, protection measures, risks.
These include types such as spam, DDoS, viruses, rootkits, and other classic malware. Protection against these threats can be implemented with the same classical protection measures.
During any project execution,while the remaining 20% of the work takes 80% of the time. Similarly, among the entire threat landscape, 5% of new-type threats will account for 70% of the company's risk. In a company where information security management processes are organized, we can manage 30% of the risk from known threats in one way or another, either by avoiding (eliminating wireless networks altogether), accepting (implementing necessary protection measures), or transferring (for example, to the integrator) that risk. However, protecting againstAPT attacks, phishing,cyber espionage, and national operations, as well as a large number of other attacks, is much more challenging. The consequences of these 5% of threats will be far more severe (), than the consequences of spam or viruses, against which antivirus software protects.
Everyone is likely to encounter 5% of threats. Recently, we had to install an open-source solution that uses an application from the PEAR (PHP Extension and Application Repository) repository. The attempt to install this application via pear install failed, as was unavailable (now there's already a placeholder on it), I had to install it from GitHub. And just recently, it was revealed that PEAR has fallen victim to.
We can also recall, the NePetya ransomware epidemic via the accounting software update module. Threats are becoming increasingly sophisticated, and a logical question arises - "How do we resist these 5% of threats?"
Definition of Threat Hunting
So, Threat Hunting is the proactive and iterative process of seeking and discovering advanced threats that cannot be detected by traditional protective means. Advanced threats include, for example, attacks such as APT, zero-day vulnerability attacks, Living off the Land, and so on.
It can also be rephrased that TH is a process of hypothesis verification. This is predominantly a manual process with elements of automation, in which an analyst, relying on their knowledge and qualifications, sifts through large volumes of information in search of signs of compromise corresponding to an initially defined hypothesis about the presence of a specific threat. A distinctive feature of it is the variety of information sources.
It should be noted that Threat Hunting is not a software or hardware product. It is not alerts that can be seen in some solution. It is not a process of searching for IOCs (indicators of compromise). And it is not some passive activity that goes on without the participation of cybersecurity analysts. Threat Hunting is primarily a process.
Components of Threat Hunting
The three main components of Threat Hunting: data, technology, people.
Data (what?), including Big Data. All kinds of traffic streams, information about previously conducted APTs, analytics, data on user activity, network data, information from employees, information from the dark web, and much more.
Technology (how?) processing this data - all possible ways to process this data, including Machine Learning.
People (who?) Those who have extensive experience in analyzing various attacks, developed intuition, and the ability to detect an attack. Typically, these are information security analysts who must be able to generate hypotheses and find confirmation for them. They are the key link in the process.
The PARIS Model
Adam Bateman The PARIS model for the ideal TH process. The name hints at a famous landmark in France. This model can be viewed from two perspectives – top down and bottom up.
In the threat hunting process, moving from the bottom up model, we will deal with numerous pieces of evidence of malicious activity. Each piece of evidence has a measure called confidence – a characteristic that reflects the weight of this evidence. There are 'concrete', direct proofs of malicious activity, allowing us to quickly reach the top of the pyramid and create an actual alert regarding a confirmed infection. Then there are indirect evidences, the sum of which may also lead us to the top of the pyramid. As always, there is much more indirect evidence than direct, meaning they need sorting and analysis, further research, and ideally this should be automated.
The PARIS Model.
The top part of the model (1 and 2) is based on automation technologies and various analytics, while the bottom part (3 and 4) relies on qualified people who manage the process. The model can be viewed from the top down, where in the upper blue section we have alerts from traditional protective measures (antivirus, EDR, firewall, signatures) that carry a high degree of confidence and trust, while below are indicators (IOC, URL, MD5, and others) that have a lower degree of confidence and require further investigation. The very bottom and thickest level (4) involves generating hypotheses, creating new scenarios for traditional protective measures. This level is not limited to the specified sources of hypotheses. The lower the level, the more qualifications are required from the analyst.
It is very important for analysts not only to check a final set of predetermined hypotheses, but to constantly work on generating new hypotheses and options for their verification.
Maturity model of TH usage
In an ideal world, TH is a continuous process. However, since there is no perfect world, let's analyze and methods in the context of people, processes, and technologies used. Let's consider the model of the ideal spherical TH. There are 5 levels of utilizing this technology. We will examine them using the evolution of a specific team of analysts as an example.
Maturity Levels
People
Processes
Technologies
Level 0
SOC Analysts
24/7
Traditional tools:
Traditional
Alert set
Passive monitoring
IDS, AV, Sandboxing,
Without TH
Working with alerts
signature analysis tools, Threat Intelligence data.
Level 1
SOC Analysts
One-time TH
EDR
Experimental
Basic knowledge of forensics
IOC Search
Partial coverage of data from network devices
Experiments with TH
Good understanding of networks and application part
Partial application
Level 2
Temporary involvement
Sprints
EDR
Periodic
Intermediate knowledge of forensics
A week per month
Full application
Temporary TH
Excellent understanding of networks and application part
Regular TH
Full automation of EDR data usage
Partial use of advanced EDR capabilities
Level 3
Dedicated TH team
24/7
Partial ability to validate TH hypotheses
Preventive
Excellent knowledge of forensics and malware
Preventive TH
Full use of advanced EDR capabilities
Specific TH cases
Excellent knowledge of the attacking side
Specific TH cases
Complete data coverage from network devices
Configuration tailored to needs
Level 4
Dedicated TH team
24/7
Full ability to validate TH hypotheses
Leading
Excellent knowledge of forensics and malware
Preventive TH
Level 3, plus:
Using TH
Excellent knowledge of the attacking side
Verification, automation, and validation of TH hypotheses
close integration of data sources;
Research capability
development tailored to needs and custom use of API.
TH maturity levels in the context of people, processes, and technologies
Level 0: traditional, without the use of TH. Regular analysts work with a standard set of alerts in passive monitoring mode using standard tools and technologies: IDS, AV, sandboxes, signature analysis tools.
Level 1: Experimental, using TH. Analysts with basic knowledge of forensics and a good understanding of networks and applied aspects can conduct one-time Threat Hunting by searching for indicators of compromise. The tools include EDR with partial data coverage from network devices. Tools are used partially.
Level 2: Periodic, temporary TH. The same analysts who have already enhanced their knowledge in forensics, networks, and applied issues are tasked with regular engagement (sprint) in Threat Hunting, say, one week per month. The tools include a complete investigation of data from network devices, automation of data analysis from EDR, and partial use of advanced EDR capabilities.
Level 3: Preventive, frequent TH cases. Our analysts have organized into a dedicated team, acquiring excellent knowledge of forensics and malware, along with understanding the methods and tactics of the attacking side. The process is already being conducted in a 24/7 mode. The team can partially verify TH hypotheses while fully utilizing the advanced capabilities of EDR with complete data coverage from network devices. Analysts are also capable of configuring tools to suit their needs.
Level 4: Leading, utilizing TH. The same team has gained the ability to conduct research, generate and automate the process of verifying TH hypotheses. Now, the tools have additional tight integration of data sources, software development tailored to needs, and unconventional use of APIs.
Threat Hunting Techniques
Basic Threat Hunting Techniques
To TH in order of the maturity of the technology used includes: basic search, statistical analysis, visualization techniques, simple aggregations, machine learning, and Bayesian methods.
The simplest method is basic search, used to narrow down the research area through specific queries. Statistical analysis is applied, for example, to construct a typical user or network activity in the form of a statistical model. Visualization techniques are used for clear representation and simplification of data analysis through graphs and charts, where patterns in the sample are much easier to discern. The technique of simple aggregations on key fields is used to optimize search and analysis. The higher the maturity level achieved in the organization’s Threat Hunting process, the more relevant the use of machine learning algorithms becomes. They are also widely used for spam filtering, malicious traffic detection, and fraud detection. A more advanced type of machine learning algorithms is Bayesian methods, which allow for classification, dimensionality reduction of samples, and topic modeling.
Diamond Model and Threat Hunting Strategies
Sergio Caltagirone, Andrew Pendergast, and Christopher Betz in their work “” outlined the key components of any malicious activity and the basic relationships between them.
Diamond Model for Malicious Activity
According to this model, there are 4 Threat Hunting strategies that are based on the corresponding key components.
1. Victim-focused strategy. We assume the victim has adversaries, and they will deliver "opportunities" via email. We look for enemy data in the email. Searching for links, attachments, etc. We seek confirmation of this hypothesis for a certain period (a month, two weeks), and if nothing is found, the hypothesis is invalid.
2. Infrastructure-focused strategy. There are several methods for employing this strategy. Depending on access and visibility, some are easier than others. For example, we monitor domain name servers known for hosting malicious domains. Or we conduct a process of tracking all new domain registrations for known patterns used by the adversary.
3. Capability-based strategy. In addition to the victim-focused strategy used by most network defenders, there is a capability-based strategy. This is the second most popular and focuses on detecting opportunities presented by the adversary, specifically 'malware' and the potential use of legitimate tools by the adversary, such as psexec, powershell, certutil, and others.
4. Adversary-focused strategy. The adversary-focused approach concentrates on the adversary themselves. This includes the use of publicly available information from open sources (OSINT), gathering data about the adversary, their tactics, techniques, and procedures (TTP), analyzing past incidents, threat intelligence data, and so on.
Information sources and hypotheses in Threat Hunting
Some information sources for Threat Hunting
There can be many information sources. An ideal analyst should be able to extract information from everything around them. Typical sources in almost any infrastructure will include data from security tools: DLP, SIEM, IDS/IPS, WAF/FW, EDR. Additionally, various indicators of compromise, Threat Intelligence services, CERT data, and OSINT will also be standard information sources. Information from the darknet can also be leveraged (for instance, there may suddenly be an order for hacking the CEO's email, or a candidate for a network engineer position may have drawn attention through their activities), information received from HR (feedback about candidates from their previous job), and information from security services (such as the results of contractor checks).
However, before utilizing all available sources, at least one hypothesis must be established.
To test hypotheses, they must first be formulated. And to come up with many quality hypotheses, a systematic approach is necessary. The process of generating hypotheses is described in more detail in, this scheme is very convenient to use as the basis for the hypothesis formulation process.
The primary source of hypotheses will be the ATT&CK matrix (Adversarial Tactics, Techniques and Common Knowledge). It essentially serves as a knowledge base and a model for assessing the behavior of adversaries carrying out their activities at the latter stages of an attack, usually described using the concept of the Kill Chain. This means during the phases after an attacker has penetrated the internal network of an organization or a mobile device. Initially, the knowledge base included descriptions of 121 tactics and techniques used in attacks, each detailed in a Wiki format. Diverse Threat Intelligence analytics are well-suited as sources for generating hypotheses. Notably, the results of infrastructure analysis and penetration tests stand out – these are the most valuable data that can provide us with reliable hypotheses due to their foundation on specific infrastructure with its distinct vulnerabilities.
Hypothesis Verification Process
Sergey Soldatov presented with a detailed description of the process; it illustrates the hypothesis verification process in a particular system. I will outline the main stages with brief descriptions.
Stage 1: TI Farm
At this stage, it is necessary to identify objects (by analyzing them along with all threat data) and assign them labels indicating their characteristics. This can be a file, URL, MD5, process, utility, or event. By passing them through Threat Intelligence systems, labels need to be applied. For instance, this site was noted in CNC in a certain year, this MD5 was linked to a specific malware, this MD5 was downloaded from a site that distributed malware.
Stage 2: Cases
In the second stage, we examine the interactions between these objects and identify relationships among all of them. We obtain marked systems that are performing suspicious activities.
Stage 3: Analyst
In the third stage, the case is handed over to an experienced analyst who possesses extensive analysis expertise, and he delivers the verdict. He breaks down the code byte by byte to determine what, where, how, why, and the purpose behind it. This body was malicious, this computer was infected. He uncovers links between objects and verifies the results through the sandbox.
The analyst's findings are passed on. Digital Forensics investigates images, Malware Analysis examines discovered artifacts, and the Incident Response team may deploy to the location to investigate something on-site. The outcome will be a confirmed hypothesis, identified attack, and counteraction strategies.
Summary
Threat Hunting is a relatively new technology capable of effectively countering customized, novel, and unconventional threats, which has significant potential given the increasing number of such threats and the complexity of corporate infrastructure. It requires three components: data, tools, and analysts. The benefits of Threat Hunting extend beyond preempting threats. It's important to remember that during the search process, we delve into our infrastructure and its weak points through the eyes of a security analyst and can further strengthen those areas.
The first steps that, in our opinion, should be taken to initiate the Threat Hunting process within your organization.
- Ensure the protection of endpoints and network infrastructure. Ensure visibility (NetFlow) and control (firewall, IDS, IPS, DLP) of all processes in your network. Know your network from the edge router to the very last host.
- Study.
- Conduct regular penetration tests on at least the key external resources, analyze the results, identify the main targets for attacks, and close their vulnerabilities.
- Implement an open-source Threat Intelligence system (e.g., MISP, Yeti) and conduct log analysis alongside it.
- Implement an Incident Response Platform (IRP): R-Vision IRP, The Hive, a sandbox for analyzing suspicious files (FortiSandbox, Cuckoo).
- Automate routine processes. Log analysis, incident creation, notifying staff — these are vast areas for automation.
- Learn to interact effectively with engineers, developers, and technical support for collaborative work on incidents.
- Document the entire process, key points, and results achieved, so you can refer back to them later or share this information with colleagues;
- Remember the social aspect: stay informed about what is happening with your employees, who you hire, and who you grant access to the organization's information resources.
- Stay informed about trends in new threats and ways to protect against them, enhance your level of technical literacy (including in the operation of IT services and subsystems), attend conferences, and communicate with colleagues.
I am ready to discuss the organization of the TH process in the comments.
Or come work with us!
Sources and Materials for Study
Source: habr.com
