Resource: https://www.threathunting.net/files/hunt-evil-practical-guide-threat-hunting.pdf
To begin, let’s clarify what threat hunting is:
Threat hunting is the human-driven, proactive and iterative search through networks, endpoints, or datasets in order to detect malicious, suspicious, or risky activities that have evaded detection by existing automated tools.
3 Myths about Threat Hunting:
1. Hunting can be fully automated
Hunting is not a reactive activity. If the main human input in a hunt is remediating the result of
something that a tool automatically found, you are being reactive and not proactive. You are
resolving an identified potential incident, which is a critically important practice in a SOC, but not
hunting.
Hunting requires the input of a human analyst and is about proactive, hypothesis-based
investigations. The purpose of hunting is specifically to find what is missed by your automated
reactive alerting systems. An alert from an automated tool can certainly give you a starting point
for an investigation or inform a hypothesis, but an analyst should work through an investigation
to understand and expand on the context of what was found to really get the full value of hunting.
2. Hunting can only be carried out with vast quantities of data and
a stack of advanced tools
Though it may seem like a new term, security analysts across a variety of sectors have been
hunting for years. Basic hunting techniques can still be very useful and effective in helping you find
the bad guys (e.g. you can perform basic outlier analysis, or “stack counting”, in Microsoft Excel).
An analyst who wants to begin threat hunting should not hesitate to dive into some of the basic
techniques with just simple data sets and tools. Take advantage of low hanging fruit!
Of course, having purpose-built tools like a Threat Hunting Platform can help you hunt at scale and
simplify the more advanced hunt procedures.
3.
Hunting is only for elite analysts; only the security 1% with
years of experience can do it
As you’ll learn, there are many different hunting techniques that have differing levels of
complexity. However, not all these techniques take years to master. Many of the same analysis
techniques used for incident response and alert investigation and triage can also be leveraged
for hunting. The key to getting started is simply knowing what questions to ask, and digging into
the datasets related to them. You learn to hunt by doing it, so if you’re an analyst who has never
hunted before, don’t be afraid to dive in.
Determining Your Hunting Maturity
The Hunting Maturity Model describes five levels of an organization’s proactive detection capability.
Each level of maturity corresponds to how effectively an organization can hunt based on the data they
collect, their ability to follow and create data analysis procedures (DAP), and their level of hunting
automation. The HMM can be used by analysts and managers to measure current maturity and provide
a roadmap for improvement. Often these improvements focus on a combination of tools, processes,
and personnel.
If you want to determine your current level of hunting maturity, below is a list of questions you can
answer to find out. You can then take your maturity level and align it to our suggestions about where
you should be focusing your efforts next.
Basic Requirements:
1. Do you have automated security alerting
(SIEM, IDS, etc)?
2. Do you already have a dedicated incident
detection or response team(s)?
– if you have answered NO, stop and see the sections below.
Minimal Capability:
1. Do you routinely collect security data
from all three data domains (network,
host, & application logs) into a centralized
repository?
2. Do you utilize threat intelligence to drive
detection (open or closed source)?
3. Do analysts in your SOC leverage Indicators
of Compromise (IoCs) from reports?
-if you have answered no, check HM0 for more information
Procedural Approach (most common):
1. Do analysts in your SOC follow published
hunting procedures to find new security
incidents?
2. Do analysts in your SOC hunt on a regular
recurring schedule: daily, weekly, etc?
3. Do you have designated hunters in your
SOC or a set rotation of analysts who hunt
so that there is always some proactive
detection effort being carried out?
-if you have answered NO, check HM1 for more information
Innovative Process:
1. Are your hunters utilizing a variety of data
analysis techniques and applying them to
identify malicious activity?
2. Do your hunters develop or publish original
hunting procedures adapted from hunts
they carry out in your environment?
3. Are you collecting security data tailored
to your environment and your hunting
practices?
4. Do you utilize a specialized threat hunting
platform (Crowdstrike, Exabeam, etc) to facilitate streamlined hunting
processes and collaboration in your hunt
team?
Leading Programs
- Are you automating successful hunting
procedures/using the outputs of your hunts
to improve alerting or automated detection
efforts?
- Do you employ data science techniques to
support your hunting procedures and help
isolate anomalies in large quantities of
data? - Do you have a methodology for scaling your
ability to carry out the hunting procedures
you are continually creating?
Four Primary Threat Hunting Techniques:
1. Searching – Simplest method of hunting, searching is the process of querying data for specific
results or artifacts, and can be performed using many tools. Searching requires finely defined
search criteria to prevent result overload. There are two primary factors to keep in mind
when carrying out a search: searching too broadly for general artifacts may produce far too
many results to be useful, and searching too specifically for artifacts on specific hosts may
produce fewer results than may be useful.
2.Clustering is a statistical technique, often carried out with machine learning, that consists
of separating groups (or clusters) of similar data points based on certain characteristics
out of a larger set of data. Hunters may use clustering for many applications, including
outlier detection, due to the fact that it can accurately find aggregate behaviors, such as an
uncommon number of instances of a certain occurrence. This technique is most effective
when dealing with a large group of data points that do not explicitly share immediately
obvious behavioral characteristics.
3.Grouping consists of taking a set of multiple unique artifacts and identifying when multiple
of them appear together based on specific criteria. The major difference between grouping
and clustering is that in grouping your input is an explicit set of items that are already of
interest. Discovered groups within these items of interest may potentially represent a tool
or a TTP that an attacker might be using. An important aspect of using this technique
consists of determining the specific criteria used to group the items, such as events having
occurred during a specific time window. This technique works best when you are hunting for
multiple, related instances of unique artifacts, such as the case of isolating reconnaissance
commands that were executed within a specific timeframe.
4. Stack Counting, also known as stacking, this is one of the most common techniques carried out by hunters to investigate a hypothesis. Stacking involves counting the number of occurrences for values of a particular type, and analyzing the outliers or extremes of those results. The effectiveness of this technique is generally diminished when dealing with large and/or diverse data sets, but it is most effective with a thoughtfully filtered input (such as endpoints of a similar function, organizational unit, etc.). Analysts should attempt to understand input well enough to predict the volume of the output. For example, if you are given a dataset
containing 100k endpoints, stack counting the contents of the Windows\Temp\ folder on
each endpoint across an enterprise will produce an enormous result set. Friendly intelligence
can be used to define filters for your input.
Datasets:
The techniques that you use are only a part of planning out your hunt and knowing what you can have at your disposal. You can’t hunt if you don’t have the right data, but what is the right data? The answer to that question will depend on what you’re looking for, but below is a general list of datasets that lend themselves well to hunting and security activities in general:
1. End points
Process execution metadata (use Linux “auditd” for complete process monitoring, alerts on unusual processes, Linux CSF, SIEM agents to track annomalies)
Registry access data (Windows)
File data (Use file system integrity monitors, and host based IDS – HIDS, Linux CSF)
Network data (track open connections, open ports, open sockets, etc)
System resources (memory, CPU, I/O usage for anomalies, via SNMP, or SIEM agents, collectd)
2. Network
Network session data (Netflow)
Zeek (previously Bro Logs) IDS
Traffic Mirroring (AWS): Zeek, Surricata – https://docs.aws.amazon.com/vpc/latest/mirroring/tm-example-open-source.html ; https://www.snaplabs.io/insights/vpc-traffic-mirroring-amp-zeek-on-aws
Proxy Logs, Firewall Logs, DNS Logs, Switch/Router Logs
3. Security Data
Threat Intelligence
Alerts and Out of Bound Rules (Datadog Cloud SIEM)
Friendly Intelligence (everything that you can learn about the company, network, people, software used, etc)
Resources:
https://threathunterplaybook.com/intro.html
https://github.com/OTRF/ThreatHunter-Playbook/tree/master/resources
https://www.mitre.org/sites/default/files/2021-11/16-3713-finding-cyber-threats-with-attack-based-analytics.pdf
