Opening

Hopefully you all were able to read our recent Threat Hunting whitepaper and had the chance to listen to our latest Threat Hunting webinar. These references should be used as the foundation of information, which leads us into the next journey: how to build out your first Threat Hunt.

Building out an organization's Threat Hunting program is a huge undertaking. So, what I would like to continue to do is take small chunks from the concepts already covered and dig a little deeper into the building process by providing a proven approach that any organization, no matter the size, can follow to get their hunting objectives off the ground. I like to refer to this process as taking a Crawl, Walk, Run approach.

The overall goal with this process is to assist in the next steps of the planning activities involved with the individual Threat Hunts (Crawl stage) to be conducted and potentially establish a normal cadence or sprint cycle from these planning efforts (leading into the Walk and Run stages).

Individual Threat Hunt Planning Process Overview

From a high-level, these are the steps that I will cover and explain why each step should take place and what information each step should contain.

1. Understand the business vertical that the organization is part of and associated threats
2. Identify geographic presence the organization has and associated threats
3. Identify threat groups targeting the organization directly, business vertical, or geographic region
4. Identify Threat Hunting framework to model
5. Identify a specific tactic of interest
6. Identify a specific technique of interest
7. Identify attack testing validation framework to use
8. Run attack simulation tests
9. Identify attack simulation visibility and detection gaps
10. Work on security operation tuning then retest
11. Implement detection query, set up alerting, and establish response criteria
12. Pulling it all together

A very important concept in Threat Hunting is choosing threats that are specific to the organization. This will prevent wasting time by haphazardly hunting for meaningless exploits or ineffectively using Threat Hunting frameworks. These guidelines will automatically produce focused Threat Hunting activities that steer clear from the general 'Find Bad' approach that many organizations use. By following the steps outlined above, an organization will be able to implement an effective operational Threat Hunting approach and begin to lay out a cohesive, repeatable Threat Hunting process.

With that being said, let's dive in and see how this all unfolds.

External Landscape Identification

These first two (2) steps are often overlooked by many organizations but are critical in significantly increasing the threat awareness outside of the organization. Once the information is gathered, an organization can considerably enhance their Threat Hunting focus, which will assist in developing a tactical approach later on.

1.       Business Vertical Threat Identification

Business vertical is a term used to describe a specific industry or market that focuses on a particular niche. Understanding the business vertical from a Threat Hunting lens will allow an organization to tighten the focus around specific threats or threat groups targeting the business sector/vertical.

Decision Example:

Mycoolorg resides in the technology business sector.

2.       Geographic Threat Identification

Many organizations are geographically dispersed but may not fully understand what this means from a threat perspective. If a U.S.-based organization has a geographic presence within a European state, there are additional security concerns to be aware of (various threat groups target organizations based on geographic region). This additional information will tighten the Threat Hunting focus around specific threat groups, tactics, or techniques.

Decision Example:

Mycoolorg is headquartered in New York but has a large international office located in the Netherlands.

Tactical Threat Focus

Now that we have identified threats relevant to the organization, we can easily pivot into the Threat Hunting planning steps. We have an organization that is U.S.-based with a presence in the European Union and sits within the technology business sector.

There are multiple ways to approach this Threat Hunting planning: an organization can choose to focus on a particular threat group or groups; a single or set of tactics; or a single or set of techniques. That said, it is highly recommended to take a Crawl approach and keep the initial Threat Hunting objectives manageable and concise. Taking the current security staff capabilities and operational bandwidth into account prior to conducting the Threat Hunting activities can help avoid scope creep or analyst burnout in these early stages.

Let's see how we can extract some tactical Threat Hunting objectives that will be of direct interest to the organization.

3.       Threat Groups of Interest

A great way to pinpoint the threat groups of interest is by navigating to the MITRE ATT&CK threat groups page. From this page the following information will be presented:

• General overview of the threat group
• Associated group descriptions
• Techniques used
• Software used

Decision Example:

By performing two (2) quick searches using the strings technology and Europe, APT29 stands out as a threat group of interest. This group has been linked to the Russian Foreign Intelligence Service (SVR) and has targeted organizations in the technology sector located in the United States and European regions.

4.       Threat Hunting Framework

An important aspect to consider during planning is to align Threat Hunting efforts to a framework, as one may be preferred over another based on the organizational objectives.

Three (3) frameworks come to mind, each depicting the attacker lifecycle in slightly different ways. All are valid, valuable frameworks with their own respective pros and cons. The choice ultimately comes down to personal or organizational preference:

• Lockheed Martin Cyber Kill Chain
• Diamond Model of Intrusion Analysis
• MITRE ATT&CK Model

It is important to understand that these frameworks are guidelines for building out planned Threat Hunting activities, but additional research should be done to identify attacker tactics and techniques that may fall outside of these framework examples.

Decision Example:

Mycoolorg decides to follow the MITRE ATT&CK framework for their Threat Hunting planning.

5.       Attack Tactic of Interest

Based on the Threat Hunting framework identified above, the organization will need to identify what attack tactic to focus on initially. This approach significantly narrows the focus of the initial Threat Hunting activities to avoid scope creep. The MITRE ATT&CK framework has 14 different Enterprise Tactics to choose from. One (1) way to choose a tactic is to identify which one poses the most significant threat to the organization.

Decision Example:

Mycoolorg decides to focus on TA0008 - Lateral Movement due to the known monitoring blind spots associated with this type of activity within the organization.

6.       Attack Technique of Interest

Based on the Threat Hunting framework and attack tactic choices above, it is time to choose a critical attack technique to focus on. A way to tie all of the pieces of research together is to use the MITRE ATT&CK Navigator to identify a relevant technique that is specific to the APT29 threat group.

Techniques highlighted in blue are specific to the APT29 threat group.

Decision Example:

Mycoolorg decides to focus on sub-Technique T1021.006 - Windows Remote Management because the protocol is used for legitimate day-to-day system administrator purposes. The main concern to address is that the organization has no baseline understanding of the known-good use of the protocol, which makes it easy for an attacker to blend in with authorized WinRM network traffic.

By navigating to the chosen technique, the organization will be presented with additional important information, such as:

• How the attack technique is used by APT29
• Threat mitigations
• Threat detections

Attack Testing and Logging Visibility

Let's move on to the fun part and see what it looks like to put together the attack testing and verify the necessary logging visibility for the chosen attack technique to understand what telemetry is required to properly hunt for this threat.

Attack simulation exercises combined with an organization's Threat Hunting activities can help build Threat Hunting queries that can be polished and turned into production threat detections.

7.       Attack Simulation Framework

Many organizations do not have an internal Penetration Testing or Red Team to assist with attack simulation. While these teams do provide value, Threat Hunting and detection testing can be conducted without them due to the availability of numerous open-source attack simulation frameworks.

Two (2) robust attack simulation frameworks to choose from are:

• Atomic Red Team (ART)
• MITRE Caldera

Of course, organizations can always work with a third-party (TrustedSec) to conduct a Threat Hunting or Threat Hunting Program Building engagement, but the focus here is to show an organization how they can perform this testing on their own with a significant level of success.

Decision Example:

Mycoolorg decides to go with Red Canary's ART attack simulation framework.

8.       Attack Simulation Testing

The ART framework has Atomic tests that can be used against multi-platform operating systems, which makes this framework highly versatile for blended system environments.

• Windows
• Linux
• MacOS

The process to get up and running with ART is very simple:

1. Install the ART execution framework and Atomics folder.

IEX (IWR 'https://raw.githubusercontent.com/redcanaryco/invoke-atomicredteam/master/install-atomicredteam.ps1' -UseBasicParsing);

Install-AtomicRedTeam -getAtomics -Force

2. Import the AtomicRedTeam module.

Import-Module .\Invoke-AtomicRedTeam.psd1

Before running the applicable attack script, T1021.006 - Windows Remote Management, it is necessary to understand what attack steps it is performing and what it is not necessarily testing. So, it is critical to analyze the testing script to validate if the testing criteria meets the organization's Threat Hunting requirements. If not, the criteria can be supplemented with manual testing.

This verification can be performed in two (2) different ways:

• From the main ART GitHub repository
• From the PowerShell command line using the Invoke-AtomicTest -ShowDetails parameter

3. Run the applicable Atomic test.

Invoke-AtomicTest T1021.006

4. Run the manual test.

invoke-command -ComputerName {DC01} -scriptblock {whoami}

To increase this testing (Walk and Run stages), organizations could incorporate a robust open-source attacker framework, such as:

• C2 Matrix
• Metasploit

Decision Example:

Mycoolorg creates an isolated lab environment to perform all testing in order to prevent any inadvertent negative impact to the production network.. The environment will consist of attacker host machines running Kali Linux and Windows, Windows 2019 domain controller (similar configuration to production environment), Windows 10 user system (based on user golden image), and SIEM (ELK stack, Microsoft Azure Sentinel, community version of Splunk, etc.).

9.       Attack Simulation Logging

Although specific Threat Hunting queries will not be shared for this use case, I will share what you can look for regarding this attack technique. I wanted to keep the focus of this blog post on the Threat Hunting process itself.

Here's a very important concept that must be understood: you cannot Threat Hunt what you cannot see! So, this portion of testing is critical to address before attempting to perform any live-fire testing and Threat Hunting within the production environment.

From the ART test script executed above, it is now time to identify what Windows event log data was created. For some, this may be the sticking point on how to identify what should be logged. A good way to identify what you should be looking for can be found using some simple Google Fu.

Two (2) great resources you will stumble across will be from @Cyb3rWard0g (Roberto Rodriguez), PowerShell Remote Session and JPCERT, WinRM Tool Analysis. There are many others, but these are solid references to build from.

Based on the basic testing of the ART T1021.006 - Windows Remote Management attack script, the following logs were generated.

Based on the manual testing of the T1021.006 - Windows Remote Management attack using the Invoke-Command, the following additional log output was generated.

As we can see from the logging output, the following Event IDs were generated:

• Sysmon Event ID 3
• Windows Event ID 6 and 4103

But, as stated above, an organization must fully understand what the attack technique consists of and what specific portions of the attack you want to focus on for the planned Threat Hunting activities. These distinctions are important because while the activity as a whole is malicious, specific minutiae or fields may contain legitimate identifiers. Alerting on legitimate data may inadvertently cause a high false positive rate for any associated Threat Hunting and detection logic.

With this being said, organizations should also consider the following baseline Windows and Sysmon Event IDs to provide greater Threat Hunting visibility and detection capabilities surrounding WinRM activity:

• Sysmon Event ID 1
• Windows Event IDs 400, 4104, 4688, and 5156

From a centralized logging perspective, to provide visibility into this attack technique, the identified Event IDs will need to be ingested into the organization's SIEM to provide the necessary Threat Hunting visibility and detection capability. To increase Threat Hunting visibility and layered detection abilities (Walk and Run stages), organizations should have a solid understanding of all deployed operational security solutions and their detection and logging capabilities.

Although there are many other attack variations for this technique that should be considered, an organization will have full capability in constructing preliminary Threat Hunting queries from the testing accomplished. As the Threat Hunting team gains a greater understanding of the various layers of the attack, deeper levels of Threat Hunting breadth and depth will be achieved (Walk and Run stages).

To increase this testing (Walk and Run stages), organizations could incorporate external datasets that provide captured logging output based on specific attack tactics and techniques.

• @sbousseaden EVTX-ATTACK-SAMPLES
• Splunk Attack Data

Additional Resources:

• Active Directory Lateral Movement Detection
• TrustedSec Sysmon Community Guide
• TrustedSec Attack Path Mapping Approach
• SpecterOps Capability Abstraction

Decision Example:

Mycoolorg immediately identifies inadequacies in the current PowerShell logging visibility and identifies that Sysmon is not installed within the environment to provide this additional telemetry.

Operationalizing Threat Hunting Testing Output

Alright, now how do we tie all this goodness together? The steps above outline a methodical approach to Threat Hunting planning that organizations of all sizes can implement. This approach has a far greater return on investment than haphazardly taking a 'Find Bad' approach or throwing a dart at a Threat Hunting framework matrix. But it is also very apparent that a significant amount of time and effort are required to accomplish thorough Threat Hunting planning, testing, and validation. Organizations must understand their security staff's capabilities and operational workloads to decide whether they can realistically maintain operational Threat Hunting activities.

Within the last two (2) phases of this approach, organizations may run into issues as they pertain to security operational tuning requirements, detection implementation, and alerting decisions.

10.       Security Operation Tuning and Retesting

As we saw in the Attack Simulation Logging section, although a significant amount of Threat Hunting activities were achieved, Mycoolorg identified that the lab environment did not properly mirror the production environment, PowerShell logging visibility was inadequate, and Sysmon was not installed on any production deployed system within the environment. Let's explore some viable options to remediate the current logging deficiencies to enable the ability to properly detect this threat.

Another important aspect of the planning considerations is to include key stakeholders early in the planning process to lessen any unforeseen operational sticking points.

Making changes to a production environment is no easy feat, but this is where the due diligence within the steps above will greatly assist in the security operational tuning requests. The following items should be considered to provide intermediate levels of Threat Hunting and detection success for this attacker technique.

• Enable PowerShell script block logging only on servers or critical assets
• Install Sysmon only on servers or critical assets
• Identify responsible teams to coordinate changes
• Perform WinRM baselining analysis and establish known-good activity
• Identify Change Management considerations

Once the necessary preliminary production changes have been made, the following steps should be taken to properly perform live-fire Threat Hunting tests within the production environment to confirm the necessary logging visibility and telemetry are available to perform future Threat Hunting activities.

• Identify what systems the testing will target
• Modify the ART attack script accordingly
• Execute the ART attack script
• Verify that all the necessary logs are visible within the SIEM
• Confirm that Threat Hunting queries identify the ART script activity

Once the ART script Threat Hunting validation has been successfully accomplished, the final step would be to perform the Threat Hunt against the production environment critical assets identified above searching for any suspicious WinRM activity.

Decision Example:

Mycoolorg agreed to only enable PowerShell script block logging and install Sysmon on critical assets due to logging overhead across the network and SIEM back-end storage constraints. Production environment WinRM ART attack script testing yielded the necessary logging visibility to enable adequate Threat Hunting and detection capabilities for this attack technique variation while targeting critical assets.

11.       Detection Query, Alerting, and Automation Implementation

OK, deep breath—we did it. We are now at the final stage of the Threat Hunting activities, but this may also be a point of contention for some Threat Hunting teams. The Threat Hunting teams that are not part of the security operations side of the organization may find it more difficult to transition the rough Threat Hunting query into a viable production-grade detection query. SIEM and SOC teams may be very reluctant to operationalize these Threat Hunt queries if they were kept in the dark regarding the overall objectives of the Threat Hunting exercise. Another critical aspect to consider is the alert criteria associated with the newly implemented detection query. Other areas to take into consideration and plan for are: What criticality should the alert receive? Are there any SLAs that should be applied? Can any automation be built into the response aspects of the alert?

As you can see, this final step is no less important than any of the steps listed above and may be the most critical step.

Here are important aspects to consider early in the Threat Hunting planning efforts that will ensure greater success during the final stages of the operational implementation efforts:

• Communicate Threat Hunting objectives to the security operational teams and management
• Recommend associated alert criteria for triggered alarms from implemented detection query
• Perform knowledge transfer of the Threat Hunting testing and validation documentation to security operational staff
• Assist in identifying any security operation response automation capabilities 

In the end, Threat Hunting activities will need to be embedded into the SOC and Incident Response workflows to create a seamlessly integrated security operation capability.

Decision Example:

Mycoolorg successfully transitioned the Threat Hunting query into a production SIEM detection query with an alert criterion of Medium (for the time being). Further baselining activities are required to establish known-good WinRM system administrator use across the network. Security operation analysts were provided a lunch and learn that went over the testing and validation performed for this attack technique. Additional collaboration will resume once the baselining activities are accomplished to discuss defensive automated responses with any associated triggered alerts.  

Pulling It All Together

After following this outline, an organization should be able to easily create an initial, overarching operational Threat Hunting process workflow (Crawl stage).

Let's see what this exercise output looks like.

12.       Individual Threat Hunting Process Output

1. Understand the business vertical the organization is part of and associated threats

Technology business sector

2. Identify the geographic presence of the organization

Headquartered in New York, but has a large international office located in the Netherlands

3. Identify threat groups targeting the organization, business vertical, or geographic region

APT29 - Russian Foreign Intelligence Service (SVR)

4. Identify Threat Hunting framework to model

MITRE ATT&CK framework

5.Identify attack testing validation framework to use

Red Canary's Atomic Red Team (ART) attack simulation framework

6. Identify a specific tactic of interest

TA0008 - Lateral Movement

7. Identify a specific technique of interest

Sub-Technique T1021.006 - Windows Remote Management

8. Run attack tests to identify visibility and detection gaps

Mycoolorg creates an isolated lab environment to perform all testing and identifies inadequacies in the current PowerShell logging visibility and that Sysmon is not installed within the environment to provide this additional telemetry.

9. Work on security operation tuning then retest

Due to logging overhead across the network and SIEM back-end storage constraints, only critical assets have PowerShell script block logging enabled and Sysmon installed. Production environment WinRM ART attack script testing produced the necessary logging visibility to enable adequate Threat Hunting and detection capabilities for this attack technique variation targeting critical assets.

10. Implement detection query, setup alerting, and establish response criteria

Mycoolorg transitions the Threat Hunting query into a production SIEM detection query with an alert criteria of Medium (for the time being). Further baselining activities are required to establish known-good WinRM Administrative use across the network. Security operation analysts are provided a lunch and learn going over the testing and validation performed for this attack technique. Additional collaboration will resume once the baselining activities are accomplished to discuss defensive automated responses with any associated triggered alerts.

11. Pulling it all together

Threat Hunting exercise output is documented, which provides a repeatable process, historical reference to hunting activities, attack technique testing and validation, identified logging visibility and gaps, and security operational playbook references.

An organization could take this planning a step further (Walk stage) by integrating the Palantir Alerting and Detection Strategy (ADS) framework to use for the individual Threat Hunting build document creation. To add a deeper layer of thoroughness (Run stage), an organization can also create an accompanying testing and validation document that would provide the low-level testing, validation of each attack technique, and the logging and detection capabilities captured. This combined documentation package would provide a very robust, repeatable process to follow and allow for security operation playbooks to be easily created.

Closing

OK, time to sit back and reflect a bit.

I know that seemed like a long process to accomplish a single Threat Hunt, but remember that the goal of Threat Hunting is to target threats that are applicable to the organization. This outlined methodology was only depicting one way of tackling the issue of getting Threat Hunting activities off the ground, with the overall objective to provide a systematic process-driven approach to Threat Hunting. From an overall security operation perspective, Threat Hunting will add significant levels of proactive threat detection, provide greater capabilities of decreasing the threat landscape, streamlining future Threat Hunting activities, and tightening the overall integration between security operations, Threat Hunting, and Incident Response workflows for the organization.

Happy hunting!

@H3dTr1p