© 2020 Content Security Pty Ltd.


MITRE ATT&CK Evaluations: Is your EDR keeping up?

The MITRE ATT&CK framework refers to a globally accessible matrix of real-world adversary tactics and techniques. As its primary function, it has acted as a foundational knowledge base and standardised language that allows for more effective threat detection across the private sector, governments and cyber security companies. However, in recent years formal evaluations based on ATT&CK have led to more strategic product development and considerations in the EDR market.

A quick summary of the MITRE ATT&CK Enterprise matrix

ATT&CK stands for Adversarial Tactics, Techniques and Common Knowledge. While we will primarily be discussing the MITRE ATT&CK Enterprise framework (which looks at Windows, Linux and MacOS systems), the approach is also split into PRE-ATT&CK (pre-compromise) and Mobile ATT&CK matrices.


The ‘tactics’ refer to why or what an adversary is trying to achieve. Under Enterprise ATT&CK, there are 12 tactics:
  • Initial access
  • Execution
  • Persistence
  • Privilege escalation
  • Defence evasion
  • Credential access
  • Discovery
  • Lateral movement
  • Collection
  • Command and control
  • Exfiltration
  • Impact


Each tactic can be considered a step in the adversary’s overall mission and can be achieved by employing a variety of techniques. MITRE has collated hundreds of techniques that are known to be used in the wild; this list is based off of a database detailing clusters of adversary activity from over 107 actors and associations.

To know ATT&CK is to understand the enemy.

Essentially, the MITRE ATT&CK framework can be used to characterise and describe known patterns of malicious behaviour and assists in prioritising network defences by showcasing the tactics, techniques and procedures (TTPs) that threat hunters (and their chosen endpoint detection solutions) should be looking out for.

MITRE ATT&CK product evaluations

Adversarial simulations such as ATT&CK evaluations are some of the best ways to test an EDR product’s behaviour against complex, real-world threats. Measuring your tools against the ATT&CK framework can ensure you are getting the coverage needed against known adversaries, and help you understand gaps in your visibility. Further, it assists in validating the configuration of your tools, demonstrating where different types of threat-actors would be caught and where they could slip through the cracks.

MITRE created Adversary Emulation Plans to allow red teams to actively simulate adversary behaviour and independently test their organization’s network security and security products against specific threats.

ATT&CK evaluation results can help you choose the right EDR

Since 2018, MITRE have been conducting their own in-depth investigations into how EDR solutions perform against known adversary behaviours. These are not competitive analyses. They do not include scoring, ranking or direct comparisons. Rather, they look at showcasing the products’ baseline detection capabilities.

If you are looking for a convenient way to evaluate vendor results, MITRE have created a Vendor Comparison tool.

The participants are tested on their ability to detect malicious activity across the full attack spectrum – from initial compromise to exfiltration and impact. These results can tell you:
  • What malicious activity the product can detect and alert you to;
  • What relevant context was provided about the malicious activity; and occasionally
  • How fast the product gathered relevant information about said activity.

The 2020 Enterprise evaluation is based on Carbanak and FIN7 emulations. Carbanak is a threat group that primarily targets banks, and FIN7 is a financially motivated group that has largely targeted the U.S. retail and hospitality sectors.

Interpreting MITRE ATT&CK results

While these evaluations do not provide the best vendor per se, it can answer the following questions for you:

  • Does your product detect known threats to the organisation?
  • How does your product present the data to your analysts?

When looking at the results, you need consider how the vendor detected the threat in each step. As in 2019’s evaluation, MITRE categorises detections into the following types:

  • None: unable to collect any information related to malicious activity.
  • Telemetry: minimally processed data that might indicate malicious activity – essentially an event log.
  • MSSP: detection was based on human analysis.
  • General: indicates anomalous or suspicious behaviour but does not provide further details on compromise.
  • Tactic: information on potential intent of the activity provided.
  • Technique: the product alerts you to what is going on and how the attack is conducted using the specific ATT&CK technique classifications.
These are all further categorised with detection modifiers that can give a clearer picture on how and when alerts were generated, such as delayed, configuration change and tainted modifiers.

Defining alerts

While it is important that the product demonstrates they can effectively categorise alerts for analysts, it is equally important to note that not all malicious activity should fall under one specific detection type. It should be given the appropriate type for its behaviour.

However, a product that consistently produces a ‘none’ or ‘telemetry’ type throughout the assessment is likely to be low performing and could possibly lead to alert fatigue. Intrusion analysts and threat hunters cannot rely on EDR solutions that make them look for the supporting context of the threat.

As an additional resource, see MITRE’s ATT&CK evaluation results interpretation guide.

What are the limitations of these assessments?

The evaluation focuses entirely on detection – not prevention. The adversary behaviour is executed stage by stage and the product is barred from enacting preventative measures. This is because MITRE is solely evaluating detection performance, and this could not be done efficiently if the product were consistently shutting down attacks. Some solutions may deliver a variety of top-of-the-range prevention controls – it is necessary to do research outside of the assessment in order to fully evaluate those solutions as a whole.

It is also important to note that some attacks might require human expertise to fully uncover. You need to look at your existing tools and skillset, and from there decide which product capabilities are necessary for you.

Another issue to consider is how EDR vendors have now mapped their products to the framework, advertising that they ‘cover all MITRE listed attacks/techniques.’ However, during the evaluation the vendor might only encounter a few procedures simulated in a test environment. Therefore, this assessment provides an estimate of the solutions’ capabilities and should not be taken as the gold standard.


It is advantageous for any vendor producing detection tools to consult the unbiased and raw data held within the MITRE ATT&CK matrices, and moreover, let the evaluations drive their product enhancements. As MITRE continues to conduct these evaluations and update their database, the ATT&CK methodology (and products modelled off this framework) will grow more sophisticated. Governments, companies, and the cyber security industry will continue to reap the rewards of staying on top of known threats and will make more informed EDR investments and implementations.

For more information please contact our cybersecurity professionals today.

Recent news