Optimizing Cost and Visibility: Ingesting MDE Device Logs into Microsoft Sentinel

Ingest MDE device logs into Microsoft Sentinel using the low-cost storage option?

Introduction

In the realm of Security Operations, having access to device logs from Microsoft Defender for Endpoint (MDE), now part of M365 Defender, within a SIEM platform is crucial for effective threat hunting. However, storing these logs can be costly, especially as log volumes grow with the number of devices in an organization. This blog post presents a solution to ingest MDE device logs into Microsoft Sentinel using the low-cost storage option, Basic Logs, available in Log Analytics Workspace. By leveraging the latest capabilities of Data Collection Rules, organizations can achieve comprehensive visibility at a reduced cost.

Understanding the Challenge

The current architecture of MDE device logs ingestion into Log Analytics Workspace relies on storing logs as Analytic Logs by default when ingested through the native M365 Defender Data Connector. Although Log Analytics Workspace offers a cheaper storage option called Basic Logs, it is not supported for all log types. To overcome this challenge, organizations have explored alternatives such as collecting logs directly from the M365 Defender Streaming API and pushing them to Log Analytics Workspace using additional resources like Function Apps or Cribl. However, this approach introduces complexity and resource overhead.

The Solution

With the latest updates from Microsoft, Data Collection Rules now support the collection of logs from "Data Connector Streams," which include native data types from Microsoft 365 Defender and other sources, except for five specific data types. This enhancement enables redirecting the log stream to a different table, leveraging Basic Logs in Log Analytics Workspace. This approach simplifies the ingestion process and reduces costs without compromising visibility.

Implementing the Solution

To implement this low-cost ingestion solution, follow these steps:

1. Configure M365 Defender Data Connector on Sentinel: Establish the M365 Defender Data Connector to establish a connection between MDE and Microsoft Sentinel.

2. Create a Custom Table in Log Analytics Workspace: Create a custom table within the Basic Logs storage option in Log Analytics Workspace. Ensure that the table's schema matches the log source's data type, which may vary depending on the log type.

3. Create a Data Collection Rule: Set up a Data Collection Rule for the desired device log table, with the option to filter the data using the transformation editor if needed.

4. Define the Output Stream: Edit the Data Collection Rule to include an output stream parameter. Specify the value to point to the output stream of the newly created custom table.

5. Validation and Archiving: Allow approximately 15 minutes for the Data Collection Rule to take effect. Validate that the device logs are being successfully ingested into the custom table. As Basic Logs have a maximum retention period of only 8 days, consider archiving logs older than this timeframe natively within Log Analytics Workspace, ensuring they are readily available when needed.

Benefits and Considerations

By implementing this solution, organizations can experience several benefits:

1. Cost Savings: Leveraging the Basic Logs storage option significantly reduces the cost associated with storing MDE device logs.

2. Enhanced Visibility: The solution provides comprehensive visibility into endpoint logs within Microsoft Sentinel, empowering effective threat hunting.

3. Optimization and Resource Efficiency: This approach optimizes the ingestion process and minimizes the resources required, streamlining operations.

Conclusion

As one of the fastest-evolving SIEM platforms, Microsoft Sentinel continues to address challenges previously encountered in security operations. The solution outlined in this blog post enables large-scale organizations to ingest heavy logs into Microsoft Sentinel at a reduced cost while maintaining optimal visibility. By following the recommended steps, organizations can effectively manage costs and leverage the full potential of their security operations.

Disclaimer: Prior to implementing this solution, thorough testing in a non-production environment is highly recommended.