Troubleshooting Data Ingestion Lag in Microsoft Sentinel from Zscaler Logs

In the ever-evolving landscape of cybersecurity monitoring, timely and accurate log ingestion is king. Recently, a curious case of data ingestion lag cropped up while monitoring large Microsoft Teams data transfers using Zscaler logs in Microsoft Sentinel. This hiccup risked hiding crucial spikes in data transfer that could indicate misuse or exfiltration.

Let’s walk through the issue, the risks, and how a smart fix resolved the delay—so your alerting stays sharp and your network stays safe.

The Setup: Detecting Microsoft Teams Large Data Transfers with KQL

The initial goal was straightforward: use Kusto Query Language (KQL) inside Microsoft Sentinel to detect instances where Microsoft Teams data transfer exceeds 50 gigabytes within a 10-minute window. This helps security teams spot unusual large-scale transfers that may signal unauthorized file sharing or bandwidth abuse.

The original query relied on Zscaler logs filtered to Teams’ CDN endpoints (statics.teams.cdn.office.net), summing sent and received bytes every 10 minutes.

Original Query Outline:

let threshold = 50000000000; // 50GB
CommonSecurityLog
| where DeviceVendor == "Zscaler"
| where DestinationHostName == "statics.teams.cdn.office.net"
| summarize sum(SentBytes), sum(ReceivedBytes) by bin(TimeGenerated, 10m), DeviceVendor, DestinationHostName
| where sum_SentBytes > threshold or sum_ReceivedBytes > threshold
| project DeviceVendor, DestinationHostName, SentTotal=format_bytes(sum_SentBytes, 5, "GB"), ReceivedTotal=format_bytes(sum_ReceivedBytes, 5, "GB"), TimeGenerated

The original script neatly sums and flags Teams data over 50GB in 10-minute chunks.

The Problem: Data Ingestion Lag

The catch? There was a noticeable ingestion delay—Zscaler logs arriving about 2 minutes late into Microsoft Sentinel. The original query looked strictly at data generated in the last 10 minutes, but with late-arriving logs, some data slipped through unnoticed.

That lag caused the monitoring window to be slightly off, risking the chance of missing critical events during the delay. In security monitoring, a delay like that is like trying to catch a thief after they’ve already left the scene.

The Fix: Accounting for Ingestion Delay in the Query

The updated KQL query compensates for the ingestion delay by extending the look-back period by the estimated lag time (2 minutes). This ensures that late-arriving logs are included in the detection window, providing a more accurate picture.

Key changes in the fixed query:

• Introduced a variable ingestion_delay to represent the 2-minute lag.
• Extended the query’s time window by adding ingestion_delay to the 10-minute look-back.
• Added a condition on ingestion_time() to filter logs ingested within the last 10 minutes, ensuring freshness.

Updated Query Outline:

let ingestion_delay = 2min;
let rule_look_back = 10min;
let size_threshold = 53687091200; // 50GB
CommonSecurityLog
| where DeviceVendor == "Zscaler"
| where DestinationHostName == "statics.teams.cdn.office.net"
| where TimeGenerated >= ago(ingestion_delay + rule_look_back)
| where ingestion_time() > ago(rule_look_back)
| summarize sum(SentBytes), sum(ReceivedBytes) by bin(TimeGenerated, 10m), DeviceVendor, DestinationHostName
| where sum_SentBytes > size_threshold or sum_ReceivedBytes > size_threshold
| project DeviceVendor, DestinationHostName, SentTotal=format_bytes(sum_SentBytes, 5, "GB"), ReceivedTotal=format_bytes(sum_ReceivedBytes, 5, "GB"), TimeGenerated

By sliding the monitoring window backward, the query catches delayed logs, closing the gap in detection. See the full article fix here.

Why This Matters: Risk and Impact

Large data transfers via Microsoft Teams might seem innocent, but they could also hint at unauthorized file sharing or data exfiltration attempts. Ignoring ingestion lag increases risk—potential breaches or compliance issues may go undetected.

Addressing ingestion lag is a small tweak with a big impact—making sure security teams aren’t blindsided by late logs and can respond promptly.