AnalyticsAzure Data Explorerfield-manual-completefield-manual-completefield-manual-complete
Streaming ingestion
Streaming ingestion is the path that gets fresh events into Azure Data Explorer quickly, often within seconds, instead of waiting for larger batch ingestion cycles. It is useful for operational telemetry, logs, alerts, and near-real-time analytics where people need to query the newest data quickly. It is not the best answer for every high-volume table; queued ingestion may be better for large sustained loads. The practical decision is latency versus throughput efficiency, table design, mapping quality, and operational cost.
Azure Data Explorer streaming ingestion, ADX streaming ingestion, Kusto streaming ingestion, low latency ingestion
Difficulty
intermediate
CLI mappings
5
Last verified
2026-05-26T19:53:46Z
Microsoft Learn
Microsoft Learn describes streaming ingestion in Azure Data Explorer as a low-latency ingestion path for continuous data streams. Event Hubs, IoT Hub, and Event Grid data connections can use it when streaming ingestion is enabled on the cluster and allowed by policy.
In Azure architecture, streaming ingestion sits in the Azure Data Explorer data ingestion plane. It commonly connects Event Hubs, IoT Hub, Event Grid, custom clients, ingestion mappings, target tables, databases, and cluster-level settings. The control plane manages clusters, databases, and data connections, while Kusto policies decide whether streaming is enabled at database or table scope. Operators also need monitoring, retention, update policies, managed identities, network access, and data format validation. It is a data platform capability, not an application hosting feature.
Why it matters
Streaming ingestion matters because some analytics lose value when data arrives minutes later. Security operations, game telemetry, factory signals, and user-facing monitoring often need the newest rows queryable quickly. At the same time, streaming ingestion is not a free performance button. It has limits, table-policy considerations, mapping requirements, and trade-offs against queued ingestion. Teams that enable it blindly can create ingestion failures, hot tables, or unnecessary cluster pressure. Teams that use it deliberately can give analysts low-latency visibility without building custom collectors, while still keeping schema, retention, identity, and governance under Azure Data Explorer control. That discipline protects production trust.
⌁
Where you see it
Signals, screens, and Azure surfaces where this term usually becomes operational.
Signal 01
In the Azure Data Explorer cluster configuration, operators verify whether streaming ingestion is enabled before relying on low-latency data connections. for production pipelines and audits
Signal 02
In Kusto database or table policies, the streaming ingestion policy shows whether specific target tables accept the streaming path. before source cutovers and incidents reviews
Signal 03
In Event Hub data connection settings, teams map source events to ADX tables with consumer group, data format, mapping rule, and identity choices. during onboarding
✦
When this becomes relevant
Specific situations where this term helps solve real Azure design, operations, migration, security, reliability, cost, or governance problems.
Make security, operations, or product telemetry queryable in Azure Data Explorer within seconds for live investigation dashboards.
Ingest many small event streams across multiple tables where low latency matters more than maximum bulk throughput.
Connect Event Hubs or IoT Hub to ADX with governed mappings so application teams do not build custom ingestion services.
Route high-volume historical backfills through queued ingestion while keeping fresh operational signals on streaming ingestion.
Test schema, mapping, and identity changes before a real-time analytics pipeline loses events during a source migration.
◆
Real-world case studies
Different enterprise-style examples that show the term being used to hit measurable objectives.
Case study 01
Game studio makes anti-cheat telemetry queryable in seconds
Scenario, objectives, solution, measured impact, and takeaway.
📌Scenario
A multiplayer game studio streamed match telemetry into Azure Data Explorer for anti-cheat investigations. Queued ingestion was reliable but analysts waited several minutes before suspicious behavior appeared in Kusto queries during live tournaments.
🎯Business/Technical Objectives
Make selected anti-cheat events queryable within five seconds.
Keep bulk match history on efficient queued ingestion.
Avoid building a custom ingestion service for tournament operations.
Preserve replay ability from Event Hubs after mapping failures.
✅Solution Using Streaming ingestion
The data platform team enabled streaming ingestion for a narrow set of high-priority ADX tables and kept larger match summaries on queued ingestion. Event Hubs data connections mapped suspicious movement, inventory, and latency events to dedicated hot tables with strict JSON mappings. CLI checks inventoried the ADX cluster, database, Event Hub data connections, identities, and regions before the Kusto streaming policy was changed. The team retained eight hours of Event Hubs data, added ingestion-failure dashboards, and wrote a runbook for falling back to queued ingestion if tournament traffic exceeded the streaming path design.
📈Results & Business Impact
Median query freshness for anti-cheat signals improved from 4.7 minutes to 3.8 seconds.
Tournament investigators reduced live-review backlog by 58% during peak matches.
Bulk match-history ingestion cost stayed stable because only hot tables used streaming ingestion.
Two mapping failures were replayed from Event Hubs without losing tournament evidence.
💡Key Takeaway for Glossary Readers
Streaming ingestion works best when low-latency tables are deliberately separated from bulk historical ingestion.
Case study 02
Robotics manufacturer shortens fault isolation on assembly lines
Scenario, objectives, solution, measured impact, and takeaway.
📌Scenario
A robotics manufacturer collected controller logs, torque readings, and stop events from assembly-line cells. Plant engineers needed fresh data in Azure Data Explorer before a small motor issue cascaded into a line stoppage.
🎯Business/Technical Objectives
Reduce time from controller event to queryable ADX row below ten seconds.
Correlate stop events with torque anomalies during active shifts.
Limit streaming ingestion to operational tables that justify the latency.
Protect production networks with governed identities and private routing.
✅Solution Using Streaming ingestion
The platform team connected factory Event Hubs to Azure Data Explorer through data connections and enabled streaming ingestion on the specific operational tables used by shift engineers. They kept engineering archives and long-form diagnostics on queued ingestion. Mappings were versioned with line, robot, cell, and event-type fields, while managed identities limited which connections could ingest. CLI output documented cluster state, database retention, data connection settings, and network properties before go-live. A Kusto workbook tracked ingestion latency, mapping errors, and table freshness so line supervisors could tell whether a query showed current production state.
📈Results & Business Impact
Controller-to-query latency dropped from 2.9 minutes to 6.1 seconds at the median.
Mean time to isolate repeat motor faults fell from 42 minutes to 14 minutes.
Only nine hot tables used streaming ingestion, avoiding unnecessary pressure on archival workloads.
Unauthorized ingestion paths were eliminated by replacing shared keys with managed identity-based connections.
💡Key Takeaway for Glossary Readers
Streaming ingestion gives operations teams fresh analytical visibility when table scope, identity, and fallback patterns are intentionally designed.
Case study 03
Port authority tracks berth sensor events during storms
Scenario, objectives, solution, measured impact, and takeaway.
📌Scenario
A port authority monitored berth sensors, crane events, and vessel-movement messages during storms. Batch-style ingestion left harbor coordinators looking at stale wind and mooring signals while deciding whether to pause container operations.
🎯Business/Technical Objectives
Make critical berth and crane signals queryable in near real time.
Keep historical vessel archives on cost-efficient ingestion paths.
Validate mappings before adding new storm-season sensors.
Create a replay plan for source outages longer than normal maintenance windows.
✅Solution Using Streaming ingestion
Engineers enabled streaming ingestion on ADX tables that powered storm dashboards and left large historical vessel feeds on queued ingestion. Event Hub data connections used a dedicated consumer group and mapping rules for berth ID, crane ID, wind reading, and alarm state. Azure CLI checks confirmed the cluster, database, data connection, managed identity, and resource group before Kusto policies were altered. The team also increased Event Hubs retention for storm season and added alerts for ingestion failures, latency, and missing berth IDs. Dry-run events from a simulated storm verified schema and dashboard freshness before the first seasonal deployment.
📈Results & Business Impact
Dashboard data freshness improved from roughly six minutes to under eight seconds.
Storm-operation pause decisions used current berth telemetry for 96% of drills.
Mapping defects for new sensors were caught in pre-season tests instead of live weather events.
Replay coverage increased from two hours to 24 hours by aligning Event Hubs retention with recovery needs.
💡Key Takeaway for Glossary Readers
Streaming ingestion is an operational-safety tool when fresh data changes real-world decisions and replay planning is built in.
Why use Azure CLI for this?
CLI is useful for streaming ingestion because the feature spans the Azure Data Explorer cluster, database, data connection, Event Hubs or IoT Hub source, and identity permissions. The portal can show each piece, but CLI can collect them together and make the configuration auditable. Engineers can verify whether the cluster supports streaming ingestion, inspect data connections, confirm mapping and consumer group choices, and capture JSON for change review. For migration or incident work, commands compare regions and environments quickly, which is essential when low-latency telemetry depends on many small settings being correct before data loss or dashboard staleness becomes visible to users.
CLI use cases
List and show Azure Data Explorer clusters to confirm region, SKU, state, identity, and network settings for the ingestion path.
Show databases and retention settings before enabling table policies that make fresh data queryable quickly.
List Event Hub data connections and confirm source resource ID, consumer group, table, mapping, and data format.
Create or update data connections through controlled deployment when a new event stream feeds an ADX table.
Export cluster, database, and data connection JSON for audit evidence before changing ingestion policy with Kusto commands.
Before you run CLI
Confirm tenant, subscription, resource group, cluster, database, table, region, and Event Hub namespace before inspecting or changing resources.
Check permissions for Azure resource management and separate Kusto database permissions needed to alter ingestion policy or mappings.
Use read-only list and show commands before creating data connections or changing settings that can start cost-impacting ingestion.
Verify source retention, replay plan, schema owner, mapping rule, and output format before testing production event streams.
What output tells you
Cluster output shows state, region, SKU, identity, URI, and networking details that frame ingestion capacity and access.
Database output shows retention, hot cache, provisioning state, and resource identifiers relevant to query freshness and cost.
Data connection output shows which Event Hub, consumer group, table, mapping, and format feed the streaming ingestion path.
Errors or missing resources help separate Azure control-plane issues from Kusto policy, mapping, or table-schema issues.
Mapped Azure CLI commands
Azure Data Explorer streaming ingestion control-plane checks
inspects
az kusto cluster show --name <cluster-name> --resource-group <resource-group>
az kusto clusterdiscoverAnalytics
az kusto database show --cluster-name <cluster-name> --resource-group <resource-group> --database-name <database-name>
az kusto databasediscoverAnalytics
az kusto data-connection event-hub list --cluster-name <cluster-name> --database-name <database-name> --resource-group <resource-group>
az kusto data-connection event-hubdiscoverAnalytics
az kusto data-connection event-hub show --cluster-name <cluster-name> --database-name <database-name> --resource-group <resource-group> --data-connection-name <connection-name>
az kusto data-connection event-hubdiscoverAnalytics
az kusto data-connection event-hubprovisionAnalytics
Architecture context
Architecturally, streaming ingestion is the low-latency branch of an Azure Data Explorer ingestion design. I separate it from queued ingestion in diagrams because it has different goals and failure modes. The source contract, mapping rule, table schema, retention policy, update policy, and query workload must be considered together. If many small tables need fresh data, streaming ingestion can be excellent. If one table receives very high sustained volume, queued ingestion may be safer and cheaper. Mature designs also include dead-letter handling, replay from Event Hubs retention, schema-change procedures, and dashboards for ingestion latency and failures. Review both paths together before production.
Security
Security impact is direct because streaming ingestion opens a fast path into analytical tables. The source identity, Event Hub permissions, data connection identity, database permissions, and network boundaries must be least privilege. Private endpoints, managed identities, customer-managed keys where used, and firewall rules should align with the sensitivity of the incoming data. A bad mapping or permissive source can push sensitive fields into tables faster than review teams notice. Operators should control who can create data connections, alter ingestion policies, change mappings, or grant database ingestor permissions because those actions affect data exposure immediately. Audit trails should capture every policy change.
Cost
Cost impact is indirect and sometimes substantial. Streaming ingestion may require a cluster size that can handle fresh data, many small writes, and concurrent queries. It can reduce business cost by making operations faster, but it can also add cluster pressure, storage, retention, monitoring, and support effort. For high-volume sustained streams, queued ingestion can be more efficient. Cost control means choosing streaming only where low latency matters, keeping schemas narrow, monitoring ingestion failures, avoiding unnecessary duplicate tables, and reviewing Event Hubs throughput, ADX capacity, retention, and update-policy work together rather than in isolation. Monthly reviews should compare both ingestion paths.
Reliability
Reliability depends on source retention, ingestion policy, mapping correctness, table schema, cluster health, and retry behavior. Streaming ingestion can reduce latency, but it also makes malformed events and schema drift visible quickly. If Event Hubs retention is too short, recovery from ingestion failures may lose data. If an update policy with joins conflicts with streaming ingestion assumptions, downstream tables can fail or lag. Reliable designs include replay windows, ingestion failure monitoring, mapping tests, versioned schemas, and a fallback to queued ingestion when sustained volume exceeds the intended streaming path. Operators should test failure and backfill scenarios before declaring the pipeline production ready.
Performance
Performance impact is direct because the goal is lower ingestion latency. The actual result depends on event volume, mapping complexity, table count, cluster capacity, update policies, and query load. Streaming ingestion works best for many small streams that need seconds-level availability, not every bulk ingestion scenario. Large records, bad mappings, oversized fields, or heavy update policies can turn a low-latency design into a bottleneck. Operators should monitor ingestion latency, failure rate, extent creation behavior, CPU, cache pressure, and query concurrency. Performance tuning may mean switching some flows to queued ingestion, simplifying mappings, or separating hot operational tables from historical analytics tables.
Operations
Operators inspect streaming ingestion by checking cluster configuration, database and table policies, data connections, ingestion mappings, identity permissions, and failure metrics. They compare ingestion latency with source event volume and query freshness. During incidents, they determine whether the problem is source delivery, Event Hub retention, data connection status, mapping failure, table policy, cluster throttling, or downstream update policy. Operational runbooks should include Azure CLI resource checks, Kusto policy commands, sample ingestion validation, replay steps, and clear ownership between platform, data engineering, and application teams. They should record which checks are Azure CLI, which are Kusto commands, and who owns each.
Common mistakes
Enabling streaming ingestion for bulk loads that would be cheaper and more efficient through queued ingestion.
Forgetting that Event Hubs retention controls how much data can be replayed after ingestion failures.
Changing table schema or mappings without testing live events and failure handling first.
Granting broad database permissions when only ingestion-specific roles or controlled data connections are needed.
Ignoring update policy restrictions and then discovering downstream tables lag or fail when streaming records arrive.