Skip to article
Analytics Stream Analytics

Reference data

Reference data is the lookup information your streaming job needs but does not receive as live events. Think device catalogs, store locations, tariff codes, risk bands, or product mappings. In Azure Stream Analytics, the event stream keeps moving while reference data supplies

Source: Microsoft Learn - Use reference data for lookups in Azure Stream Analytics Reviewed 2026-05-22T00:00:00Z

Exam trap
Treating reference data like a live stream and expecting every late file or SQL change to take effect immediately.
Production check
Confirm the reference input alias in the job matches the alias used in every query JOIN.
Practice this topic Microsoft Learn
Article details and learning context
Aliases
Azure Stream Analytics reference data, lookup data, slowly changing lookup data, streaming reference dataset
Difficulty
intermediate
CLI mappings
5
Last verified
2026-05-22T00:00:00Z
Compare related concepts Search related resources Browse the glossary

Understand the concept

In plain English

Reference data is the lookup information your streaming job needs but does not receive as live events. Think device catalogs, store locations, tariff codes, risk bands, or product mappings. In Azure Stream Analytics, the event stream keeps moving while reference data supplies context for each event. Instead of calling a database for every message, the job joins events to a prepared dataset. That makes real-time decisions easier to explain, faster to run, and less dependent on application code.

Why it matters

Reference data matters because many real-time pipelines are useless without context. A temperature reading means little until it is joined to a sensor location, equipment type, maintenance class, or threshold table. A payment event needs merchant category, country risk, or rules metadata before it can be scored. Keeping that context as reference data lets teams change business rules and lookup values without rewriting the stream query every time. It also reduces database pressure, avoids per-event API calls, and gives operators a clear place to check when enrichment suddenly stops matching expectations. It turns a raw stream into decisions that support customers and auditors.

Official wording and source

In Azure Stream Analytics, reference data is a static or slowly changing dataset used to enrich or correlate event streams through reference data joins. It can come from Blob Storage, Data Lake Storage Gen2, or Azure SQL Database and is loaded for low-latency processing.

Open Microsoft Learn

Technical context

In Azure architecture, reference data sits in the Stream Analytics input layer beside streaming inputs and before outputs. It is configured as a named input alias, backed by Blob Storage, Data Lake Storage Gen2, or Azure SQL Database, and consumed by the query through a reference data join. The control plane stores input configuration, credentials, serialization, and refresh settings. The data plane loads the dataset into the job runtime so events can be enriched, filtered, or routed without adding another service call per event.

Exam context

Compare with

Event Hubs Stream Analytics input Data source in AI Search

Where it is used

Where you see it

  1. In the Stream Analytics portal, reference data appears under Inputs with an alias, source type, serialization settings, and either blob path details or SQL connection settings.
  2. In Azure CLI output, az stream-analytics input show exposes the input properties that prove which reference dataset the job will load and query. during incident review.
  3. In query design, engineers notice reference data in JOIN clauses that correlate live events with slow-changing tables such as device, tariff, or location mappings. inside the query editor.

Common situations

  • Enrich IoT telemetry with device, site, and maintenance metadata without querying an operational database for every event.
  • Apply current pricing, tariff, fraud, or entitlement rules to events while keeping lookup values outside application code.
  • Join clickstream or network events to region, category, or risk tables for real-time routing and alerting decisions.
  • Migrate a legacy stream processor by moving static dimension tables into Blob, ADLS Gen2, or SQL reference sources.
  • Keep compliance mappings, classification codes, or threshold tables versioned separately from the Stream Analytics query.

Illustrative Azure scenarios

These examples show how the concept can affect design and operations. They are illustrative scenarios, not customer claims.

Scenario 01 City transit agency enriches bus telemetry with route metadata Scenario, objectives, solution, measured impact, and takeaway.
Scenario

MetroTrail Transit streamed bus GPS and sensor events into Azure Stream Analytics, but dispatchers could not tell which route, depot, and service window each event belonged to during disruptions. The old enrichment service queried a database for every message and slowed during rush hour.

Goals
  • Attach route, depot, and service-window context to telemetry within seconds.
  • Remove per-event database calls from the streaming path.
  • Keep route changes visible before the morning schedule starts.
  • Give operators a clear rollback path for bad route tables.
Approach using Reference data

The platform team published route and depot tables as timestamped reference data in Data Lake Storage Gen2, then configured a Stream Analytics reference input with a stable alias used by the query JOIN. A Data Factory pipeline generated the next-day lookup file before 3:00 a.m., validated row counts, and kept the previous valid snapshot for rollback. Azure CLI checks listed inputs, captured the reference input definition, and verified the job state before each schedule update. Azure Monitor tracked late input events, join match rate, and output delays so dispatch could see whether enrichment was healthy.

Potential outcomes
  • Database calls from the telemetry pipeline dropped by ninety-four percent during peak service.
  • Route context appeared in dispatcher dashboards in under twelve seconds for normal event bursts.
  • Incorrect route assignments fell from forty-three incidents per month to six after snapshot validation.
  • Schedule rollback time improved from nearly an hour to less than ten minutes.
What to learn

Reference data lets a streaming job make real-time events operationally useful without turning every event into a database lookup.

Scenario 02 Energy retailer applies tariff changes during real-time meter scoring Scenario, objectives, solution, measured impact, and takeaway.
Scenario

NorthGrid Energy needed to score smart-meter events against tariff bands that changed by region, season, and demand-response campaign. The previous code-based rules required redeployment whenever pricing analysts updated a tariff.

Goals
  • Move tariff logic into governed lookup data instead of service code.
  • Apply new tariff bands before scheduled demand-response windows.
  • Reduce incorrect customer alerts caused by stale regional rules.
  • Preserve evidence of which tariff snapshot was active during disputes.
Approach using Reference data

Engineers modeled tariff bands as Stream Analytics reference data loaded from Azure SQL Database, where pricing analysts maintained approved rows. The job joined meter events to the tariff table by region, account class, and effective campaign code. Refresh settings were aligned with the pricing approval workflow, and the SQL snapshot container was monitored for failed refreshes. CLI runbooks exported the input definition, job status, and query aliases before each high-impact campaign. Support analysts used the snapshot timestamp and output fields to prove which rule set scored a disputed event.

Potential outcomes
  • Pricing-rule deployments fell from two per week to near zero because analysts updated SQL rows instead.
  • Demand-response campaign setup time dropped from three days to four hours.
  • Customer alert corrections decreased by sixty-eight percent in the first billing cycle.
  • Audit evidence for disputed scores was produced in minutes instead of multiple engineering handoffs.
What to learn

Reference data is a clean way to let approved business rules change quickly while the streaming query stays stable.

Scenario 03 Game studio correlates player events with anti-cheat rule tables Scenario, objectives, solution, measured impact, and takeaway.
Scenario

A multiplayer game studio streamed match events to detect suspicious behavior, but its rule service could not keep up during tournament weekends. Analysts also needed to update weapon, map, and device-risk tables without shipping a new detector.

Goals
  • Join player events with current anti-cheat lookup tables at tournament scale.
  • Let analysts update rule metadata without redeploying detection code.
  • Keep false-positive investigations tied to the exact lookup version used.
  • Avoid latency spikes during match-start bursts.
Approach using Reference data

The analytics team stored anti-cheat rule metadata as compact blob reference data and configured Stream Analytics to join match events against weapon class, map zone, and device-risk codes. The publishing pipeline wrote new snapshots ahead of tournament start times and blocked releases when row counts or required codes were missing. Azure CLI preflight checks confirmed the input alias, storage path, and job state. Dashboards compared event throughput, reference join coverage, alert volume, and output latency during live events, while old snapshots were retained only for the investigation window.

Potential outcomes
  • Match-start alert latency stayed below the five-second internal target during the championship weekend.
  • False-positive review time dropped by fifty-five percent because analysts could see the active rule snapshot.
  • Detector redeployments for rule metadata fell from weekly to monthly.
  • Storage cleanup removed thirty percent of stale snapshot files after the retention rule was introduced.
What to learn

Reference data gives streaming detection systems current context without putting brittle rule lookups in the hot event path.

Azure CLI

After ten years running Azure estates, I use Azure CLI for reference data because portal screenshots are weak evidence when a streaming job misses a lookup. CLI lets me list job inputs, inspect the exact reference input type, confirm the alias used in the query, capture source paths, and compare configuration across environments. It is also safer during incidents: I can prove whether the job is stopped, whether the input definition changed, and whether outputs are still connected before anyone edits the query. Scripted checks prevent slow, error-prone portal clicking during an outage. That consistency also protects handoffs between platform and data teams.

Useful for

  • List every Stream Analytics input and identify which ones are reference inputs versus streaming inputs before a change window.
  • Show the reference input JSON and compare alias, source, serialization, and path settings across dev, test, and production.
  • Stop and restart a job after publishing a new snapshot when the runbook requires a controlled reload and validation step.
  • Export input configuration as audit evidence before replacing a storage account key, SQL credential, or path pattern.
  • Check job state, inputs, and outputs together when enrichment disappears but Event Hub ingestion still looks healthy.

Before you run a command

  • Confirm tenant, subscription, resource group, Stream Analytics job name, region, input alias, and source storage or SQL resource first.
  • Check permissions for the job and backing data source because reading configuration may require different roles than updating input definitions.
  • Treat start, stop, update, and delete commands as production-affecting actions because they can interrupt event processing or change enrichment behavior.
  • Know whether the dataset contains sensitive mappings before exporting JSON output into tickets, logs, or shared incident channels.
  • Use JSON output for evidence and table output for quick triage, but preserve IDs, aliases, timestamps, and source properties exactly.

What the output tells you

  • The input type confirms whether the job is using reference data or a streaming source with a similar alias.
  • The properties block shows source account, path, SQL settings, serialization, and refresh-related fields that drive lookup behavior.
  • Provisioning state and job state separate configuration deployment problems from runtime processing or query errors.
  • Input alias values show the exact name that must match the Stream Analytics query JOIN expression.
  • Resource IDs and locations reveal whether the job points to the intended environment or to a copied test dependency.

Mapped commands

Stream Analytics reference data inspection

direct 
az stream-analytics input list --job-name <job> --resource-group <resource-group>
az stream-analytics inputdiscoverAnalytics
read-only Microsoft Learn 
az stream-analytics input show --job-name <job> --name <input-alias> --resource-group <resource-group>
az stream-analytics inputdiscoverAnalytics
read-only Microsoft Learn 
az stream-analytics job show --name <job> --resource-group <resource-group>
az stream-analytics jobdiscoverAnalytics
read-only Microsoft Learn 
az stream-analytics job stop --name <job> --resource-group <resource-group>
az stream-analytics joboperateAnalytics
mutating Microsoft Learn 
az stream-analytics job start --name <job> --resource-group <resource-group>
az stream-analytics joboperateAnalytics
mutating Microsoft Learn

Architecture context

Architecturally, reference data is the dimension layer of a streaming system. I place it close to Stream Analytics, but I manage it like production data: owned, versioned, refreshed, monitored, and cleaned up. Blob or ADLS reference data works well for scheduled snapshots, especially when file names include effective date and time. SQL reference data fits slowly changing lookup tables that need a refresh cadence. The critical design point is freshness. The job can keep running while using old context, so architects must define how new snapshots arrive, how old files retire, and how operators detect stale enrichment before customers do.

Security
Security impact is real even though reference data is not usually the highest-volume stream. Lookup tables can contain sensitive mappings such as customer segments, site codes, device ownership, pricing tiers, or fraud rules. Operators must protect the storage account or SQL database, avoid exposing access keys in deployment history, and prefer managed identity or scoped secrets where the service supports them. RBAC should separate people who can change Stream Analytics jobs from people who can alter reference data content. Network restrictions, private endpoints on backing stores, encryption, and diagnostic logging are part of the evidence trail. Review access after every pipeline or ownership change.
Cost
Reference data has no separate bill as a named concept, but it creates cost through storage, SQL Database, Stream Analytics streaming units, ETL jobs that prepare snapshots, and operational time spent diagnosing stale lookups. Oversized reference datasets can require more streaming units or limit query complexity, while long-retained blob sequences create unnecessary storage and listing overhead. Frequent SQL refreshes add database load and may require a larger SKU. FinOps owners should track who publishes the lookup data, how often it changes, how many jobs consume it, and whether abandoned snapshots or test jobs are still running. Chargeback reports should identify shared lookup publishers and consumers.
Reliability
Reference data has an indirect but important reliability role. A Stream Analytics job can be healthy while producing wrong results because the lookup dataset is missing, stale, too large, badly named, or unavailable during startup. Blob reference data depends on path patterns and timestamp ordering; SQL reference data depends on refresh cadence and database reachability. Large datasets can increase latency or reduce supported query complexity. Reliable designs publish snapshots before their effective time, keep old data only as long as needed, test job restarts, and monitor both input errors and downstream enrichment rates. Alerting should include both freshness and join-success trends.
Performance
Performance depends on keeping reference data appropriately sized and predictable. Stream Analytics loads reference data for low-latency joins, so small, well-structured lookup tables are usually fast. Problems appear when the dataset grows into a substitute warehouse, refreshes arrive late, serialization is inefficient, or the query combines reference joins with heavy windowing and temporal logic. Operators should baseline end-to-end latency before and after adding a lookup, watch late input events, and test restart behavior. If latency rises, reduce reference data size, pre-aggregate dimensions, simplify joins, or move heavier enrichment into another processing stage. Do not hide heavy analytical joins inside a real-time lookup.
Operations
Operators manage reference data by checking the input alias, source type, storage path, SQL connection, serialization, refresh behavior, and job status. They also validate the query join that uses the alias, because a correct input can still be ignored by a broken query. Good runbooks include the expected dataset owner, effective-time convention, file cleanup rule, refresh interval, and sample rows used for testing. During incidents, operators compare the current input definition with deployment history, inspect activity logs, check storage or SQL access, and confirm that enriched outputs contain expected joined fields. They should also record who can publish, approve, and retire snapshots.

Common mistakes

  • Treating reference data like a live stream and expecting every late file or SQL change to take effect immediately.
  • Changing the input alias without updating the query, which leaves the job deployed but unable to compile or enrich correctly.
  • Uploading blob snapshots after their effective time, causing the job to miss or delay the intended lookup version.
  • Letting reference datasets grow past practical limits instead of trimming old codes, aggregating dimensions, or splitting logic.
  • Rotating storage keys or SQL credentials without validating that the Stream Analytics input can still read the source.