ForEach activity
A ForEach activity is an Azure Data Factory or Azure Synapse pipeline control-flow activity that iterates over a collection and executes child activities for each item. Teams use it to run repeatable pipeline steps for files, tables, partitions, API pages, customers, dates, or metadata-driven work items without manually building one activity per item. It is not a mapping data flow transformation, a guarantee of ordered execution when parallelism is enabled, a substitute for idempotent design, or a safe pattern for unlimited fan-out.
Source: Microsoft Learn - ForEach activity in Azure Data Factory and Azure Synapse Reviewed 2026-05-14
- Exam trap
- Treating ForEach activity as a label instead of checking the exact resource scope, live configuration, owner, and dependencies.
- Production check
- Verify resource scope, enabled state, identity, network path, diagnostics, owner tags, and linked resources before changing production behavior.
Article details and learning context
- Aliases
- ADF ForEach activity, Azure Data Factory ForEach, Synapse ForEach activity, pipeline loop activity
- Difficulty
- intermediate
- CLI mappings
- 6
- Last verified
- 2026-05-14
Understand the concept
Why it matters
ForEach activity matters because it is one of the main ways Azure data pipelines turn metadata into scalable repeatable work. Without clear vocabulary, teams may overload sources with parallel loops, create duplicate writes, misuse variables in concurrent execution, hide failures inside nested activities, or forget that one bad item can affect a whole run. It also affects security, reliability, operations, cost, and performance because one configuration choice can change who can act, what fails, how quickly work completes, what evidence exists, and how much the platform costs. Good glossary discipline helps teams ask who owns it, what depends on it, which metric proves health, and what rollback path exists before a release.
Official wording and source
A ForEach activity is an Azure Data Factory or Azure Synapse pipeline control-flow activity that iterates over a collection and executes child activities for each item.
Technical context
Technically, the ForEach activity is configured or observed through Data Factory pipelines, Synapse pipelines, items expressions, inner activities, batchCount, isSequential, activity runs, pipeline parameters, variables, datasets, integration runtimes, triggers, and monitoring logs. It depends on collection shape, expression language, child activity idempotency, parallelism setting, integration runtime capacity, linked service availability, retry policy, parameter passing, variable usage, and downstream service limits. Operators inspect it through the Azure portal, ARM or Bicep, Azure CLI, SDK or REST calls, Azure Monitor, diagnostic logs, and application telemetry. During troubleshooting, connect scope, permissions, runtime state, metrics, and downstream evidence before changing production settings.
Exam context
Compare with
Where it is used
Where you see it
- A pipeline JSON definition contains a ForEach activity with items, activities, isSequential, batchCount, parameters, and nested child activity definitions. Review scope, owners, metrics, and rollback evidence.
- Pipeline monitoring shows many child activity runs under one parent loop, often with item-specific failures, retry attempts, or uneven item durations. Review scope, owners, metrics, and rollback evidence.
- Design notes mention metadata-driven ingestion, table lists, file partitions, API pagination, customer-by-customer processing, or concerns about parallel fan-out. Review scope, owners, metrics, and rollback evidence.
Common situations
- Loop over a metadata table, date list, or file listing and execute the same child activities for each item.
- Troubleshoot one failed item in a nested pipeline by correlating ForEach activity runs with child activity output.
- Tune parallelism so a pipeline finishes faster without overloading a database, API, storage account, or integration runtime.
- Support incident response by correlating Azure configuration, diagnostic logs, metrics, deployment history, and application traces.
Illustrative Azure scenarios
These examples show how the concept can affect design and operations. They are illustrative scenarios, not customer claims.
Scenario 01 ForEach activity in action for manufacturing analytics Scenario, objectives, solution, measured impact, and takeaway.
NorthPeak Analytics, a manufacturing analytics organization, needed to solve a production challenge: a factory ingestion pipeline copied sensor files from 180 production lines and failed whenever one line produced malformed metadata. The architecture team used ForEach activity to make the design measurable, governable, and easier to support.
- Process every line independently
- Identify the failing line quickly
- Avoid duplicate file loads
- Keep daily ingestion under two hours
Engineers used a ForEach activity over a metadata list of production lines, passed each line ID to Copy activity, and enabled bounded parallelism. Failure output captured the item value so support could rerun only the affected line. Before cutover, engineers captured read-only configuration, validated identity and network access, compared expected behavior with Azure Monitor or service logs, and stored rollback instructions in the change record. Operators received a runbook with first-response checks, known failure modes, owner contacts, and escalation paths. The team also reviewed owner tags, diagnostic coverage, alert routing, and incident communication paths so support could confirm the workflow without changing production state. The team also reviewed owner tags, diagnostic coverage, alert routing, and incident communication paths so support could confirm the workflow without changing production state.
- Daily ingestion finished in 91 minutes
- Duplicate loads were eliminated with idempotent paths
- Failed line identification took minutes instead of hours
- Reruns targeted only bad metadata items
ForEach activity is strongest when each loop item can be tracked, retried, and explained independently.
Scenario 02 ForEach activity in action for public sector Scenario, objectives, solution, measured impact, and takeaway.
MetroGrant Office, a public sector organization, needed to solve a production challenge: monthly grant reports required one pipeline per agency, but manual pipeline duplication caused inconsistent parameters. The architecture team used ForEach activity to make the design measurable, governable, and easier to support.
- Replace duplicate pipelines
- Support 42 agencies
- Keep agency secrets protected
- Create run evidence for each item
The team stored agency metadata in a control table and used ForEach to pass agency ID, dataset path, and reporting period into child activities. Linked service secrets stayed in Key Vault and monitoring captured item-level outcomes. Before cutover, engineers captured read-only configuration, validated identity and network access, compared expected behavior with Azure Monitor or service logs, and stored rollback instructions in the change record. Operators received a runbook with first-response checks, known failure modes, owner contacts, and escalation paths. The team also reviewed owner tags, diagnostic coverage, alert routing, and incident communication paths so support could confirm the workflow without changing production state. The team also reviewed owner tags, diagnostic coverage, alert routing, and incident communication paths so support could confirm the workflow without changing production state.
- Pipeline count dropped from 42 to one
- Agency-specific failures became visible
- Secret handling passed review
- Monthly reporting finished a day earlier
A metadata-driven ForEach loop reduces copy-paste pipeline risk when parameters and permissions are governed.
Scenario 03 ForEach activity in action for online travel Scenario, objectives, solution, measured impact, and takeaway.
BlueHarbor Travel, a online travel organization, needed to solve a production challenge: an API ingestion job hit partner rate limits when all destination pages ran in parallel. The architecture team used ForEach activity to make the design measurable, governable, and easier to support.
- Respect partner API limits
- Reduce failed retries
- Finish ingestion before business opening
- Make throttling visible
Architects tuned ForEach batchCount, added retry delays, and logged page IDs that hit throttling. Azure Monitor alerts compared activity-run failures with API response codes during the ingestion window. Before cutover, engineers captured read-only configuration, validated identity and network access, compared expected behavior with Azure Monitor or service logs, and stored rollback instructions in the change record. Operators received a runbook with first-response checks, known failure modes, owner contacts, and escalation paths. The team also reviewed owner tags, diagnostic coverage, alert routing, and incident communication paths so support could confirm the workflow without changing production state. The team also reviewed owner tags, diagnostic coverage, alert routing, and incident communication paths so support could confirm the workflow without changing production state. The team also reviewed owner tags, diagnostic coverage, alert routing, and incident communication paths so support could confirm the workflow without changing production state.
- 429 responses fell by 68 percent
- Ingestion completed before opening time
- Retry cost decreased through bounded fan-out
- Partner support accepted the run evidence
ForEach performance tuning is really downstream protection, not just pipeline speed.
Azure CLI
Azure CLI helps validate ForEach activity because it captures reproducible evidence for scope, configuration, permissions, runtime state, diagnostics, and related resources before a production change.
Useful for
- List or show Azure resources and related configuration for ForEach activity.
- Capture read-only evidence before changing identity, networking, triggers, capacity, policy, deployment, or automation settings.
- Compare Azure metrics, logs, run history, deployment operations, and application evidence during production incidents.
Before you run a command
- Confirm the tenant, subscription, resource group, resource names, environment, and time window are the intended scope.
- Run read-only list, show, metrics, operation, or query commands before any create, update, delete, start, stop, policy, or deployment change.
- Get approval for mutating commands because configuration changes can expose data, break workflows, increase cost, or alter compliance evidence.
What the output tells you
- Resource IDs, enabled state, configuration values, identity settings, network posture, and ownership metadata show the current design.
- Metrics, logs, run history, or deployment operations show whether the platform behaved as expected during the reviewed time window.
- Application and downstream evidence shows whether the issue is Azure configuration, permissions, client behavior, data readiness, or business processing.
Mapped commands
Some evidence is visible only in service logs, SDK behavior, deployment output, SQL metadata, portal configuration, or application telemetry; Azure CLI still validates surrounding resources and operational scope.
Architecture context
ForEach activity is a pipeline orchestration construct in Azure Data Factory and Synapse pipelines, not a compute engine by itself. I use it when a workflow must repeat child activities across files, tables, partitions, tenants, API pages, or parameter values. The architecture decision is about concurrency, dependency ordering, failure handling, idempotency, retry behavior, and how each iteration receives inputs. A small loop can be harmless, but a production loop can create hundreds of storage reads, database calls, copy activities, or external API requests at once. I review batch count, parallelism, logging, and downstream throttling before release. The best implementations make every iteration traceable so operators can rerun one failed unit without replaying the entire pipeline.
- Security
- Security for the ForEach activity starts with knowing who can edit pipelines, linked services, parameters, datasets, managed identity permissions, secrets, debug runs, and logs that may reveal source paths, table names, customer identifiers, or API payload metadata. Review items expression, child activities, sequential or parallel mode, batchCount, parameters, variable usage, retries, timeout, failure behavior, run history, and downstream throttling limits before approving production changes. Prefer managed identity and Microsoft Entra ID where the service supports it, keep secrets in approved vaults, scope roles narrowly, and protect diagnostics that may reveal sensitive names, payloads, or operational patterns. During audits, capture Activity Log entries, role assignments, network settings, diagnostic settings, and owner approvals so teams can prove access and behavior were intentional.
- Cost
- Cost for the ForEach activity is driven by number of loop items, child activity count, data movement volume, integration runtime hours, debug sessions, failed retries, parallel fan-out, logging retention, and duplicate processing from non-idempotent activities. The expensive mistake is not only Azure consumption; it is also duplicate processing, failed retries, audit cleanup, manual investigations, and unnecessary capacity caused by weak design evidence. Review whether the workload truly needs the selected tier, frequency, retention, diagnostics, network path, and automation pattern. Use tags, budgets, alerts, and recurring reviews so teams can explain why the current design exists and remove stale resources safely.
- Reliability
- Reliability for the ForEach activity depends on valid item collections, stable linked services, idempotent child activities, retry settings, bounded parallelism, integration runtime health, downstream throttling behavior, and monitoring that identifies the failing loop item. A healthy Azure resource can still fail the business workflow if downstream services, identities, triggers, clients, or data contracts are wrong. Test retries, failover assumptions, disabled states, stale configuration, private DNS problems, timeout behavior, and duplicate processing before relying on the design. Keep runbooks for first-response checks, known limits, owner escalation, and rollback so support teams can recover without guessing. This keeps ForEach activity review specific across architecture, security, operations, and incident response.
- Performance
- Performance for the ForEach activity depends on batchCount, sequential setting, source and sink throughput, child activity duration, integration runtime capacity, expression complexity, downstream API limits, file size distribution, and failure retry delays. Measure platform-side metrics and application-side completion metrics because fast service response does not always mean the business task finished. Use realistic data sizes, concurrency, filter patterns, region placement, authentication paths, and downstream limits in tests. When performance regresses, compare configuration changes, resource limits, client logs, diagnostic data, and workload timing before adding capacity or blaming one Azure service. This keeps ForEach activity review specific across architecture, security, operations, and incident response.
- Operations
- Operations for the ForEach activity require named owners, documented resource IDs, expected behavior, diagnostic settings, and first-response checks. Before a change, capture read-only CLI output, portal screenshots when useful, deployment history, and relevant application configuration. During incidents, avoid changing several settings at once. Compare service metrics, logs, run history, identity evidence, network state, and downstream health in the same time window. Keep release notes clear enough for support teams to verify current behavior quickly. This keeps ForEach activity review specific across architecture, security, operations, and incident response. This keeps ForEach activity review specific across architecture, security, operations, and incident response.
Common mistakes
- Treating ForEach activity as a label instead of checking the exact resource scope, live configuration, owner, and dependencies.
- Changing several settings at once without saving read-only evidence, rollback instructions, and the expected metric change.
- Assuming the Azure resource succeeded means the end-to-end business workflow completed correctly and safely.