Data Factory

Understand the concept

In plain English

Data Factory is Azure’s orchestration service for moving and transforming data between systems. It lets data engineers build pipelines that copy files, call transformations, run notebooks, trigger stored procedures, and schedule workflows. In plain English, it is the traffic controller for data movement: it knows when data should be picked up, where it should land, which transformation should run, and whether the work succeeded. It is especially useful when data lives across cloud services, on-premises systems, SaaS platforms, and analytics stores.

Why it matters

Data Factory matters because data pipelines fail in ways that business users feel immediately: dashboards miss deadlines, downstream models use stale data, compliance exports arrive late, and operations teams cannot tell where the break occurred. A well-designed factory turns scattered data movement into observable workflows with parameters, retries, triggers, and ownership. It matters for migration because old batch jobs often need hybrid connectivity while systems move to Azure. It matters for governance because connections, secrets, lineage expectations, and operational logs need structure. Without Data Factory discipline, teams often recreate fragile scripts that are hard to monitor, secure, or hand over.

Technical context

Technically, Data Factory sits in the integration and data-platform control plane. A factory contains linked services, datasets, pipelines, activities, triggers, integration runtimes, parameters, variables, and monitoring records. It connects to storage accounts, databases, SaaS connectors, REST endpoints, Synapse, Databricks, Azure Functions, Key Vault, and on-premises systems through self-hosted integration runtime. It orchestrates work rather than storing most business data itself. Architecture decisions include identity, private endpoints, managed virtual networks, Git integration, trigger design, retry behavior, and activity monitoring.

Exam context

What to know

operator field manual with architecture, governance, CLI, and case-study depth

Compare with

Where it is used

Where you see it

1. In the authoring canvas, Data Factory appears as pipelines, activities, linked services, datasets, triggers, parameters, variables, and integration runtimes. during development and review. with owners.
2. In monitoring views, operators see pipeline runs, activity durations, trigger windows, retry attempts, failed connectors, error messages, and integration runtime health. during SLA checks. each morning.
3. In deployment pipelines, Data Factory appears as ARM or Bicep templates, factory JSON, Git branches, publish artifacts, and environment parameter files. during controlled promotion between environments.

Common situations

• Orchestrate hybrid ETL or ELT pipelines that move data from on-premises systems into Azure analytics stores.
• Schedule and monitor recurring copy, transformation, validation, and publishing workflows tied to business reporting deadlines.
• Trigger Databricks, Synapse, stored procedure, Function, or REST activities as part of one governed data workflow.
• Use self-hosted integration runtime when private or on-premises sources cannot be reached directly from Azure.
• Create parameterized pipelines that can be deployed across development, test, and production with different connections and schedules.

Illustrative Azure scenarios

These examples show how the concept can affect design and operations. They are illustrative scenarios, not customer claims.

Azure CLI

I use Azure CLI for Data Factory because pipeline operations need automation and evidence beyond the design canvas. After years of running data platforms, I want to create or inspect factories, list pipelines and triggers, start runs, query pipeline-run status, and export configuration during incidents. The portal is excellent for visual authoring, but CLI is better for repeatable environment setup, deployment checks, and runbook diagnostics. It also helps when a dashboard is late: I can query the exact run, activity state, trigger window, and failure message without clicking through every blade under pressure. That evidence keeps late-night support focused on facts.

az datafactory factory list --resource-group <rg> --output table

az datafactory pipeline list --resource-group <rg> --factory-name <factory-name> --output table

az datafactory trigger list --resource-group <rg> --factory-name <factory-name> --output table

az datafactory pipeline create-run --resource-group <rg> --factory-name <factory-name> --name <pipeline-name> --parameters @params.json

az datafactory pipeline-run query-by-factory --resource-group <rg> --factory-name <factory-name> --last-updated-after <utc> --last-updated-before <utc>

az datafactory trigger stop --resource-group <rg> --factory-name <factory-name> --name <trigger-name>

Architecture context

Architecturally, Data Factory is the orchestration layer that coordinates data movement and transformation across the platform. I avoid putting heavy business logic directly into copy orchestration when a database, Databricks job, Spark pool, or stored procedure is the right execution engine. The factory should know dependencies, schedules, parameters, credentials, retries, and monitoring paths. Strong designs separate development and production factories, use Git integration for collaboration, store secrets in Key Vault, isolate networks with private endpoints when needed, and make integration runtime placement explicit. A pipeline should be readable enough that another engineer can support it at 2 a.m. Document which transformations belong outside the factory to avoid hidden coupling.

Security

Security in Data Factory centers on connection credentials, managed identity, integration runtime placement, network exposure, and who can publish pipeline changes. Use Key Vault for secrets, managed identities where supported, private endpoints or managed virtual networks when data sources must stay private, and least-privilege roles for authors and operators. Self-hosted integration runtimes should be patched and placed carefully because they bridge on-premises or private networks to Azure workflows. Monitor linked service changes and trigger modifications. A compromised factory can move sensitive data, call external endpoints, or expose credentials through careless logs. Review debug output because sampled data can contain sensitive fields.

Cost

Data Factory cost comes from orchestration activity runs, data movement, data flow execution, integration runtime usage, connector behavior, and downstream compute it triggers. A pipeline that runs too often, copies unchanged data, or leaves mapping data flow clusters active can become expensive. Self-hosted integration runtimes add VM or server cost outside the service bill. FinOps reviews should connect pipeline runs to business value, watch failure retries, and eliminate duplicate movement between storage accounts. The cheapest pipeline is not always best; a reliable incremental load may cost more per run but reduce warehouse, support, and reprocessing cost. Alert on repeated failures because retries can waste money silently.

Reliability

Reliability depends on pipeline design, trigger windows, retry policy, idempotent activities, dependency handling, and clear failure recovery. Data Factory can orchestrate complex workflows, but it cannot fix non-idempotent loads, missing checkpoints, poor file naming, or secrets that expire without alerting. Production pipelines need activity timeouts, retries, alerts, rerun procedures, and backfill logic. Integration runtime health matters because a failed self-hosted node can block hybrid movement. Operators should design for partial failure: one bad source file should not corrupt every downstream table, and a rerun should not duplicate business records. Backfill tests should run before executives depend on dashboard freshness. Document owners.

Performance

Performance depends on connector throughput, integration runtime placement, source throttling, parallelism, file sizes, partitioning, network path, and the transformation engine being called. Data Factory can move large data volumes, but a poorly placed self-hosted runtime, tiny files, serial copy activities, or overloaded source database can dominate runtime. Mapping data flows need appropriate compute size and startup awareness. Operators should measure activity duration, rows copied, throughput, queue time, retry counts, and downstream job duration. Before scaling, check whether the bottleneck is source, network, sink, integration runtime, or transformation logic. Baseline normal runs so unusual delays stand out quickly. Track baselines. Continuously.

Operations

Operators monitor Data Factory through pipeline runs, activity runs, trigger runs, integration runtime status, diagnostic logs, alerts, and deployment history. Common jobs include restarting failed runs, checking parameter values, validating linked services, rotating credentials, pausing triggers during maintenance, and proving whether upstream or downstream systems caused a delay. Good runbooks identify the business owner, source, destination, expected schedule, retry policy, and backfill process for each critical pipeline. Operators also review long-running activities and connector failures because they often signal source throttling, schema drift, network changes, or expired secrets. Keep failed-run examples in training so support teams recognize patterns. Review owners.

Common mistakes

• Rerunning non-idempotent copy activities without checking whether target rows, files, or partitions already exist.
• Keeping secrets inside linked service JSON instead of using Key Vault and managed identities where possible.
• Scheduling pipelines without alerts or business owners, leaving reporting teams to discover failures manually.
• Using one self-hosted integration runtime for unrelated workloads until maintenance or capacity issues create a broad outage.