Skip to article
Analytics Azure Data Factory

Incremental copy

Incremental copy is the Azure concept that controls how data platforms move changes without reprocessing every source row or file. Teams see it when working with data factory pipelines, copy activity. It is not a full reload, a one-time migration, a backup job, or a blind append into a lake; that distinction matters because bad assumptions create duplicate records, missed changes. Use the term when reviewing ownership, access, monitoring, cost, recovery, or performance. It keeps architects, operators, security reviewers, and support teams focused on the same resource, setting, or behavior.

Source: Microsoft Learn - Incrementally copy data in Azure Data Factory Reviewed 2026-05-15

Exam trap
Treating Incremental copy as a harmless label instead of checking the live resource, scope, owner, and dependencies.
Production check
Verify the intended scope, owner tags, diagnostics, and dependent resources before changing Incremental copy.
Practice this topic Microsoft Learn
Article details and learning context
Aliases
incremental load, delta copy, watermark copy, CDC copy, last modified copy
Difficulty
Intermediate
CLI mappings
5
Last verified
2026-05-15
Compare related concepts Search related resources Browse the glossary

Understand the concept

Why it matters

Incremental copy matters because it turns an architecture choice into day-to-day workload behavior. If the team misunderstands it, the failure usually appears as duplicate records, missed changes, stale reports before anyone notices the documentation gap. The term also affects security, reliability, operations, cost, and performance because one setting can influence access, recovery, automation, user experience, and budget. Naming it precisely helps engineers compare portal settings, CLI output, infrastructure-as-code, monitoring data, and incident notes without guessing. It also gives reviewers a practical checklist: where is it configured, who owns it, what depends on it, what evidence proves it works, and how rollback happens.

Official wording and source

Incremental copy is the Azure concept that controls how data platforms move changes without reprocessing every source row or file. Microsoft Learn places it in Incrementally copy data in Azure Data Factory; operators confirm scope, configuration, dependencies, and production impact.

Open Microsoft Learn

Technical context

Technically, Incremental copy sits in Data Factory pipelines, Copy activity, dataset parameters, watermark tables. Key fields include source query, watermark column, previous watermark, current watermark. Operators verify it with pipeline run output, copy activity metrics, rowsRead and rowsCopied, watermark table values. In production reviews, connect the term to resource scope, identity, network path, diagnostics, cost ownership, and rollback. Confirm subscription, resource group, service tier, dependent workload, and current Azure evidence before changing it. Use current Azure evidence before changing production settings.

Exam context

Compare with

Change data capture in Data Factory Data Factory debug run Execute Pipeline activity

Where it is used

Where you see it

  1. In the Azure portal, Incremental copy appears near data factory pipelines, copy activity, where owners review configuration, health, access, and dependent workload impact before safe production changes.
  2. In CLI or REST output, Incremental copy shows up through pipeline run output, copy activity metrics and related fields that confirm live Azure state during audits, releases, and incidents.
  3. In incident reviews, Incremental copy is discussed when users report duplicate records, and engineers compare logs, metrics, ownership, dependencies, recent changes, support impact, and deployment evidence together.

Common situations

  • Design and review Incremental copy as part of a production Azure workload.
  • Troubleshoot incidents where Incremental copy affects user-visible behavior or operator evidence.
  • Document ownership, rollback, monitoring, and cost impact for Incremental copy during governance reviews.

Illustrative Azure scenarios

These examples show how the concept can affect design and operations. They are illustrative scenarios, not customer claims.

Scenario 01 Incremental copy in action for watermark-based finance loads Scenario, objectives, solution, measured impact, and takeaway.
Scenario

Harborline Bank, a financial services organization, needed to replace nightly full-table loads that delayed risk dashboards and overloaded an Azure SQL source. The team had to improve the design without disrupting existing users or weakening governance.

Goals
  • Use Incremental copy to solve the immediate workload problem
  • Keep security and compliance evidence available for review
  • Reduce manual support effort during operations
  • Measure results with production telemetry and owner signoff
Approach using Incremental copy

Architects treated Incremental copy as a production control point rather than a background detail. They reviewed the current Azure resources, confirmed owners, and documented how the term connected to identity, networking, monitoring, cost, and rollback. Engineers implemented watermark columns, Copy activity parameters, a control table, private endpoints, and activity-run monitoring, then validated the change with read-only CLI checks and portal evidence. The rollout used a pilot scope first, with diagnostic logging enabled before wider release. Support teams received a runbook explaining expected output, common failure modes, and the safest rollback path. Security reviewers checked access boundaries and data-handling assumptions before the change moved to production.

Potential outcomes
  • cut source reads by 82 percent
  • moved dashboard freshness from six hours to forty minutes
  • reduced failed overnight loads by 38 percent
  • kept every reload tied to a pipeline run ID
What to learn

Incremental copy is valuable when teams connect the Azure setting to measurable security, reliability, operational, cost, and performance outcomes.

Scenario 02 Incremental copy in action for CDC retail inventory ingestion Scenario, objectives, solution, measured impact, and takeaway.
Scenario

Fabrikam Outdoor, a retail organization, needed to ingest store inventory changes into a lake without copying unchanged product rows during every promotion cycle. The team had to improve the design without disrupting existing users or weakening governance.

Goals
  • Use Incremental copy to solve the immediate workload problem
  • Keep security and compliance evidence available for review
  • Reduce manual support effort during operations
  • Measure results with production telemetry and owner signoff
Approach using Incremental copy

Architects treated Incremental copy as a production control point rather than a background detail. They reviewed the current Azure resources, confirmed owners, and documented how the term connected to identity, networking, monitoring, cost, and rollback. Engineers implemented SQL CDC, parameterized pipelines, ADLS bronze folders, retry policies, and owner-tagged triggers, then validated the change with read-only CLI checks and portal evidence. The rollout used a pilot scope first, with diagnostic logging enabled before wider release. Support teams received a runbook explaining expected output, common failure modes, and the safest rollback path. Security reviewers checked access boundaries and data-handling assumptions before the change moved to production.

Potential outcomes
  • reduced ingestion compute by 47 percent
  • kept inventory lag under fifteen minutes
  • lowered duplicate product records by 91 percent
  • gave merchandisers reliable change visibility
What to learn

Incremental copy is valuable when teams connect the Azure setting to measurable security, reliability, operational, cost, and performance outcomes.

Scenario 03 Incremental copy in action for last-modified claims files Scenario, objectives, solution, measured impact, and takeaway.
Scenario

Northbridge Assurance, a insurance organization, needed to copy only new and updated claim documents from Blob storage into a governed lake zone. The team had to improve the design without disrupting existing users or weakening governance.

Goals
  • Use Incremental copy to solve the immediate workload problem
  • Keep security and compliance evidence available for review
  • Reduce manual support effort during operations
  • Measure results with production telemetry and owner signoff
Approach using Incremental copy

Architects treated Incremental copy as a production control point rather than a background detail. They reviewed the current Azure resources, confirmed owners, and documented how the term connected to identity, networking, monitoring, cost, and rollback. Engineers implemented LastModified filtering, Get Metadata checks, Copy activity output review, and rerun-safe sink paths, then validated the change with read-only CLI checks and portal evidence. The rollout used a pilot scope first, with diagnostic logging enabled before wider release. Support teams received a runbook explaining expected output, common failure modes, and the safest rollback path. Security reviewers checked access boundaries and data-handling assumptions before the change moved to production.

Potential outcomes
  • shortened claim document refresh from four hours to thirty minutes
  • avoided 14 TB of repeated monthly transfers
  • improved audit evidence for each batch
  • made failed file windows easy to replay
What to learn

Incremental copy is valuable when teams connect the Azure setting to measurable security, reliability, operational, cost, and performance outcomes.

Azure CLI

CLI checks are useful for Incremental copy because they capture live Azure state, reduce guesswork, and separate safe inspection from approved changes.

Useful for

  • Confirm the live Azure resource or configuration related to Incremental copy before approving a production change.
  • Capture read-only evidence for Incremental copy during incident response, audit review, or release validation.
  • Compare CLI output with infrastructure-as-code, portal settings, and runbook expectations for Incremental copy.

Before you run a command

  • Confirm tenant, subscription, resource group, service name, and environment before trusting command output.
  • Run list or show commands first, then save evidence before any create, update, delete, restore, or deploy action.
  • Check whether the command exposes secrets, customer data, training examples, file paths, keys, or private endpoints.
  • Have an approved rollback path and owner contact ready before changing production configuration.

What the output tells you

  • Whether the expected Azure resource exists and whether Incremental copy is configured at the intended scope.
  • Which names, IDs, locations, states, tiers, policies, identities, and dependent resources are active right now.
  • Whether live Azure state differs from the design document, deployment template, release ticket, or support runbook.
  • Which metric, log query, portal page, or application test should be checked before closing the issue.

Mapped commands

Incremental copy operational checks

direct 
az datafactory pipeline-run query-by-factory --factory-name <factory-name> --resource-group <resource-group> --last-updated-after <utc-start> --last-updated-before <utc-end>
az datafactory pipeline-rundiscoverAnalytics
read-only Microsoft Learn 
az datafactory pipeline-run show --factory-name <factory-name> --resource-group <resource-group> --run-id <run-id>
az datafactory pipeline-rundiscoverAnalytics
read-only Microsoft Learn 
az datafactory activity-run query-by-pipeline-run --factory-name <factory-name> --resource-group <resource-group> --run-id <run-id> --last-updated-after <utc-start> --last-updated-before <utc-end>
az datafactory activity-rundiscoverAnalytics
read-only Microsoft Learn 
az datafactory trigger-run query-by-factory --factory-name <factory-name> --resource-group <resource-group> --last-updated-after <utc-start> --last-updated-before <utc-end>
az datafactory trigger-rundiscoverAnalytics
read-only Microsoft Learn 
az datafactory pipeline show --factory-name <factory-name> --resource-group <resource-group> --name <pipeline-name>
az datafactory pipelinediscoverAnalytics
read-only Microsoft Learn

Architecture context

Technically, Incremental copy sits in Data Factory pipelines, Copy activity, dataset parameters, watermark tables. Key fields include source query, watermark column, previous watermark, current watermark. Operators verify it with pipeline run output, copy activity metrics, rowsRead and rowsCopied, watermark table values. In production reviews, connect the term to resource scope, identity, network path, diagnostics, cost ownership, and rollback. Confirm subscription, resource group, service tier, dependent workload, and current Azure evidence before changing it.

Security
Security for Incremental copy starts with source credentials, managed identity, private endpoints, least-privilege reads, sink write permissions. Review who can read, create, update, delete, restore, deploy, or invoke the related resource, and verify that privileged changes create audit evidence. Prefer Microsoft Entra ID, managed identities, private endpoints, key rotation, customer-managed keys, and policy controls where the service supports them. Keep secrets, credentials, personal data, and regulated content out of scripts and examples unless the data-handling design explicitly allows it. During approval, check tenant boundaries, network exposure, diagnostic logs, and break-glass procedures so a configuration mistake does not become an incident.
Cost
Cost for Incremental copy is driven by data scanned, files copied, integration runtime hours, trigger frequency, duplicate processing. The common mistake is treating the term as free because it is a setting, schema choice, job, or child resource instead of a cost influence. Check whether charges come from storage, requests, tokens, replicas, retention, backups, training, data transfer, diagnostics, or engineer time spent recovering from bad configuration. Use tags, budgets, Azure Cost Management, and owner reviews to connect usage to a workload. When reducing cost, confirm the change will not remove recovery evidence, security controls, or needed performance headroom. Confirm the owner understands the tradeoff before resizing, retaining, or redeploying.
Reliability
Reliability for Incremental copy depends on watermark accuracy, source change tracking, trigger cadence, retry behavior, idempotent sinks. A resource can exist and still fail the business workflow when permissions, network paths, limits, schema settings, or downstream services are wrong. Define the health signal before production use, then test the expected failure mode with a controlled change. Monitor platform metrics, application traces, deployment history, and user symptoms in the same time window during incidents. Recovery plans should include owner contact, safe rollback, validation queries, and customer-impact checks, not just proof that the Azure resource exists. Confirm this behavior is tested before the workload depends on it.
Performance
Performance for Incremental copy depends on source query selectivity, copy parallelism, file size, partition pruning, integration runtime capacity. Measure the real workload instead of assuming the default configuration is enough. Look at latency, throughput, concurrency, request size, metadata operations, query complexity, token counts, or recovery duration depending on the service. Compare production metrics with load tests and with the limits of the selected tier or model. Tuning should be incremental and reversible, because a change that improves one path can hurt another. Always verify user-facing behavior after configuration, schema, deployment, or data-layout changes. Capture before-and-after metrics so tuning is based on evidence rather than assumptions.
Operations
Operations for Incremental copy require pipeline run reviews, activity output checks, trigger history, watermark audits, rerun windows. Treat the term as something support teams must inspect quickly, not only as a design-time concept. Keep a runbook with portal locations, CLI commands, expected output, known dependencies, approval rules, and rollback steps. Review it during releases, migrations, incidents, access changes, and cost investigations. Good operations practice also means tagging owners, enabling diagnostics, storing evidence from read-only checks, and documenting exceptions. When the term changes, update handoff notes so future operators know what normal looks like. Keep the same evidence available to the next on-call engineer.

Common mistakes

  • Treating Incremental copy as a harmless label instead of checking the live resource, scope, owner, and dependencies.
  • Running a mutating command in the wrong subscription, resource group, account, service, index, share, or deployment.
  • Assuming a successful deployment proves the feature works without checking logs, metrics, access, and rollback evidence.
  • Ignoring cost, retention, quotas, network exposure, or data classification until an incident forces emergency cleanup.