Data Factory is the orchestrator underneath most Azure data estates. Built the usual way, it becomes one pipeline per source and a maintenance burden that grows with every request. Built as a metadata-driven framework, adding a new source is a config change — no new pipeline, no redeploy, no regression testing of the feeds that already work.
Overview
One pipeline, every source
The pattern is a master pipeline that reads a metadata config and a single parameterised child pipeline that serves every object in it. Object names, schemas, load type, column mappings and watermarks live in configuration, not in code. It costs more to design and repays from the second source onward — and keeps repaying with every one after that.
What we deliver
What separates a framework from a pile of pipelines
Most of the difficulty in ingestion is not moving rows. It is knowing what changed, coping with a source that will not hand over everything at once, and being certain a run that failed halfway did not quietly report success.
- Metadata-driven master/child pattern, so onboarding a source is a config drop
- Incremental loads bounded by a watermark, with full loads still available
- Pagination for sources that cap rows per response, terminating on a short page
- Watermarks committed only after the final page, so a partial run is retried not skipped
- Credentials resolved from Key Vault at runtime, never stored in the pipeline
- Medallion layering in ADLS Gen2 and promotion through Azure DevOps
Why it matters
When Data Factory, and when not
It is the right tool for orchestrating movement across an Azure estate, especially where sources are varied and governance matters. If the whole estate is already Fabric, Fabric pipelines remove a moving part; if the work is heavy transformation rather than movement, that belongs in Databricks or the warehouse rather than in ADF activities.
Workflow
How we work in it.
- 01
Assessment
- Source inventory: systems, owners, update cadence and access route
- Which sources support incremental extraction, and on what column
- Response limits, pagination behaviour and rate constraints
- Existing pipelines and what can be retired
- 02
Framework Design
- Metadata schema: object, target, load type, mapping, watermark
- Master and child pipeline split
- Linked services and datasets, fully parameterised
- Secrets, managed identity and network access
- 03
Build
- Landing to bronze, then merged onward on the business key
- Per-source page sizes tuned against real payloads
- Per-iteration logging, so a failure is diagnosable to the page
- Bundled and promoted through Dev, Test and Production
- 04
Operate
- Run monitoring with row counts per source and per page
- Retries on transient failures, alerts on the rest
- Onboarding new sources by configuration
- Cost review and schedule tuning
Related in Integration & pipelines
