SAP holds the operational heart of a business and is famously hard to get data out of. There is usually no direct database access, the OData services cap how much they will return in one response, and the tables are large enough that a nightly full extract is not an option. Every design decision here exists to work within those limits rather than against them.
Overview
The cap is the whole problem
Ask an SAP OData service for a large result set and you will not get it. You will get a capped response, an error, or — worst of all — a truncated response that the pipeline reports as a success. A design that ignores this does not fail loudly; it produces confidently incomplete data that nobody questions until a number looks wrong.
What we deliver
How this is built
Two mechanisms compose. A watermark decides what is in scope; a bounded loop decides how that scope is carried across. Neither alone is sufficient — a filter without paging still overruns the cap on a heavy day, and paging without a filter walks the entire entity every night.
- Extraction through the supported OData service layer, not around it
- Incremental loads bounded by a change-date watermark
- A pagination loop that requests a page at a time and stops when the source runs dry
- Termination on a short page or an absent next link, never on a fixed iteration count
- Per-page row counts logged, so silent truncation is visible rather than assumed
- Merge on the business key, which makes a repeated row harmless
Why it matters
Integration, not implementation
We are not an SAP implementation partner and do not configure your ERP. We move its data into a warehouse the business can report on, without adding analytical load to a live operational system, and without anyone needing an SAP licence to look at a number.
Workflow
How we work in it.
- 01
Scoping
- Which entities reporting needs, and which fields within them
- Available OData services and authorisation model
- Response caps, page sizes and gateway load tolerance
- Change-date columns available per entity
- 02
Design
- Metadata configuration per entity, including page size
- Watermark strategy and full-load fallback
- Field projection, so only required fields cross the wire
- Target schema and business keys
- 03
Build
- Bounded pagination with a safety ceiling
- Each page landed before the next is requested
- Watermark advanced only after the final page commits
- Deduplication on the business key at the cleansing layer
- 04
Operate
- Iteration counts monitored, not just run-level totals
- Retries on transient gateway failures
- Page sizes tuned per entity against real payloads
- New entities onboarded by configuration
Related in Integration & pipelines
