ERP Incremental Extract Watermark SQL Generator
Full table dumps get slower every month. An incremental extract pulls only rows that changed since the last run, using a watermark stored in a control table. This generator writes the T-SQL for that pattern with a batch size and a safety overlap.
Fill in
A datetime column that changes on every insert and update.
Re-read this much time before the watermark to catch late-committed transactions.
1. One-time setup: watermark control table
IF OBJECT_ID('dbo.etl_watermark') IS NULL
BEGIN
CREATE TABLE dbo.etl_watermark (
source_table sysname NOT NULL PRIMARY KEY,
last_mod datetime2(3) NOT NULL,
last_run_utc datetime2(3) NOT NULL DEFAULT SYSUTCDATETIME()
);
END
IF NOT EXISTS (SELECT 1 FROM dbo.etl_watermark WHERE source_table = 'item_transactions')
INSERT INTO dbo.etl_watermark (source_table, last_mod)
VALUES ('item_transactions', '1900-01-01');2. Each run: read the next batch
DECLARE @from datetime2(3); DECLARE @to datetime2(3); SELECT @from = DATEADD(MINUTE, -5, last_mod) FROM dbo.etl_watermark WHERE source_table = 'item_transactions'; SELECT TOP (50000) trans_id, item, whse, qty, trans_type, RecordDate INTO #batch FROM dbo.item_transactions WHERE RecordDate > @from ORDER BY RecordDate, trans_id; SELECT @to = MAX(RecordDate) FROM #batch; -- Hand #batch to your loader here, then upsert on trans_id at the destination. SELECT * FROM #batch;
3. After a successful load: advance the watermark
UPDATE dbo.etl_watermark
SET last_mod = @to,
last_run_utc = SYSUTCDATETIME()
WHERE source_table = 'item_transactions'
AND @to IS NOT NULL
AND @to > last_mod;
DROP TABLE #batch;4. Reconciliation check
SELECT COUNT(*) AS source_rows, MAX(RecordDate) AS source_max_mod FROM dbo.item_transactions WHERE RecordDate > '1900-01-01'; -- Compare with the row count and max RecordDate at the destination.
Verify the modified column on your ERP version: many tables use a RecordDate or UpdatedDate that is set by triggers or the application, and some updates bypass it. A rowversion column is more reliable where available. Deletes are not captured by this pattern; use a soft-delete flag or change tracking. The overlap re-reads some rows, so the destination must upsert on the key. A batch ending exactly at a timestamp shared by more than the batch size can stall, so raise the batch size if the watermark stops advancing. Test on a copy and run read-only first.
Incremental pipeline design
Optional: we can review your ERP tables and propose a safe incremental pipeline design.
Optional. The tool and every export work without this. No spam.
How to use it
Fill in the schema, table, key column and the column that records the last change. List the columns to extract, set a batch size and an overlap in minutes. Copy the four blocks into your job. Run the setup once, then run the read and advance steps on a schedule. Your loader should upsert on the key at the destination so that the overlap does not create duplicates. Use the reconciliation block after each run or daily.
- Confirm the modified column really changes on updates
- Upsert at the destination on the primary key
- Keep the overlap larger than your longest write transaction
How the watermark works
The control table stores the highest modified timestamp that was loaded for each source table. Each run reads rows with a timestamp above the watermark minus a small overlap, ordered by timestamp then key, in a bounded batch. After a successful load the watermark moves forward to the maximum timestamp in the batch. If a run fails before the update, the watermark stays put and the same rows are read again, which makes the process safe to retry.
Why the overlap matters
A transaction that starts before the watermark but commits after it can write a row with an older timestamp than the rows you already loaded. Without an overlap, that row is missed forever. Re-reading the last few minutes catches it, at the cost of reading some rows twice. Choose an overlap longer than your longest typical write transaction, and rely on the key-based upsert to avoid duplicates.
Limits to know
This pattern cannot see hard deletes, and it depends on the modified column being reliably maintained. Some ERP tables do not update it for every change. In those cases use rowversion, SQL Server change tracking or change data capture instead, and run a periodic full reconciliation of counts and sums to catch drift.
Frequently Asked Questions
Why not just use a full reload each night?
A full reload is simple but grows with the table, and large ERP history tables can take hours. An incremental load touches only changed rows, so runtime and load on the ERP stay stable as data accumulates.
What if my table has no modified column?
Look for rowversion, a change tracking setup, or a monotonically increasing key for append-only tables. If none exist, an incremental pull is unreliable, and you may need triggers, change data capture or a periodic full compare.
How do I capture deleted rows?
Pull the key list periodically and compare it with the destination, or use change tracking or CDC, which report deletes. Some ERPs soft-delete with a status flag, which this pattern does capture as an update.
How big should the overlap be?
Larger than your longest write transaction. Five minutes suits many ERP tables, but a nightly interface that holds a transaction for 30 minutes needs more. The cost of a larger overlap is a few extra rows read, which the upsert handles.
If you want incremental ERP data feeding a warehouse or an on-prem AI layer, we can design and run the pipeline with you, read-only by default.
Related Tools
CSV Export File Size and Split Calculator
Estimate the size of an ERP CSV extract and how many files you need to stay under Excel's row limit or an upload size cap.
ERP OperationsERP Nightly Batch Job Parallelism and Runtime Calculator
Find how many parallel threads shorten your ERP nightly batch, and when contention starts to make it slower.
Developer ToolsSQL Query Performance Tuning Log Worksheet
An editable log of slow ERP queries with before and after timings that computes percent improvement and CPU hours saved per day.
EpicorEpicor REST API Paging and Throughput Calculator
Estimate how long an Epicor REST extract takes for a given page size, latency and concurrency, and find the sweet spot.
More Developer Tools tools
See all- SQL Server Index Size Estimator Calculator
- SQL Server Transaction Log Sizing Calculator
- SQL Server Page Compression Savings Calculator
- SAP Sales Order ABAP SELECT Generator (VBAK, VBAP, VBFA)
- SAP Sales Order Create BAPI ABAP Generator (BAPI_SALESORDER_CREATEFROMDAT2)
- SAP ABAP Open Items SELECT Generator
- SAP ACDOCA SQL Query Generator
- SAP QM Inspection Lot ABAP SELECT Generator
- SAP Quality Notification Pareto SQL Query Generator