# Paginated OData Fetch Loop (Python Generator)

> Source: https://netray.co/tools/paginated-odata-fetch-loop-python-generator
> Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems.
> Section: Free Tools

## Short answer

Build a Python loop that follows nextLink pages from an OData entity set and writes the result to CSV with a hard row cap.

## Category

developer

## Audience

- ERP developer
- Data engineer
- Integration developer

## Tool

**Kind:** template

**Output:** code

**Language:** python

**Fields:**

#### OData service root URL

**Key:** service_url

**Type:** text

**Default:** https://erp.example.com/odata/v4

#### Entity set name

**Key:** entity

**Type:** text

**Default:** SalesOrders

#### $select fields (comma separated)

**Key:** select

**Type:** text

**Default:** OrderNumber,CustomerId,OrderDate,Status

#### $filter expression

**Key:** filter

**Type:** text

**Default:** Status eq 'Open'

#### $orderby

**Key:** orderby

**Type:** text

**Default:** OrderNumber asc

#### Page size ($top)

**Key:** page_size

**Type:** number

**Default:** 500

#### Hard row cap

**Key:** max_rows

**Type:** number

**Default:** 50000

#### Environment variable for bearer token

**Key:** token_env

**Type:** text

**Default:** ERP_ODATA_TOKEN

#### Output CSV file

**Key:** outfile

**Type:** text

**Default:** sales_orders.csv

**Blocks:**

#### odata_export.py

**Body:** import csv
import os

import requests

SERVICE = "{{service_url}}"
ENTITY = "{{entity}}"
MAX_ROWS = {{max_rows}}
OUTFILE = "{{outfile}}"

params = {
    "$select": "{{select}}",
    "$filter": "{{filter}}",
    "$orderby": "{{orderby}}",
    "$top": "{{page_size}}",
}
headers = {
    "Authorization": "Bearer " + os.environ["{{token_env}}"],
    "Accept": "application/json",
}

session = requests.Session()
url = SERVICE + "/" + ENTITY
fields = [name.strip() for name in params["$select"].split(",")]
total = 0

with open(OUTFILE, "w", newline="", encoding="utf-8") as handle:
    writer = csv.DictWriter(handle, fieldnames=fields, extrasaction="ignore")
    writer.writeheader()
    first = True
    while url and total < MAX_ROWS:
        # The nextLink already carries all query options, so send params only once
        response = session.get(
            url, headers=headers, params=params if first else None, timeout=60
        )
        response.raise_for_status()
        payload = response.json()
        for row in payload.get("value", []):
            writer.writerow(row)
            total += 1
            if total >= MAX_ROWS:
                break
        url = payload.get("@odata.nextLink")
        first = False

print("Wrote " + str(total) + " rows to " + OUTFILE)
if url:
    print("Stopped at the row cap, more pages remain")

**Notes:** OData v4 returns @odata.nextLink. OData v2 services such as many SAP Gateway services use d.results and d.__next instead, and $skiptoken or $skip may be needed. Server-driven paging can override your $top. Verify query option support, case sensitivity of field names and filter syntax with your ERP, and test on a non-production system.

## Content

**Intro:** Pulling a large entity from an ERP over OData in one call usually fails or times out. This generator writes a Python script that requests a page, follows the server's next link until the data ends, and writes rows to CSV with a cap that protects your system.

**Sections:**

#### How to use it

**Body:** Enter the service root, entity set, fields, a filter and an order. Choose a modest page size and a hard row cap. Export the token into the environment variable and run the script on a test tenant first. Open the CSV to confirm column names. Narrow the filter before you raise the cap, because fewer rows and fewer fields are the easiest ways to reduce load on the ERP.

#### The method behind it

**Body:** OData v4 servers return @odata.nextLink when more results exist. That link already includes the query options, so the loop sends parameters only on the first request and then just follows the link. This is more reliable than computing $skip offsets, which can skip or repeat rows if data changes between pages. Stable ordering by a key helps when a service falls back to offsets.

**Bullets:**

- Use $select to return only the fields you need
- Filter on indexed or key fields where possible
- Keep page sizes moderate, a few hundred to a few thousand
- Cap total rows during testing
- Read-only service account for extracts

#### ERP notes

**Body:** Dynamics 365 Finance and Operations, Business Central, SAP Gateway services and Epicor all expose OData with local differences: paging limits, cross-company flags, and how large filters are handled. Check the documented maximum page size and API capacity limits before scheduling this against production. For continuing sync, filter on a modified timestamp instead of re-reading everything.

#### Common mistakes

**Body:** Teams send the same parameters with every page request and end up in an endless loop or duplicated rows. Others skip $select and pull wide tables, forget URL encoding for filter values, or run the extract at month end when the ERP is busiest. Always log the row count and stop at a cap.

**Cta:** Netray connects AI agents and analytics to ERP data with careful, read-only extraction. If your extracts are slow or risky, we can help redesign them.

**Faq:**

#### Why follow @odata.nextLink instead of using $skip?

**Answer:** The next link is created by the server and encodes where the next page starts, which stays correct even if rows change between requests. Offset paging with $skip can miss or repeat records on live data and gets slower on large offsets.

#### Does this work with SAP OData v2?

**Answer:** Not directly. Many v2 services return results under d.results and the next URL under d.__next, and use $skiptoken or $skip. Adjust the two lines that read the rows and the next link. The overall loop is the same.

#### How big should my page size be?

**Answer:** A few hundred to a few thousand rows is typical. Smaller pages mean more requests, larger pages risk timeouts and memory use. Servers often enforce their own maximum, so test and follow what the service returns.

#### Can I run this against production?

**Answer:** Yes if you use a read-only account, a narrow filter, a row cap, and an off-peak schedule, and you confirm API capacity limits. Test on a non-production tenant first and watch ERP performance during the first run.

## Search terms

odata pagination python, odata nextLink loop, odata $top $skip python, erp odata extract to csv, d365 odata python script, sap odata python paging, odata $select $filter python, python requests odata

---
Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json