Streaming Reads

Streaming reads fetch data in batches without loading the full object into memory. Use read_data_streaming() / aread_data_streaming() when datasets are large or when you process incrementally.

Why streaming?

Approach	Memory	Best for
`read_data()`	Holds full result	Analysis, smaller tables
`read_data_streaming()`	Bounded by batch size	Large exports, ETL, collecting entry IDs for batched `delete_data`

💡

Each iteration is a list[dict] batch, not a single row. Loop over batch, or extend a list, depending on your workload.

Basic streaming

from dealcloud_sdk import DealCloud, DealCloudConfig
 
config = DealCloudConfig(
    siteUrl="yoursite.dealcloud.com",
    clientId=12345,
    clientSecret="your-secret",
)
dc = DealCloud.from_config_object(config)
 
for batch in dc.read_data_streaming("Company", fields=["EntryId", "CompanyName"]):
    for row in batch:
        process(row)

With fields and query

for batch in dc.read_data_streaming(
    "Company",
    fields=["CompanyName", "Revenue"],
    query="{Status: 'Active'}",
):
    for company in batch:
        print(company["CompanyName"], company.get("Revenue"))

Async streaming

aread_data_streaming() also yields batches:

import asyncio
from dealcloud_sdk import DealCloud, DealCloudConfig
 
async def run():
    config = DealCloudConfig(
        siteUrl="yoursite.dealcloud.com",
        clientId=12345,
        clientSecret="your-secret",
    )
    dc = DealCloud.from_config_object(config)
    async for batch in dc.aread_data_streaming("Company", fields=["EntryId"]):
        await handle_batch(batch)
 
# asyncio.run(run())

Parallel async work per batch

import asyncio
from dealcloud_sdk import DealCloud, DealCloudConfig
 
async def process_batch(batch):
    await asyncio.gather(*[process_one_row(row) for row in batch])
 
async def main():
    config = DealCloudConfig(
        siteUrl="yoursite.dealcloud.com",
        clientId=12345,
        clientSecret="your-secret",
    )
    dc = DealCloud.from_config_object(config)
    async for batch in dc.aread_data_streaming("Company"):
        await process_batch(batch)

Parallel page fetching

Pages are fetched concurrently by default and yielded in order. Concurrency is bounded by max_concurrency, which defaults to the client's read_concurrency (config concurrencyLimits.read, default 4 to stay within the API's ~5 requests/second limit). This applies to read_data() as well, since it reads through the same pipeline.

# Use the configured read_concurrency (default)
for batch in dc.read_data_streaming("Company"):
    process(batch)
 
# Or override per call (stay within the API rate limit)
for batch in dc.read_data_streaming("Company", max_concurrency=4):
    process(batch)
 
# Force strictly sequential fetching
for batch in dc.read_data_streaming("Company", max_concurrency=1):
    process(batch)

💡

Pagination stops when the API returns an empty page (there is no total-count call), so with max_concurrency > 1 a few speculative requests may be issued near the end of the data. Batches are always yielded in page order.

Views

For configured views, use read_view_streaming() / aread_view_streaming() (see Views).

Basic reads — read_data() when you want the full dataset
Performance — Batching and concurrency
Async operations — Async patterns

Query Syntax Typed Data (Pydantic)