Polars Integration

Polars (opens in a new tab) is a fast DataFrame library. With output="polars" / "polars_lazy", the SDK still loads data via intapp-rest-client; Polars is the in-memory (or Lazy) representation on top.

💡

Polars support requires the polars extra:

pip: pip install dealcloud-sdk[polars]
uv: uv add dealcloud-sdk[polars]
poetry: poetry add dealcloud-sdk -E polars

When to Use Polars

Use Case	Recommendation
Small datasets (<10k rows)	pandas or list
Large datasets (>100k rows)	polars
Memory-constrained environments	polars
Existing pandas workflow	pandas
Maximum performance	polars
Query optimization (lazy eval)	polars_lazy

Basic Usage

import polars as pl
 
# Read as Polars DataFrame
df = dc.read_data("Company", output="polars")
 
# Fast Polars operations
filtered = df.filter(pl.col("Revenue") > 1000000)
sorted_df = df.sort("Revenue", descending=True)
 
# Group and aggregate
by_industry = df.group_by("Industry").agg([
    pl.col("Revenue").sum().alias("TotalRevenue"),
    pl.col("Name").count().alias("Count"),
])
 
# Reference/choice/user fields can be resolved to names on the Polars path too
df_named = dc.read_data("Company", output="polars", resolve="name")

Performance Comparison

import time
 
# Benchmark: Reading 500,000 rows
 
# pandas
start = time.time()
df_pandas = dc.read_data("LargeObject", output="pandas")
print(f"pandas: {time.time() - start:.2f}s")  # ~45s
 
# polars  
start = time.time()
df_polars = dc.read_data("LargeObject", output="polars")
print(f"polars: {time.time() - start:.2f}s")  # ~8s (5x faster)

Operation	pandas	Polars	Speedup
DataFrame construction	Slow	Fast	10-50x
Memory usage	High	Low	~50% less
Multi-threading	Limited	Native	Auto

LazyFrame for Query Optimization

LazyFrame defers execution until .collect() is called, allowing Polars to optimize the entire query:

import polars as pl
 
# Get LazyFrame
lf = dc.read_data("Transaction", output="polars_lazy")
 
# Build complex query (lazy - nothing executed yet)
result = (
    lf
    .filter(pl.col("Date") > "2025-01-01")
    .filter(pl.col("Amount") > 0)
    .group_by("Company")
    .agg([
        pl.col("Amount").sum().alias("TotalAmount"),
        pl.col("Amount").count().alias("TransactionCount"),
        pl.col("Amount").mean().alias("AvgAmount"),
    ])
    .sort("TotalAmount", descending=True)
    .limit(100)
)
 
# Polars optimizes: pushes filters, reorders operations
df = result.collect()

Writing with Polars DataFrames

Polars DataFrames are accepted as input for all write operations, and can also be returned as output:

Polars Input

import polars as pl
 
# Create Polars DataFrame
companies = pl.DataFrame({
    "Name": ["Acme Corp", "Beta Inc", "Gamma LLC"],
    "Revenue": [1000000, 500000, 750000],
    "Industry": ["Tech", "Finance", "Tech"],
})
 
# Insert from Polars
dc.insert_data("Company", companies)
 
# Update from Polars (with EntryIds)
updates = pl.DataFrame({
    "EntryId": [123, 456],
    "Revenue": [1200000, 600000],
})
dc.update_data("Company", updates)
 
# Upsert from Polars
dc.upsert_data("Company", companies)

Polars Output

Write methods now support output="polars" to return results as Polars DataFrames:

import polars as pl
 
# Insert and get Polars DataFrame result
companies = pl.DataFrame({
    "Name": ["Acme Corp", "Beta Inc"],
    "Revenue": [1000000, 500000],
})
 
result = dc.insert_data("Company", companies, output="polars")
# result is pl.DataFrame with EntryIds populated
 
# Update and get Polars result
updates = pl.DataFrame({"EntryId": [123, 456], "Revenue": [1500000, 600000]})
result = dc.update_data("Company", updates, output="polars")
 
# Upsert and get Polars result
result = dc.upsert_data("Company", companies, output="polars")

Delta sync

sync_delta passes output through to the underlying read, so output="polars" / "polars_lazy" populate result.modified_data as Polars when there are modified rows.

Cell Operations with DataFrames

write_cells() also accepts Polars and pandas DataFrames:

import polars as pl
 
# Write cells from Polars DataFrame
cell_updates = pl.DataFrame({
    "EntryId": [123, 456, 789],
    "Status": [[101], [102], [103]],  # Multi-select as lists
})
 
dc.write_cells("Company", cell_updates, mode="update")

Typed Methods with Polars

Use typed_read_data_polars() for model-documented fields with Polars performance:

from pydantic import BaseModel
from typing import Optional
 
class Company(BaseModel):
    EntryId: int
    Name: str
    Revenue: Optional[float] = None
    Industry: Optional[str] = None
 
# Get Polars DataFrame with model-defined fields
df = dc.typed_read_data_polars(Company, object_id="Company")
 
# Model documents expected fields, but returns Polars DataFrame
result = df.filter(pl.col("Revenue") > 500000)
 
# LazyFrame variant
lf = dc.typed_read_data_polars(Company, object_id="Company", lazy=True)

⚠️

typed_read_data_polars() does NOT validate data against the Pydantic model. Use typed_read_data() if you need validation.

File utilities (`read_csv` / `read_file`)

The SDK file helpers live in dealcloud_sdk.utils.file_utils. They accept output="polars" (default remains "pandas"): files are still parsed with pandas (same encoding and Excel engines), then converted with polars.from_pandas() so behavior matches existing CSV/Excel handling.

from dealcloud_sdk.utils.file_utils import read_csv, read_file
 
pl_df = read_csv("import.csv", output="polars")
pl_df = read_file("data.xlsx", output="polars")

Streaming with Polars

read_data_streaming yields batches of list[dict], not Polars frames. Wrap each batch yourself:

import polars as pl
 
for batch in dc.read_data_streaming("Company", fields=["EntryId", "Name"]):
    df = pl.DataFrame(batch)
    # process df...

There is no separate read_data_streaming_polars() API. For lazy full-object reads that fit in memory, use read_data(..., output="polars_lazy") instead.

Excel export and profiling

export_to_excel, export_with_formatting, and append_to_excel accept Polars DataFrames (and dicts of sheets keyed by name). Export still uses pandas + openpyxl internally after to_pandas().

profile_dataframe accepts a Polars DataFrame and converts it with to_pandas() before profiling.

Converting Between Formats

import polars as pl
import pandas as pd
 
# Polars → pandas
df_polars = dc.read_data("Company", output="polars")
df_pandas = df_polars.to_pandas()
 
# pandas → Polars
df_polars = pl.from_pandas(df_pandas)
 
# Polars → list of dicts
rows = df_polars.to_dicts()
 
# list → Polars
df = pl.DataFrame(rows)

Streaming Large Datasets

For datasets larger than RAM, use LazyFrame with streaming:

# 10 million rows - doesn't fit in memory
lf = dc.read_data("HugeTable", output="polars_lazy")
 
# Process in streaming mode
result = (
    lf
    .filter(pl.col("Status") == "Active")
    .group_by("Region")
    .agg(pl.col("Value").sum())
    .collect(engine="streaming")  # Processes in chunks
)

Views with Polars

When reading views with Polars, include_nulls is automatically set to False:

# include_nulls=True is ignored for views
df = dc.read_data(view_id="My View", output="polars", include_nulls=True)
 
# Result: Only columns with actual data (not all possible fields)

💡

include_nulls with Views: The include_nulls parameter is automatically set to False for views to prevent excessive columns. This works the same for all output formats (pandas, polars, list). See Reading from Views for details.

Best Practices

Use Polars for large datasets - Anything over 100k rows benefits significantly
Use LazyFrame for complex queries - Lets Polars optimize the entire query
Avoid frequent conversions - Stay in one format throughout your pipeline
Use streaming for huge datasets - .collect(engine="streaming") for larger-than-RAM
Leverage native Polars functions - They're faster than .apply() with Python functions

Next Steps

Basic Read Operations - Core read_data() usage
Streaming - Memory-efficient processing
Typed Methods - Pydantic model integration

Reference Fields Insert Data