May 14, 2026
Blog May 14, 2026 Infor Data Extraction Infor data extraction has three real paths: Streaming Pipelines, the Data Lake API,...
Infor Streaming Pipelines cap most contracts at around 165,000 transactions per day per pipeline. For day-to-day operational data this is plenty. For initial loads of large tables this is a wall. A million-row table takes six days. A five-million-row table takes a month. The fix is to bypass Streaming Pipelines for the initial load and use the Data Lake API with SqlBulkCopy instead, which has no equivalent throttle and moves millions of rows in minutes.
The 165K transaction limit is the daily cap most Infor Streaming Pipelines contracts apply to each pipeline. Once a pipeline hits 165,000 transactions in a day, it throttles. The pipeline doesn’t stop, it just falls behind, and the backlog clears against the same daily allowance the next day.
The exact number varies by contract. Some customers have higher caps. Some have lower. 165K is common enough that it’s the number most Infor forum posts mention.
The throttle counts each row movement as one transaction. A bulk update affecting 10,000 rows counts as 10,000 transactions, not one.
An initial load copies every existing row from a source table to the destination. The throttle applies the same way it does to ongoing operations: each row equals one transaction.
Here’s how long initial loads take at the 165K daily cap:
|
Table size
|
Days to load at 165K/day
|
Real-world impact
|
|---|---|---|
|
100,000 rows
|
Less than 1 day
|
Fine
|
|
500,000 rows
|
~3 days
|
Workable
|
|
1 Million rows
|
~6 days
|
Project Delay
|
|
5 Million rows
|
~30 days
|
Unworkable
|
|
20 million rows
|
~120 days
|
Forget it
|
|
Schema
|
Generic datatypes
|
You define
|
|
Setup effort
|
Low
|
Medium-high
|
|
Multi-tenant SaaS
|
Yes
|
Yes
|
|
CSD / ISM (on-prem)
|
Yes
|
Yes
|
Tables with millions of rows are normal in Infor CSD and ISM. Order history, transaction ledgers, inventory movements, audit logs. The first table you try to load is usually one of these, and that’s when teams discover the throttle.
Yes, but it’s not cheap. Infor sells additional capacity for Streaming Pipelines. Pricing scales steeply. For most teams, the cost of buying enough capacity to do an initial load in a reasonable timeframe is more than the cost of building a Data Lake API integration once and using it forever.
Paid capacity also doesn’t address the monitoring and replay issues that come with Streaming Pipelines. More throughput, same operational pain.
The standard fix is to use the Infor Data Lake API for the initial load. The API has no equivalent transaction throttle. You pull CSV directly, and bulk-load it into SQL Server with SqlBulkCopy. Once the initial load is done, you switch to Streaming Pipelines (or CDC, if available) for incremental updates.
Authenticate with Infor’s identity service, then request the table as CSV. In .NET:
var token = await infor.GetAccessTokenAsync();
var request = new HttpRequestMessage(
HttpMethod.Get,
$"{dataLakeBaseUrl}/data/v2/tablename?format=csv"
);
request.Headers.Authorization =
new AuthenticationHeaderValue("Bearer", token);
var response = await http.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
await using var csvStream = await response.Content.ReadAsStreamAsync();
SqlBulkCopy is the right tool for this. It writes thousands of rows per second to SQL Server using minimal logging and no triggers.
using var reader = new CsvDataReader(
new StreamReader(csvStream)
);
using var bulk = new SqlBulkCopy(connectionString) {
DestinationTableName = "dbo.tablename",
BatchSize = 10000,
BulkCopyTimeout = 600
};
await bulk.WriteToServerAsync(reader);
Once the initial load is done, configure Streaming Pipelines (or CDC) for ongoing changes. The daily incremental volume almost always fits comfortably under the throttle. The throttle only hurts initial loads, not steady state.
The Compass NOT NULL trap is the first error most teams hit on their initial load. The Compass schema (which the Data Lake API exposes) marks many fields as NOT NULL. Current data conforms. Historical data, written years ago when validation rules were different, often has nulls in fields that are now required.
The result: the initial load fails on the first non-conforming row, and the error message doesn’t immediately point at the constraint.
The fix: relax NOT NULL constraints in your destination tables before the initial load. After the load completes, you can either leave them relaxed or add them back if current data is clean. Most teams leave them relaxed.
Cesovas’ Mirrordex replaces both with a single log-based CDC stream that has no throttle and runs in your environment.
Most Streaming Pipelines contracts cap each pipeline at around 165,000 daily transactions. Once you hit the limit, the pipeline throttles. The cap applies per pipeline, per day. Buying more capacity is possible but pricing scales steeply.
Infor Data Lake APIs have practical limits around request rates, payload sizes, concurrent queries, and Streaming Pipeline transaction caps. Many environments throttle a pipeline after roughly 165,000 transactions per day per pipeline. Large integrations usually require batching, pagination, and incremental sync strategies.
The most common workaround is distributing workloads across multiple pipelines and reducing unnecessary transactions through batching and incremental syncs. Many teams also move heavy bulk loads outside Streaming Pipelines into external ETL or database processes.
No. SqlBulkCopy is optimized for fast inserts only. Updates and deletes are typically handled afterward using staging tables with SQL MERGE, UPDATE, or DELETE statements.
Often yes. The Data Lake API is polling-based, so it works well for batch and bulk but not for real-time. Many teams use the Data Lake API for initial loads and high-volume tables, and Streaming Pipelines for low-volume reference data where ease of setup matters more than throughput.
Blog May 14, 2026 Infor Data Extraction Infor data extraction has three real paths: Streaming Pipelines, the Data Lake API,...
The 165K transaction limit is the daily cap most Infor Streaming Pipelines contracts apply to each pipeline. Once a pipeline hits 165,000 transactions in a day, it throttles. The pipeline doesn’t stop, it just falls behind, and the backlog clears against the same daily allowance the next day.
The exact number varies by contract. Some customers have higher caps. Some have lower. 165K is common enough that it’s the number most Infor forum posts mention.
The throttle counts each row movement as one transaction. A bulk update affecting 10,000 rows counts as 10,000 transactions, not one.
|
Key Parameters
|
Streaming Pipelines
|
Data Lake API
|
DLSync
|
|---|---|---|---|
|
Latency
|
Near real-time
|
Polling interval
|
Near real-time
|
|
Source load
|
Low
|
Medium
|
Negligible
|
|
Throttle/limits
|
~165K txn/day
|
100K rows or 10MB per result; daily compute cap
|
None
|
|
Captures
|
Inserts + updates
|
Whatever you query (bulk copy = inserts only)
|
Inserts + updates + deletes
|
|
Destination
|
Azure SQL only
|
Anywhere you load it
|
SQL Server (on-prem or cloud)
|
|
Schema
|
Generic datatypes
|
You define
|
Native Infor types preserved
|
|
Setup effort
|
Low
|
Medium-high
|
Low (Cesova installs)
|
|
Multi-tenant SaaS
|
Yes
|
Yes
|
Not currently
|
|
CSD / ISM (on-prem)
|
Yes
|
Yes
|
Yes
|
Engineering operational certainty
© 2026 Cesova. All rights reserved.