Skip to content

Commit 4c940f7

Browse files
committed
Added documentation about batch hints
1 parent 4511e58 commit 4c940f7

File tree

2 files changed

+5
-2
lines changed

2 files changed

+5
-2
lines changed

src/content/docs/pipelines/configuration/batching.mdx

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,13 @@ There are three ways to define how requests are batched:
1313
2. `batch-max-rows`: The maximum number of rows or events in a batch before data is written. Default, and maximum, is 10,000 rows.
1414
3. `batch-max-seconds`: The maximum duration of a batch before data is written, in seconds. Default is 15 seconds, maximum is 600 seconds.
1515

16+
Pipelines batch definitions are hints. A pipeline will follow these hints closely, but batches will not be exact.
17+
1618
All three batch definitions work together. Whichever limit is reached first triggers the delivery of a batch.
1719

1820
For example, a `batch-max-mb` = 100 MB and a `batch-max-seconds` = 600 means that if 100 MB of events are posted to the Pipeline, the batch will be delivered. However, if it takes longer than 600 seconds for 100 MB of events to be posted, a batch of all the messages that were posted during those 600 seconds will be created and delivered.
1921

22+
2023
## Batch settings
2124

2225
You can configure the following batch-level settings to adjust how Pipelines create a batch:

src/content/docs/pipelines/reference/limits.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,5 +19,5 @@ Many of these limits will increase during Pipelines' public beta period. [Follow
1919
| Requests per second | 10,000 |
2020
| Maximum payload per request | 1 MB |
2121
| Maximum batch size | 100 MB |
22-
| Maximum batch records | 10,000 |
23-
| Maximum batch duration | 600s |
22+
| Maximum batch records | 10,000,000 |
23+
| Maximum batch duration | 300s |

0 commit comments

Comments
 (0)