Skip to content

Commit 7cb859d

Browse files
kasiakoziolkasiakoziol
committed
CSPL-4003 Docs update
1 parent 20d9d6c commit 7cb859d

File tree

1 file changed

+91
-8
lines changed

1 file changed

+91
-8
lines changed

‎docs/IndexIngestionSeparation.md‎

Lines changed: 91 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,11 @@ This separation enables:
77
- Data durability: Off‑load buffer management and retry logic to a durable message bus.
88
- Operational clarity: Separate monitoring dashboards for ingestion throughput vs indexing latency.
99

10+
# <span style="color:red;"> Important Note</span>
11+
12+
# <span style="color:red;">As of now, only brand new deployments are supported for Index and Ingestion Separation. No migration path is implemented, described or tested for existing deployments to move from a standard model to Index & Ingestion separation model.</span>
13+
14+
1015
# IngestorCluster
1116

1217
IngestorCluster is introduced for high‑throughput data ingestion into a durable message bus. Its Splunk pods are configured to receive events (outputs.conf) and publish them to a message bus.
@@ -18,8 +23,8 @@ In addition to common spec inputs, the IngestorCluster resource provides the fol
1823
| Key | Type | Description |
1924
| ---------- | ------- | ------------------------------------------------- |
2025
| replicas | integer | The number of replicas (defaults to 3) |
21-
| pushBus | PushBus | Message bus configuration for publishing messages |
22-
| pipelineConfig | PipelineConfig | Configuration for pipeline |
26+
| pushBus | PushBus | Message bus configuration for publishing messages (required) |
27+
| pipelineConfig | PipelineConfig | Configuration for pipeline (required) |
2328

2429
PushBus inputs can be found in the table below. As of now, only SQS type of message bus is supported.
2530

@@ -38,9 +43,10 @@ SQS message bus inputs can be found in the table below.
3843
| largeMessageStoreEndpoint | string | AWS S3 Large Message Store endpoint (e.g. https://s3.us-west-2.amazonaws.com) |
3944
| largeMessageStorePath | string | S3 path for Large Message Store (e.g. s3://bucket-name/directory) |
4045
| deadLetterQueueName | string | Name of the SQS dead letter queue |
41-
| maxRetriesPerPart | integer | Max retries per part for retry policy max_count (The only one supported as of now) |
42-
| retryPolicy | string | Retry policy (max_retry is the only one supported as of now) |
46+
| maxRetriesPerPart | integer | Max retries per part for max_count retry policy |
47+
| retryPolicy | string | Retry policy (e.g. max_count) |
4348
| sendInterval | string | Send interval (e.g. 5s) |
49+
| encodingFormat | string | Encoding format (e.g. s2s) |
4450

4551
PipelineConfig inputs can be found in the table below.
4652

@@ -84,6 +90,7 @@ spec:
8490
maxRetriesPerPart: 4
8591
retryPolicy: max_count
8692
sendInterval: 5s
93+
encodingFormat: s2s
8794
pipelineConfig:
8895
remoteQueueRuleset: false
8996
ruleSet: true
@@ -104,8 +111,8 @@ In addition to common spec inputs, the IndexerCluster resource provides the foll
104111
| Key | Type | Description |
105112
| ---------- | ------- | ------------------------------------------------- |
106113
| replicas | integer | The number of replicas (defaults to 3) |
107-
| pullBus | PushBus | Message bus configuration for pulling messages |
108-
| pipelineConfig | PipelineConfig | Configuration for pipeline |
114+
| pullBus | PushBus | Message bus configuration for pulling messages (required) |
115+
| pipelineConfig | PipelineConfig | Configuration for pipeline (required) |
109116

110117
PullBus inputs can be found in the table below. As of now, only SQS type of message bus is supported.
111118

@@ -124,9 +131,10 @@ SQS message bus inputs can be found in the table below.
124131
| largeMessageStoreEndpoint | string | AWS S3 Large Message Store endpoint (e.g. https://s3.us-west-2.amazonaws.com) |
125132
| largeMessageStorePath | string | S3 path for Large Message Store (e.g. s3://bucket-name/directory) |
126133
| deadLetterQueueName | string | Name of SQS dead letter queue |
127-
| maxRetriesPerPart | integer | Max retries per part for retry policy max_count (The only one supported as of now) |
128-
| retryPolicy | string | Retry policy (max_retry is the only one supported as of now) |
134+
| maxRetriesPerPart | integer | Max retries per part for max_count retry policy |
135+
| retryPolicy | string | Retry policy (e.g. max_count) |
129136
| sendInterval | string | Send interval (e.g. 5s) |
137+
| encodingFormat | string | Encoding format (e.g. s2s) |
130138

131139
PipelineConfig inputs can be found in the table below.
132140

@@ -182,6 +190,7 @@ spec:
182190
maxRetriesPerPart: 4
183191
retryPolicy: max_count
184192
sendInterval: 5s
193+
encodingFormat: s2s
185194
pipelineConfig:
186195
remoteQueueRuleset: false
187196
ruleSet: true
@@ -227,6 +236,78 @@ The following additional configuration parameters may be used for all Splunk Ent
227236
| livenessInitialDelaySeconds | number | Defines initialDelaySeconds for the liveness probe |
228237
| imagePullSecrets | [ImagePullSecrets](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) | Config to pull images from private registry (Use in conjunction with image config from [common spec](#common-spec-parameters-for-all-resources)) |
229238

239+
# Helm Charts
240+
241+
IngestorCluster can be deployed using a dedicated Helm chart. IndexerCluster helm chart has as well been enhanced to support new inputs.
242+
243+
## Example
244+
245+
Below examples describe how to define values for IngestorCluster and IndexerCluster similarly to the above yaml files specifications.
246+
247+
```
248+
ingestorCluster:
249+
enabled: true
250+
name: ingestor
251+
replicaCount: 3
252+
serviceAccount: ingestion-role-sa
253+
pipelineConfig:
254+
remoteQueueRuleset: false
255+
ruleSet: true
256+
remoteQueueTyping: false
257+
remoteQueueOutput: false
258+
typing: true
259+
indexerPipe: true
260+
pushBus:
261+
type: sqs_smartbus
262+
sqs:
263+
queueName: ing-ind-separation-q
264+
authRegion: us-west-2
265+
endpoint: https://sqs.us-west-2.amazonaws.com
266+
largeMessageStoreEndpoint: https://s3.us-west-2.amazonaws.com
267+
largeMessageStorePath: s3://ing-ind-separation/smartbus-test
268+
deadLetterQueueName: ing-ind-separation-dlq
269+
maxRetriesPerPart: 4
270+
retryPolicy: max_count
271+
sendInterval: 5s
272+
encodingFormat: s2s
273+
```
274+
275+
```
276+
clusterManager:
277+
enabled: true
278+
name: cm
279+
replicaCount: 1
280+
serviceAccount: ingestion-role-sa
281+
282+
indexerCluster:
283+
enabled: true
284+
name: indexer
285+
replicaCount: 3
286+
serviceAccount: ingestion-role-sa
287+
clusterManagerRef:
288+
name: cm
289+
pipelineConfig:
290+
remoteQueueRuleset: false
291+
ruleSet: true
292+
remoteQueueTyping: false
293+
remoteQueueOutput: false
294+
typing: true
295+
indexerPipe: true
296+
pullBus:
297+
type: sqs_smartbus
298+
sqs:
299+
queueName: ing-ind-separation-q
300+
authRegion: us-west-2
301+
endpoint: https://sqs.us-west-2.amazonaws.com
302+
largeMessageStoreEndpoint: https://s3.us-west-2.amazonaws.com
303+
largeMessageStorePath: s3://ing-ind-separation/smartbus-test
304+
deadLetterQueueName: ing-ind-separation-dlq
305+
maxRetriesPerPart: 4
306+
retryPolicy: max_count
307+
sendInterval: 5s
308+
encodingFormat: s2s
309+
```
310+
230311
# Service Account
231312

232313
To be able to configure ingestion and indexing resources correctly in a secure manner, it is required to provide these resources with the service account that is configured with a minimum set of permissions to complete required operations. With this provided, the right credentials are used by Splunk to peform its tasks.
@@ -552,6 +633,7 @@ spec:
552633
maxRetriesPerPart: 4
553634
retryPolicy: max_count
554635
sendInterval: 5s
636+
encodingFormat: s2s
555637
pipelineConfig:
556638
remoteQueueRuleset: false
557639
ruleSet: true
@@ -712,6 +794,7 @@ spec:
712794
maxRetriesPerPart: 4
713795
retryPolicy: max_count
714796
sendInterval: 5s
797+
encodingFormat: s2s
715798
pipelineConfig:
716799
remoteQueueRuleset: false
717800
ruleSet: true

0 commit comments

Comments
 (0)