Skip to content

Commit ceb2c71

Browse files
kasiakoziolkasiakoziol
committed
CSPL-4022 Docs update
1 parent db02bfd commit ceb2c71

File tree

1 file changed

+104
-45
lines changed

1 file changed

+104
-45
lines changed

‎docs/IndexIngestionSeparation.md‎

Lines changed: 104 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -31,11 +31,12 @@ SQS message bus inputs can be found in the table below.
3131
| ---------- | ------- | ------------------------------------------------- |
3232
| queueName | string | Name of the SQS queue |
3333
| authRegion | string | Region where the SQS queue is located |
34-
| endpoint | string | AWS SQS endpoint (e.g. https://sqs.us-west-2.amazonaws.com) |
35-
| largeMessageStoreEndpoint | string | AWS S3 Large Message Store endpoint (e.g. https://s3.us-west-2.amazonaws.com) |
36-
| largeMessageStorePath | string | S3 path for Large Message Store (e.g. s3://bucket-name/directory) |
34+
| endpoint | string | AWS SQS endpoint
35+
| largeMessageStoreEndpoint | string | AWS S3 Large Message Store endpoint |
36+
| largeMessageStorePath | string | S3 path for Large Message Store |
3737
| deadLetterQueueName | string | Name of the SQS dead letter queue |
3838

39+
Change of any of the bus inputs does not restart Splunk. It just updates the config values with no disruptions.
3940

4041
## Example
4142
```
@@ -69,11 +70,9 @@ In addition to common spec inputs, the IngestorCluster resource provides the fol
6970

7071
## Example
7172

72-
The example presented below configures IngestorCluster named ingestor with Splunk 9.4.4 image that resides in a default namespace and is scaled to 3 replicas that serve the ingestion traffic. This IngestorCluster custom resource is set up with the service account named ingestor-sa allowing it to perform SQS and S3 operations. Push Bus inputs allow the user to specify queue and bucket settings for the ingestion process.
73-
74-
In this case, the setup uses bus configuration resource reference that is the SQS and S3 based configuration where the messages are stored in sqs-test queue in us-west-2 region with dead letter queue set to sqs-dlq-test queue. The large message store is set to ingestion bucket in smartbus-test directory. Based on these inputs, default-mode.conf and outputs.conf files are configured accordingly.
73+
The example presented below configures IngestorCluster named ingestor with Splunk 10.0.0 image that resides in a default namespace and is scaled to 3 replicas that serve the ingestion traffic. This IngestorCluster custom resource is set up with the service account named ingestor-sa allowing it to perform SQS and S3 operations. Push Bus reference allows the user to specify queue and bucket settings for the ingestion process.
7574

76-
Change of any of the bus inputs does not restart Splunk. It just updates the config values with no disruptions.
75+
In this case, the setup the SQS and S3 based configuration where the messages are stored in sqs-test queue in us-west-2 region with dead letter queue set to sqs-dlq-test queue. The large message store is set to ingestion bucket in smartbus-test directory. Based on these inputs, default-mode.conf and outputs.conf files are configured accordingly.
7776

7877
```
7978
apiVersion: enterprise.splunk.com/v4
@@ -85,7 +84,7 @@ metadata:
8584
spec:
8685
serviceAccount: ingestor-sa
8786
replicas: 3
88-
image: splunk/splunk:9.4.4
87+
image: splunk/splunk:10.0.0
8988
busConfigurationRef:
9089
name: bus-config
9190
```
@@ -105,11 +104,9 @@ In addition to common spec inputs, the IndexerCluster resource provides the foll
105104

106105
## Example
107106

108-
The example presented below configures IndexerCluster named indexer with Splunk 9.4.4 image that resides in a default namespace and is scaled to 3 replicas that serve the indexing traffic. This IndexerCluster custom resource is set up with the service account named ingestor-sa allowing it to perform SQS and S3 operations. Pull Bus inputs allow the user to specify queue and bucket settings for the indexing process.
107+
The example presented below configures IndexerCluster named indexer with Splunk 10.0.0 image that resides in a default namespace and is scaled to 3 replicas that serve the indexing traffic. This IndexerCluster custom resource is set up with the service account named ingestor-sa allowing it to perform SQS and S3 operations. Pull Bus reference allows the user to specify queue and bucket settings for the indexing process.
109108

110-
In this case, In this case, the setup uses bus configuration resource reference that is the SQS and S3 based configuration where the messages are stored in and retrieved from sqs-test queue in us-west-2 region with dead letter queue set to sqs-dlq-test queue. The large message store is set to ingestion bucket in smartbus-test directory. Based on these inputs, default-mode.conf, inputs.conf and outputs.conf files are configured accordingly.
111-
112-
Change of any of the bus inputs does not restart Splunk. It just updates the config values with no disruptions.
109+
In this case, the setup uses the SQS and S3 based configuration where the messages are stored in and retrieved from sqs-test queue in us-west-2 region with dead letter queue set to sqs-dlq-test queue. The large message store is set to ingestion bucket in smartbus-test directory. Based on these inputs, default-mode.conf, inputs.conf and outputs.conf files are configured accordingly.
113110

114111
```
115112
apiVersion: enterprise.splunk.com/v4
@@ -120,7 +117,7 @@ metadata:
120117
- enterprise.splunk.com/delete-pvc
121118
spec:
122119
serviceAccount: ingestor-sa
123-
image: splunk/splunk:9.4.4
120+
image: splunk/splunk:10.0.0
124121
---
125122
apiVersion: enterprise.splunk.com/v4
126123
kind: IndexerCluster
@@ -133,7 +130,7 @@ spec:
133130
name: cm
134131
serviceAccount: ingestor-sa
135132
replicas: 3
136-
image: splunk/splunk:9.4.4
133+
image: splunk/splunk:10.0.0
137134
busConfigurationRef:
138135
name: bus-config
139136
```
@@ -419,7 +416,7 @@ $ make install
419416
```
420417

421418
```
422-
$ kubectl apply -f SOK_IMAGE_VERSION/splunk-operator-cluster.yaml --server-side
419+
$ kubectl apply -f ${SOK_IMAGE_VERSION}/splunk-operator-cluster.yaml --server-side
423420
```
424421

425422
```
@@ -524,7 +521,69 @@ $ aws iam list-attached-role-policies --role-name eksctl-ind-ing-sep-demo-addon-
524521
}
525522
```
526523

527-
3. Install IngestorCluster resource.
524+
3. Install BusConfiguration resource.
525+
526+
```
527+
$ cat bus.yaml
528+
apiVersion: enterprise.splunk.com/v4
529+
kind: BusConfiguration
530+
metadata:
531+
name: bus
532+
finalizers:
533+
- enterprise.splunk.com/delete-pvc
534+
spec:
535+
type: sqs_smartbus
536+
sqs:
537+
queueName: sqs-test
538+
authRegion: us-west-2
539+
endpoint: https://sqs.us-west-2.amazonaws.com
540+
largeMessageStoreEndpoint: https://s3.us-west-2.amazonaws.com
541+
largeMessageStorePath: s3://ingestion/smartbus-test
542+
deadLetterQueueName: sqs-dlq-test
543+
```
544+
545+
```
546+
$ kubectl apply -f bus.yaml
547+
```
548+
549+
```
550+
$ kubectl get busconfiguration
551+
NAME PHASE AGE MESSAGE
552+
bus Ready 20s
553+
```
554+
555+
```
556+
kubectl describe busconfiguration
557+
Name: bus
558+
Namespace: default
559+
Labels: <none>
560+
Annotations: <none>
561+
API Version: enterprise.splunk.com/v4
562+
Kind: BusConfiguration
563+
Metadata:
564+
Creation Timestamp: 2025-10-27T10:25:53Z
565+
Finalizers:
566+
enterprise.splunk.com/delete-pvc
567+
Generation: 1
568+
Resource Version: 12345678
569+
UID: 12345678-1234-5678-1234-012345678911
570+
Spec:
571+
Sqs:
572+
Auth Region: us-west-2
573+
Dead Letter Queue Name: sqs-dlq-test
574+
Endpoint: https://sqs.us-west-2.amazonaws.com
575+
Large Message Store Endpoint: https://s3.us-west-2.amazonaws.com
576+
Large Message Store Path: s3://ingestion/smartbus-test
577+
Queue Name: sqs-test
578+
Type: sqs_smartbus
579+
Status:
580+
Message:
581+
Phase: Ready
582+
Resource Rev Map:
583+
Events: <none>
584+
```
585+
586+
4. Install IngestorCluster resource.
528587

529588
```
530589
$ cat ingestor.yaml
@@ -537,7 +596,7 @@ metadata:
537596
spec:
538597
serviceAccount: ingestor-sa
539598
replicas: 3
540-
image: splunk/splunk:9.4.4
599+
image: splunk/splunk:10.0.0
541600
busConfigurationRef:
542601
name: bus-config
543602
```
@@ -568,19 +627,10 @@ Metadata:
568627
Resource Version: 12345678
569628
UID: 12345678-1234-1234-1234-1234567890123
570629
Spec:
571-
Image: splunk/splunk:9.4.4
572-
Push Bus:
573-
Sqs:
574-
Auth Region: us-west-2
575-
Dead Letter Queue Name: ing-ind-separation-dlq
576-
Endpoint: https://sqs.us-west-2.amazonaws.com
577-
Large Message Store Endpoint: https://s3.us-west-2.amazonaws.com
578-
Large Message Store Path: s3://ing-ind-separation/smartbus-test
579-
Max Retries Per Part: 4
580-
Queue Name: ing-ind-separation-q
581-
Retry Policy: max_count
582-
Send Interval: 3s
583-
Type: sqs_smartbus
630+
Bus Configuration Ref:
631+
Name: bus-config
632+
Namespace: default
633+
Image: splunk/splunk:10.0.0
584634
Replicas: 3
585635
Service Account: ingestor-sa
586636
Status:
@@ -595,6 +645,15 @@ Status:
595645
Is Deployment In Progress: false
596646
Last App Info Check Time: 0
597647
Version: 0
648+
Bus Configuration:
649+
Sqs:
650+
Auth Region: us-west-2
651+
Dead Letter Queue Name: sqs-dlq-test
652+
Endpoint: https://sqs.us-west-2.amazonaws.com
653+
Large Message Store Endpoint: https://s3.us-west-2.amazonaws.com
654+
Large Message Store Path: s3://ingestion/smartbus-test
655+
Queue Name: sqs-test
656+
Type: sqs_smartbus
598657
Message:
599658
Phase: Ready
600659
Ready Replicas: 3
@@ -635,20 +694,20 @@ disabled = true
635694
disabled = true
636695
637696
sh-4.4$ cat /opt/splunk/etc/system/local/outputs.conf
638-
[remote_queue:ing-ind-separation-q]
697+
[remote_queue:sqs-test]
639698
remote_queue.sqs_smartbus.max_count.max_retries_per_part = 4
640699
remote_queue.sqs_smartbus.auth_region = us-west-2
641-
remote_queue.sqs_smartbus.dead_letter_queue.name = ing-ind-separation-dlq
700+
remote_queue.sqs_smartbus.dead_letter_queue.name = sqs-dlq-test
642701
remote_queue.sqs_smartbus.encoding_format = s2s
643702
remote_queue.sqs_smartbus.endpoint = https://sqs.us-west-2.amazonaws.com
644703
remote_queue.sqs_smartbus.large_message_store.endpoint = https://s3.us-west-2.amazonaws.com
645-
remote_queue.sqs_smartbus.large_message_store.path = s3://ing-ind-separation/smartbus-test
704+
remote_queue.sqs_smartbus.large_message_store.path = s3://ingestion/smartbus-test
646705
remote_queue.sqs_smartbus.retry_policy = max_count
647706
remote_queue.sqs_smartbus.send_interval = 5s
648707
remote_queue.type = sqs_smartbus
649708
```
650709

651-
4. Install IndexerCluster resource.
710+
5. Install IndexerCluster resource.
652711

653712
```
654713
$ cat idxc.yaml
@@ -659,7 +718,7 @@ metadata:
659718
finalizers:
660719
- enterprise.splunk.com/delete-pvc
661720
spec:
662-
image: splunk/splunk:9.4.4
721+
image: splunk/splunk:10.0.0
663722
serviceAccount: ingestor-sa
664723
---
665724
apiVersion: enterprise.splunk.com/v4
@@ -669,7 +728,7 @@ metadata:
669728
finalizers:
670729
- enterprise.splunk.com/delete-pvc
671730
spec:
672-
image: splunk/splunk:9.4.4
731+
image: splunk/splunk:10.0.0
673732
replicas: 3
674733
clusterManagerRef:
675734
name: cm
@@ -709,24 +768,24 @@ sh-4.4$ cat /opt/splunk/etc/system/local/inputs.conf
709768
[splunktcp://9997]
710769
disabled = 0
711770
712-
[remote_queue:ing-ind-separation-q]
771+
[remote_queue:sqs-test]
713772
remote_queue.sqs_smartbus.max_count.max_retries_per_part = 4
714773
remote_queue.sqs_smartbus.auth_region = us-west-2
715-
remote_queue.sqs_smartbus.dead_letter_queue.name = ing-ind-separation-dlq
774+
remote_queue.sqs_smartbus.dead_letter_queue.name = sqs-dlq-test
716775
remote_queue.sqs_smartbus.endpoint = https://sqs.us-west-2.amazonaws.com
717776
remote_queue.sqs_smartbus.large_message_store.endpoint = https://s3.us-west-2.amazonaws.com
718-
remote_queue.sqs_smartbus.large_message_store.path = s3://ing-ind-separation/smartbus-test
777+
remote_queue.sqs_smartbus.large_message_store.path = s3://ingestion/smartbus-test
719778
remote_queue.sqs_smartbus.retry_policy = max_count
720779
remote_queue.type = sqs_smartbus
721780
sh-4.4$ cat /opt/splunk/etc/system/local/outputs.conf
722-
[remote_queue:ing-ind-separation-q]
781+
[remote_queue:sqs-test]
723782
remote_queue.sqs_smartbus.max_count.max_retries_per_part = 4
724783
remote_queue.sqs_smartbus.auth_region = us-west-2
725-
remote_queue.sqs_smartbus.dead_letter_queue.name = ing-ind-separation-dlq
784+
remote_queue.sqs_smartbus.dead_letter_queue.name = sqs-dlq-test
726785
remote_queue.sqs_smartbus.encoding_format = s2s
727786
remote_queue.sqs_smartbus.endpoint = https://sqs.us-west-2.amazonaws.com
728787
remote_queue.sqs_smartbus.large_message_store.endpoint = https://s3.us-west-2.amazonaws.com
729-
remote_queue.sqs_smartbus.large_message_store.path = s3://ing-ind-separation/smartbus-test
788+
remote_queue.sqs_smartbus.large_message_store.path = s3://ingestion/smartbus-test
730789
remote_queue.sqs_smartbus.retry_policy = max_count
731790
remote_queue.sqs_smartbus.send_interval = 5s
732791
remote_queue.type = sqs_smartbus
@@ -747,7 +806,7 @@ disabled = false
747806
disabled = true
748807
```
749808

750-
5. Install Horizontal Pod Autoscaler for IngestorCluster.
809+
6. Install Horizontal Pod Autoscaler for IngestorCluster.
751810

752811
```
753812
$ cat hpa-ing.yaml
@@ -830,7 +889,7 @@ NAME REFERENCE TARGETS MINPODS MAXPODS REPLICA
830889
ing-hpa IngestorCluster/ingestor cpu: 115%/50% 3 10 10 8m54s
831890
```
832891

833-
6. Generate fake load.
892+
7. Generate fake load.
834893

835894
- HEC_TOKEN: HEC token for making fake calls
836895

@@ -991,7 +1050,7 @@ splunk-ingestor-ingestor-2 1/1 Running 0 45m
9911050
```
9921051

9931052
```
994-
$ aws s3 ls s3://ing-ind-separation/smartbus-test/
1053+
$ aws s3 ls s3://ingestion/smartbus-test/
9951054
PRE 29DDC1B4-D43E-47D1-AC04-C87AC7298201/
9961055
PRE 43E16731-7146-4397-8553-D68B5C2C8634/
9971056
PRE C8A4D060-DE0D-4DCB-9690-01D8902825DC/

0 commit comments

Comments
 (0)