You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/IndexIngestionSeparation.md
+16-45Lines changed: 16 additions & 45 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,6 +12,10 @@ This separation enables:
12
12
> [!WARNING]
13
13
> **As of now, only brand new deployments are supported for Index and Ingestion Separation. No migration path is implemented, described or tested for existing deployments to move from a standard model to Index & Ingestion separation model.**
14
14
15
+
# Document Variables
16
+
17
+
- SPLUNK_IMAGE_VERSION: Splunk Enterprise Docker Image version
18
+
15
19
# BusConfiguration
16
20
17
21
BusConfiguration is introduced to store message bus configuration to be shared among IngestorCluster and IndexerCluster.
@@ -70,9 +74,9 @@ In addition to common spec inputs, the IngestorCluster resource provides the fol
70
74
71
75
## Example
72
76
73
-
The example presented below configures IngestorCluster named ingestor with Splunk 10.0.0 image that resides in a default namespace and is scaled to 3 replicas that serve the ingestion traffic. This IngestorCluster custom resource is set up with the service account named ingestor-sa allowing it to perform SQS and S3 operations. Push Bus reference allows the user to specify queue and bucket settings for the ingestion process.
77
+
The example presented below configures IngestorCluster named ingestor with Splunk ${SPLUNK_IMAGE_VERSION} image that resides in a default namespace and is scaled to 3 replicas that serve the ingestion traffic. This IngestorCluster custom resource is set up with the service account named ingestor-sa allowing it to perform SQS and S3 operations. Push Bus reference allows the user to specify queue and bucket settings for the ingestion process.
74
78
75
-
In this case, the setup the SQS and S3 based configuration where the messages are stored in sqs-test queue in us-west-2 region with dead letter queue set to sqs-dlq-test queue. The large message store is set to ingestion bucket in smartbus-test directory. Based on these inputs, default-mode.conf and outputs.conf files are configured accordingly.
79
+
In this case, the setup uses the SQS and S3 based configuration where the messages are stored in sqs-test queue in us-west-2 region with dead letter queue set to sqs-dlq-test queue. The large message store is set to ingestion bucket in smartbus-test directory. Based on these inputs, default-mode.conf and outputs.conf files are configured accordingly.
76
80
77
81
```
78
82
apiVersion: enterprise.splunk.com/v4
@@ -84,7 +88,7 @@ metadata:
84
88
spec:
85
89
serviceAccount: ingestor-sa
86
90
replicas: 3
87
-
image: splunk/splunk:10.0.0
91
+
image: splunk/splunk:${SPLUNK_IMAGE_VERSION}
88
92
busConfigurationRef:
89
93
name: bus-config
90
94
```
@@ -104,7 +108,7 @@ In addition to common spec inputs, the IndexerCluster resource provides the foll
104
108
105
109
## Example
106
110
107
-
The example presented below configures IndexerCluster named indexer with Splunk 10.0.0 image that resides in a default namespace and is scaled to 3 replicas that serve the indexing traffic. This IndexerCluster custom resource is set up with the service account named ingestor-sa allowing it to perform SQS and S3 operations. Pull Bus reference allows the user to specify queue and bucket settings for the indexing process.
111
+
The example presented below configures IndexerCluster named indexer with Splunk ${SPLUNK_IMAGE_VERSION} image that resides in a default namespace and is scaled to 3 replicas that serve the indexing traffic. This IndexerCluster custom resource is set up with the service account named ingestor-sa allowing it to perform SQS and S3 operations. Pull Bus reference allows the user to specify queue and bucket settings for the indexing process.
108
112
109
113
In this case, the setup uses the SQS and S3 based configuration where the messages are stored in and retrieved from sqs-test queue in us-west-2 region with dead letter queue set to sqs-dlq-test queue. The large message store is set to ingestion bucket in smartbus-test directory. Based on these inputs, default-mode.conf, inputs.conf and outputs.conf files are configured accordingly.
110
114
@@ -117,7 +121,7 @@ metadata:
117
121
- enterprise.splunk.com/delete-pvc
118
122
spec:
119
123
serviceAccount: ingestor-sa
120
-
image: splunk/splunk:10.0.0
124
+
image: splunk/splunk:${SPLUNK_IMAGE_VERSION}
121
125
---
122
126
apiVersion: enterprise.splunk.com/v4
123
127
kind: IndexerCluster
@@ -130,47 +134,14 @@ spec:
130
134
name: cm
131
135
serviceAccount: ingestor-sa
132
136
replicas: 3
133
-
image: splunk/splunk:10.0.0
137
+
image: splunk/splunk:${SPLUNK_IMAGE_VERSION}
134
138
busConfigurationRef:
135
139
name: bus-config
136
140
```
137
141
138
142
# Common Spec
139
143
140
-
The spec section is used to define the desired state for a resource. All custom resources provided by the Splunk Operator (with an exception for BusConfiguration) include the following
| image | string | Container image to use for pod instances (overrides RELATED_IMAGE_SPLUNK_ENTERPRISE environment variable) |
146
-
| imagePullPolicy | string | Sets pull policy for all images (either "Always" or the default: "IfNotPresent") |
147
-
| livenessInitialDelaySeconds | number | Sets the initialDelaySeconds for liveness probe (default: 300) |
148
-
| readinessInitialDelaySeconds | number | Sets the initialDelaySeconds for readiness probe (default: 10) |
149
-
| extraEnv |[EnvVar](https://v1-17.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#envvar-v1-core)| Sets the extra environment variables to be passed to the Splunk instance containers (WARNING: Setting environment variables used by Splunk or Ansible will affect Splunk installation and operation) |
150
-
| schedulerName | string | Name of [Scheduler](https://kubernetes.io/docs/concepts/scheduling/kube-scheduler/) to use for pod placement (defaults to "default-scheduler") |
151
-
| affinity |[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#affinity-v1-core)|[Kubernetes Affinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity) rules that control how pods are assigned to particular nodes |
152
-
| resources |[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#resourcerequirements-v1-core)| The settings for allocating [compute resource requirements](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/) to use for each pod instance (The default settings should be considered for demo/test purposes. Please see [Hardware Resource Requirements](https://github.com/splunk/splunk-operator/blob/develop/docs/README.md#hardware-resources-requirements) for production values.) |
153
-
| serviceTemplate |[Service](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#service-v1-core)| Template used to create [Kubernetes services](https://kubernetes.io/docs/concepts/services-networking/service/)|
154
-
| topologySpreadConstraint |[TopologySpreadConstraint](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/)| Template used to create [Kubernetes TopologySpreadConstraint](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/)|
155
-
156
-
The following additional configuration parameters may be used for all Splunk Enterprise resources.
| etcVolumeStorageConfig | StorageClassSpec | Storage class spec for Splunk etc volume as described in [StorageClass](StorageClass.md)|
161
-
| varVolumeStorageConfig | StorageClassSpec | Storage class spec for Splunk var volume as described in [StorageClass](StorageClass.md)|
162
-
| volumes |[Volume](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#volume-v1-core)| List of one or more [Kubernetes volumes](https://kubernetes.io/docs/concepts/storage/volumes/) (These will be mounted in all container pods as `/mnt/<name>`) |
163
-
| defaults | string | Inline map of [default.yml](https://github.com/splunk/splunk-ansible/blob/develop/docs/advanced/default.yml.spec.md) used to initialize the environment |
164
-
| defaultsUrl | string | Full path or URL for one or more [default.yml](https://github.com/splunk/splunk-ansible/blob/develop/docs/advanced/default.yml.spec.md) files (separated by commas) |
165
-
| licenseUrl | string | Full path or URL for a Splunk Enterprise license file |
166
-
| licenseManagerRef |[ObjectReference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#objectreference-v1-core)| Reference to a Splunk Operator managed LicenseManager instance (via name and optionally namespace) to use for licensing |
167
-
| clusterManagerRef |[ObjectReference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#objectreference-v1-core)| Reference to a Splunk Operator managed ClusterManager instance (via name and optionally namespace) to use for indexing |
168
-
| monitoringConsoleRef | string | Logical name assigned to the Monitoring Console pod (You can set the name before or after the MC pod creation) |
169
-
| serviceAccount |[ServiceAccount](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/)| Represents the service account used by the pods deployed by the CRD |
170
-
| extraEnv |[EnvVar](https://v1-17.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#envvar-v1-core)| Extra environment variables to be passed to the Splunk instance containers |
171
-
| readinessInitialDelaySeconds | number | Defines initialDelaySeconds for readiness probe |
172
-
| livenessInitialDelaySeconds | number | Defines initialDelaySeconds for the liveness probe |
173
-
| imagePullSecrets |[ImagePullSecrets](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/)| Config to pull images from private registry (Use in conjunction with image config from [common spec](#common-spec-parameters-for-all-resources)) |
144
+
Common spec values for all SOK Custom Resources can be found in [CustomResources doc](CustomResources.md).
174
145
175
146
# Helm Charts
176
147
@@ -334,7 +305,7 @@ To automatically adjust the number of replicas to serve the ingestion traffic ef
334
305
335
306
## Example
336
307
337
-
The exmaple presented below configures HorizontalPodAutoscaler named ingestor-hpa that resides in a default namespace to scale IngestorCluster custom resource named ingestor. With average utilization set to 50, the HorizontalPodAutoscaler resource will try to keep the average utilization of the pods in the scaling target at 50%. It will be able to scale the replicas starting from the minimum number of 3 with the maximum number of 10 replicas.
308
+
The exmaple presented below configures HorizontalPodAutoscaler named ingestor-hpa that resides in a default namespace (same namespace as resources it is managing) to scale IngestorCluster custom resource named ingestor. With average utilization set to 50, the HorizontalPodAutoscaler resource will try to keep the average utilization of the pods in the scaling target at 50%. It will be able to scale the replicas starting from the minimum number of 3 with the maximum number of 10 replicas.
0 commit comments