Skip to content

Commit 1a5e48a

Browse files
committed
New reports:
- `IndexerLevel - events per second benchmark` - `IndexerLevel - savedsearches by indexer execution time` - `SearchHeadLevel - indexes per savedsearch` - `SearchHeadLevel - macros in use` - `SearchHeadLevel - Indexes for savedsearch without subsearches` - `SearchHeadLevel - platform_stats.remote_searches metrics populating search 24 hour` Updated alerts: - `AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only` - updated criteria - `IndexerLevel - RemoteSearches find datamodel acceleration with wildcards` - updated regex - `MonitoringConsole - one or more servers require configuration` - changed criteria - `MonitoringConsole - one or more servers require configuration automated` - rewrote the alert - `SearchHeadLevel - Indexer Peer Connection Failures` - updated comments - `SearchHeadLevel - Detect searches hitting corrupt buckets` - updated comments - `SearchHeadLevel - Users with auto-finalized searches` - updated comments - `SearchHeadLevel - splunk_search_messages dispatch` - updated comments - `SearchHeadLevel - Lookups within savedsearches` - corrected URL - `SearchHeadLevel - Sourcetypes usage from search telemetry data` - description update - `SearchHeadLevel - Jobs endpoint example` - updated description - `SearchHeadLevel - SmartStore cache misses - dashboards` - minor update to regex - `SearchHeadLevel - SmartStore cache misses - combined` - minor update to regex - `SearchHeadLevel - Search Messages field extractor slow` - updated comments - `SearchHeadLevel - Search Messages user level` - updated comments - `SearchHeadLevel - Search Messages admins only` - updated criteria and comments Updated reports: - `IndexerLevel - RemoteSearches - lookup usage` - typo fixed in description - `IndexerLevel - Report on bucket corruption` - updated comments - `SearchHeadLevel - summary indexing searches not using durable search` - corrected REST context - `SearchHeadLevel - Lookups within savedsearches` - corrected REST context - `SearchHeadLevel - platform_stats.audit metrics users` - added v2/v1 endpoints for search/jobs/export - `SearchHeadLevel - platform_stats.audit metrics api` - added v2/v1 endpoints for search/jobs/export - `SearchHeadLevel - platform_stats.audit metrics users 24hour` - added v2/v1 endpoints for search/jobs/export Updated to use macro `splunkadmins_clustermaster_host` instead of splunk_server=local: - `ClusterMasterLevel - Primary bucket count per peer` - `ClusterMasterLevel - excess buckets on master` - `IndexerLevel - ClusterMaster Advising SearchOrRep Factor Not Met` Updated to use macro `splunkadmins_restmacro` instead of splunk_server=local: - `IndexerLevel - Indexer replication queue issues to some peers` - `SearchHeadLevel - Alerts that have not fired an action in X days` - `SearchHeadLevel - Accelerated DataModels Access Info` - `SearchHeadLevel - Accelerated DataModels with wildcard or no index specified` - `SearchHeadLevel - authorize.conf settings will prevent some users from appearing in the UI` - `SearchHeadLevel - Data Model Acceleration Completion Status` - `SearchHeadLevel - DataModel Fields` - `SearchHeadLevel - Dashboard refresh intervals` - `SearchHeadLevel - Dashboards using depends and running searches in the background` - `SearchHeadLevel - Dashboards using special characters` - `SearchHeadLevel - Dashboards with all time searches set` - `SearchHeadLevel - Dashboards that may benefit from base or post-process searches` - `SearchHeadLevel - DataModels report` - `SearchHeadLevel - Disabled modular inputs are running` - `SearchHeadLevel - Detect changes to knowledge objects non-directory` - `SearchHeadLevel - EventTypes report` - `SearchHeadLevel - Index access list by user` - `SearchHeadLevel - IndexesPerUser Report` - `SearchHeadLevel - Knowledge bundle status on indexers` - `SearchHeadLevel - Lookup file owners` - `SearchHeadLevel - Lookup CSV size` - `SearchHeadLevel - Macro report` - `SearchHeadLevel - platform_stats.users savedsearches` - `SearchHeadLevel - platform_stats.users dashboards` - `SearchHeadLevel - Saved Searches with privileged owners and excessive write perms` - `SearchHeadLevel - Summary searches using realtime search scheduling` - `SearchHeadLevel - SavedSearches using special characters` - `SearchHeadLevel - Splunk alert actions exceeding the max_action_results limit` - `SearchHeadLevel - summary indexing searches not using durable search` - `SearchHeadLevel - Tags report` Other macro updates: - `DeploymentServer - Count by application`
1 parent faaa83a commit 1a5e48a

File tree

9 files changed

+427
-109
lines changed

9 files changed

+427
-109
lines changed

README.md

Lines changed: 83 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,7 @@ The below list of alerts and reports are actively used since version 8.0.x and i
203203
- `IndexerLevel - Connection errors to SmartStore`
204204
- `IndexerLevel - ClusterMaster Advising SearchOrRep Factor Not Met`
205205
- `IndexerLevel - Data parsing error`
206+
- `IndexerLevel - events per second benchmark`
206207
- `IndexerLevel - IndexConfig Warnings from Splunk indexers`
207208
- `IndexerLevel - Indexer Queues May Have Issues`
208209
- `IndexerLevel - Indexer replication queue issues to some peers`
@@ -247,13 +248,15 @@ The below list of alerts and reports are actively used since version 8.0.x and i
247248
- `SearchHeadLevel - Lookup file owners`
248249
- `SearchHeadLevel - Lookups within dashboards`
249250
- `SearchHeadLevel - Lookups within savedsearches`
251+
- `SearchHeadLevel - macros in use`
250252
- `SearchHeadLevel - Peer timeouts or authentication issues`
251253
- `SearchHeadLevel - platform_stats access summary`
252254
- `SearchHeadLevel - platform_stats.audit metrics api`
253255
- `SearchHeadLevel - platform_stats.audit metrics searches`
254256
- `SearchHeadLevel - platform_stats.audit metrics users`
255257
- `SearchHeadLevel - platform_stats.audit metrics users 24hour`
256258
- `SearchHeadLevel - platform_stats.remote_searches metrics populating search`
259+
- `SearchHeadLevel - platform_stats.remote_searches metrics populating search 24 hour`
257260
- `SearchHeadLevel - platform_stats.user_stats.introspection metrics populating search`
258261
- `SearchHeadLevel - platform_stats.users dashboards`
259262
- `SearchHeadLevel - platform_stats.users savedsearches`
@@ -278,7 +281,7 @@ The below list of alerts and reports are actively used since version 8.0.x and i
278281
Some CSV lookups are now replaced with kvstore entries due to the ability to sync the kvstore across multiple search head or search head cluster(s) via apps like [KV Store Tools Redux](https://splunkbase.splunk.com/app/5328/)
279282

280283
## platform_stats reports
281-
There are a number of reports with the keyword "platform_stats" in the title, these were designed to run mcollect commands and to collect data into a metric index
284+
There are a number of reports with the keyword "platform_stats" in the title, these were designed to run mcollect commands (or to use summary indexing and durable search) to collect data into a metrics index
282285
The metrics then contain detailed information around the number of users using Splunk per-search head cluster, data indexed at the indexing tier, resource usage per user et cetera.
283286
There is plenty of detail in here but dashboards were not included for the information built from them, contributions welcome
284287

@@ -325,6 +328,83 @@ The following ideas relate to this issue:
325328
Feel free to open an issue on github or use the contact author on the SplunkBase link and I will try to get back to you when possible, thanks!
326329

327330
## Release Notes
331+
### 3.0.13
332+
New reports:
333+
- `IndexerLevel - events per second benchmark`
334+
- `IndexerLevel - savedsearches by indexer execution time`
335+
- `SearchHeadLevel - indexes per savedsearch`
336+
- `SearchHeadLevel - macros in use`
337+
- `SearchHeadLevel - Indexes for savedsearch without subsearches`
338+
- `SearchHeadLevel - platform_stats.remote_searches metrics populating search 24 hour`
339+
340+
Updated alerts:
341+
- `AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only` - updated criteria
342+
- `IndexerLevel - RemoteSearches find datamodel acceleration with wildcards` - updated regex
343+
- `MonitoringConsole - one or more servers require configuration` - changed criteria
344+
- `MonitoringConsole - one or more servers require configuration automated` - rewrote the alert
345+
- `SearchHeadLevel - Indexer Peer Connection Failures` - updated comments
346+
- `SearchHeadLevel - Detect searches hitting corrupt buckets` - updated comments
347+
- `SearchHeadLevel - Users with auto-finalized searches` - updated comments
348+
- `SearchHeadLevel - splunk_search_messages dispatch` - updated comments
349+
- `SearchHeadLevel - Lookups within savedsearches` - corrected URL
350+
- `SearchHeadLevel - Sourcetypes usage from search telemetry data` - description update
351+
- `SearchHeadLevel - Jobs endpoint example` - updated description
352+
- `SearchHeadLevel - SmartStore cache misses - dashboards` - minor update to regex
353+
- `SearchHeadLevel - SmartStore cache misses - combined` - minor update to regex
354+
- `SearchHeadLevel - Search Messages field extractor slow` - updated comments
355+
- `SearchHeadLevel - Search Messages user level` - updated comments
356+
- `SearchHeadLevel - Search Messages admins only` - updated criteria and comments
357+
358+
359+
Updated reports:
360+
- `IndexerLevel - RemoteSearches - lookup usage` - typo fixed in description
361+
- `IndexerLevel - Report on bucket corruption` - updated comments
362+
- `SearchHeadLevel - summary indexing searches not using durable search` - corrected REST context
363+
- `SearchHeadLevel - Lookups within savedsearches` - corrected REST context
364+
- `SearchHeadLevel - platform_stats.audit metrics users` - added v2/v1 endpoints for search/jobs/export
365+
- `SearchHeadLevel - platform_stats.audit metrics api` - added v2/v1 endpoints for search/jobs/export
366+
- `SearchHeadLevel - platform_stats.audit metrics users 24hour` - added v2/v1 endpoints for search/jobs/export
367+
368+
Updated to use macro `splunkadmins_clustermaster_host` instead of splunk_server=local:
369+
- `ClusterMasterLevel - Primary bucket count per peer`
370+
- `ClusterMasterLevel - excess buckets on master`
371+
- `IndexerLevel - ClusterMaster Advising SearchOrRep Factor Not Met`
372+
373+
Updated to use macro `splunkadmins_restmacro` instead of splunk_server=local:
374+
- `IndexerLevel - Indexer replication queue issues to some peers`
375+
- `SearchHeadLevel - Alerts that have not fired an action in X days`
376+
- `SearchHeadLevel - Accelerated DataModels Access Info`
377+
- `SearchHeadLevel - Accelerated DataModels with wildcard or no index specified`
378+
- `SearchHeadLevel - authorize.conf settings will prevent some users from appearing in the UI`
379+
- `SearchHeadLevel - Data Model Acceleration Completion Status`
380+
- `SearchHeadLevel - DataModel Fields`
381+
- `SearchHeadLevel - Dashboard refresh intervals`
382+
- `SearchHeadLevel - Dashboards using depends and running searches in the background`
383+
- `SearchHeadLevel - Dashboards using special characters`
384+
- `SearchHeadLevel - Dashboards with all time searches set`
385+
- `SearchHeadLevel - Dashboards that may benefit from base or post-process searches`
386+
- `SearchHeadLevel - DataModels report`
387+
- `SearchHeadLevel - Disabled modular inputs are running`
388+
- `SearchHeadLevel - Detect changes to knowledge objects non-directory`
389+
- `SearchHeadLevel - EventTypes report`
390+
- `SearchHeadLevel - Index access list by user`
391+
- `SearchHeadLevel - IndexesPerUser Report`
392+
- `SearchHeadLevel - Knowledge bundle status on indexers`
393+
- `SearchHeadLevel - Lookup file owners`
394+
- `SearchHeadLevel - Lookup CSV size`
395+
- `SearchHeadLevel - Macro report`
396+
- `SearchHeadLevel - platform_stats.users savedsearches`
397+
- `SearchHeadLevel - platform_stats.users dashboards`
398+
- `SearchHeadLevel - Saved Searches with privileged owners and excessive write perms`
399+
- `SearchHeadLevel - Summary searches using realtime search scheduling`
400+
- `SearchHeadLevel - SavedSearches using special characters`
401+
- `SearchHeadLevel - Splunk alert actions exceeding the max_action_results limit`
402+
- `SearchHeadLevel - summary indexing searches not using durable search`
403+
- `SearchHeadLevel - Tags report`
404+
405+
Other macro updates:
406+
- `DeploymentServer - Count by application`
407+
328408
### 3.0.12
329409
New alerts:
330410
- `MonitoringConsole - one or more servers require configuration`
@@ -540,10 +620,12 @@ New reports:
540620
- `SearchHeadLevel - Searches by search type`
541621

542622
Updated macros:
623+
- `search_type_from_sid`
543624
- `splunkadmins_splunkd_source`
544625
- `splunkadmins_splunkuf_source`
545626
- `splunkadmins_mongo_source`
546627
- `splunkadmins_license_usage_source`
628+
- `splunkadmins_deploymentserver_splunkserver`
547629

548630
To include a trailing wildcard (so splunkd.log.1 matches or similar)
549631

default/app.conf

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ supported_themes = light,dark
1414
[launcher]
1515
author = Gareth Anderson
1616
description = Alerts and dashboards as described in the Splunk 2017 conf presentation How did you get so big?
17-
version = 3.0.12
17+
version = 3.0.13
1818

1919
[package]
2020
id = SplunkAdmins

default/data/ui/nav/default.xml

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -141,7 +141,7 @@
141141
<view name="splunk_forwarder_output_tuning" />
142142
<view name="splunk_forwarder_data_balance_tuning" />
143143
<view name="splunk_introspection_io_stats" />
144-
<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FForwarderLevel%20-%20Channel%20churn%20issues">Channel churn issues</a>
144+
<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FForwarderLevel%20-%20Channel%20churn%20issues">Channel churn issues</a>
145145
</collection>
146146
<collection label="syslog-ng">
147147
<saved name="syslog-ng - cache statistics summary" />
@@ -222,7 +222,10 @@
222222
<saved name="IndexerLevel - Knowledge bundle upload stats" />
223223
<saved name="SearchHeadLevel - Knowledge bundle replication times metrics.log" />
224224
<saved name="SearchHeadLevel - Search Messages field extractor slow" />
225-
<saved name="IndexerLevel - IndexWriter pause duration" />
225+
<saved name="IndexerLevel - IndexWriter pause duration" />
226+
<saved name="IndexerLevel - events per second benchmark" />
227+
<saved name="IndexerLevel - savedsearches by indexer execution time" />
228+
<saved name="SearchHeadLevel - Indexes for savedsearch without subsearches" />
226229
</collection>
227230
<collection label="SmartStore">
228231
<saved name="SearchHeadLevel - SmartStore cache misses - savedsearches" />
@@ -267,9 +270,10 @@
267270
<saved name="SearchHeadLevel - platform_stats.audit metrics users 24hour" />
268271
<saved name="SearchHeadLevel - platform_stats.users dashboards" />
269272
<saved name="SearchHeadLevel - platform_stats.users savedsearches" />
270-
<saved name="SearchHeadLevel - platform_stats.user_stats.introspection metrics populating search" />
273+
<saved name="SearchHeadLevel - platform_stats.user_stats.introspection metrics populating search" />
271274
<saved name="SearchHeadLevel - platform_stats access summary" />
272275
<saved name="SearchHeadLevel - platform_stats.remote_searches metrics populating search" />
276+
<saved name="SearchHeadLevel - platform_stats.remote_searches metrics populating search 24 hour" />
273277
<saved name="SearchHeadLevel - audit.log - lookup usage" />
274278
<saved name="SearchHeadLevel - Lookup Editor lookup updates" />
275279
<saved name="IndexerLevel - platform_stats.counters hosts" />
@@ -294,6 +298,8 @@
294298
<saved name="SearchHeadLevel - audit.log - lookup usage" />
295299
<saved name="SearchHeadLevel - Detect lookups that have not being accessed for a period of time" />
296300
<saved name="SearchHeadLevel - Lookup Editor lookup updates" />
301+
<saved name="SearchHeadLevel - indexes per savedsearch" />
302+
<saved name="SearchHeadLevel - macros in use" />
297303
<saved name="SearchHeadLevel - Search Queries Per Day Audit Logs" />
298304
<saved name="SearchHeadLevel - Search Queries By Type Audit Logs" />
299305
<saved name="SearchHeadLevel - Search Queries By Type Audit Logs macro version" />
@@ -317,6 +323,7 @@
317323
<saved name="IndexerLevel - RemoteSearches Indexes Stats" />
318324
<saved name="IndexerLevel - RemoteSearches Indexes Stats Wilcard" />
319325
<saved name="IndexerLevel - RemoteSearches - lookup usage" />
326+
<saved name="IndexerLevel - events per second benchmark" />
320327
</collection>
321328
<collection label="Data Models">
322329
<saved name="SearchHeadLevel - Data Model Acceleration Completion Status" />
@@ -442,6 +449,8 @@
442449
<saved name="SearchHeadLevel - Detect changes to knowledge objects directory" />
443450
<saved name="SearchHeadLevel - Detect changes to knowledge objects non-directory" />
444451
<saved name="SearchHeadLevel - Lookup updates within SHC" />
452+
<saved name="SearchHeadLevel - indexes per savedsearch" />
453+
<saved name="SearchHeadLevel - macros in use" />
445454
<saved name="SearchHeadLevel - SHC conf log summary" />
446455
<saved name="SearchHeadLevel - Searches dispatched as owner by other users" />
447456
<saved name="SearchHeadLevel - Lookup CSV size" />
@@ -470,11 +479,16 @@
470479
<saved name="SearchHeadLevel - Knowledge Bundle contents" />
471480
<saved name="SearchHeadLevel - license usage per sourcetype per index" />
472481
<saved name="syslog-ng - cache statistics summary" />
482+
<saved name="IndexerLevel - events per second benchmark" />
483+
<saved name="IndexerLevel - savedsearches by indexer execution time" />
484+
<saved name="SearchHeadLevel - Indexes for savedsearch without subsearches" />
473485
</collection>
474486
<collection label="Summary_Reports">
475487
<saved name="SearchHeadLevel - audit.log - lookup usage" />
476488
<saved name="SearchHeadLevel - Lookup Editor lookup updates" />
477489
<saved name="SearchHeadLevel - license usage per sourcetype per index" />
490+
<saved name="SearchHeadLevel - indexes per savedsearch" />
491+
<saved name="SearchHeadLevel - macros in use" />
478492
<saved name="SearchHeadLevel - platform_stats.audit metrics searches" />
479493
<saved name="SearchHeadLevel - platform_stats.audit metrics users" />
480494
<saved name="SearchHeadLevel - platform_stats.audit metrics users 24hour" />
@@ -483,7 +497,8 @@
483497
<saved name="SearchHeadLevel - platform_stats.audit metrics api" />
484498
<saved name="SearchHeadLevel - platform_stats.user_stats.introspection metrics populating search" />
485499
<saved name="SearchHeadLevel - platform_stats access summary" />
486-
<saved name="SearchHeadLevel - platform_stats.remote_searches metrics populating search" />
500+
<saved name="SearchHeadLevel - platform_stats.remote_searches metrics populating search" />
501+
<saved name="SearchHeadLevel - platform_stats.remote_searches metrics populating search 24 hour" />
487502
<saved name="IndexerLevel - platform_stats.counters hosts" />
488503
<saved name="IndexerLevel - platform_stats.counters hosts 24hour" />
489504
<saved name="IndexerLevel - platform_stats.indexers totalgb measurement" />
@@ -525,6 +540,9 @@
525540
<collection label="Summary_Reports">
526541
<saved name="SearchHeadLevel - audit.log - lookup usage" />
527542
<saved name="SearchHeadLevel - Lookup Editor lookup updates" />
543+
<saved name="SearchHeadLevel - license usage per sourcetype per index" />
544+
<saved name="SearchHeadLevel - indexes per savedsearch" />
545+
<saved name="SearchHeadLevel - macros in use" />
528546
<saved name="SearchHeadLevel - platform_stats.audit metrics searches" />
529547
<saved name="SearchHeadLevel - platform_stats.audit metrics users" />
530548
<saved name="SearchHeadLevel - platform_stats.audit metrics api" />
@@ -534,6 +552,7 @@
534552
<saved name="SearchHeadLevel - platform_stats.user_stats.introspection metrics populating search" />
535553
<saved name="SearchHeadLevel - platform_stats access summary" />
536554
<saved name="SearchHeadLevel - platform_stats.remote_searches metrics populating search" />
555+
<saved name="SearchHeadLevel - platform_stats.remote_searches metrics populating search 24 hour" />
537556
<saved name="IndexerLevel - platform_stats.counters hosts" />
538557
<saved name="IndexerLevel - platform_stats.counters hosts 24hour" />
539558
<saved name="IndexerLevel - platform_stats.indexers totalgb measurement" />

default/data/ui/views/detect_excessive_search_use.xml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@
6060
<table>
6161
<title>Note: cluster command in use, introspection data may better list all dashboards in use</title>
6262
<search>
63-
<query>index=_internal (sourcetype=splunkd_access (method="GET" AND "/services/search/jobs/export") OR method="POST") OR (sourcetype=splunkd_ui_access method=POST "/report?" OR "/search?" OR "/search/jobs" OR "/saved/searches" NOT "/search/parser HTTP" NOT "/user-prefs/data/user-prefs/") OR (sourcetype=splunkd_ui_access method=GET "/app/" NOT "/search HTTP" NOT "/dashboards HTTP" NOT "/alerts HTTP" NOT "/reports HTTP") user IN ($username$)
63+
<query>index=_internal (sourcetype=splunkd_access (method="GET" AND "/services/search/jobs/export") OR method="POST") OR (sourcetype=splunkd_ui_access method=POST "/report?" OR "/search?" OR "/search/jobs" OR "/servicesNS/*/*/search/jobs" OR "/saved/searches" NOT "/search/parser HTTP" NOT "/user-prefs/data/user-prefs/") OR (sourcetype=splunkd_ui_access method=GET "/app/" NOT "/search HTTP" NOT "/dashboards HTTP" NOT "/alerts HTTP" NOT "/reports HTTP") user IN ($username$)
6464
| cluster t=0.95 showcount=true
6565
| rex field=uri "/servicesNS/[^/]+/(?P&lt;app&gt;[^/]+)"
6666
| rex field=uri "/[^/]+/app/(?P&lt;app&gt;[^/]+)/(?P&lt;dashboard_name&gt;[^/\?]+)"

default/data/ui/views/indexer_max_data_queue_sizes_by_name.xml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -127,7 +127,7 @@
127127
<chart>
128128
<title>The replication queue appears to directly relate to the indexing queue, any blockage of the indexing queue will then block the replication queue and temporarily slow data ingestion. The replication queue appears to be extremely sensitive to the other indexers indexing queue so it can be a useful measure of an issue...</title>
129129
<search>
130-
<query>index=_internal `indexerhosts` "replication queue for " "full" OR "has room now" sourcetype=splunkd
130+
<query>index=_internal `indexerhosts` "replication queue for " "full" OR "has room now" sourcetype=splunkd `splunkadmins_splunkd_source`
131131
| rename peer AS guid
132132
| join guid
133133
[| rest /services/search/distributed/peers

default/data/ui/views/indexer_max_data_queue_sizes_by_name_v8.xml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -127,7 +127,7 @@
127127
<chart>
128128
<title>The replication queue appears to directly relate to the indexing queue, any blockage of the indexing queue will then block the replication queue and temporarily slow data ingestion. The replication queue appears to be extremely sensitive to the other indexers indexing queue so it can be a useful measure of an issue...</title>
129129
<search>
130-
<query>index=_internal `indexerhosts` "replication queue for " "full" OR "has room now" sourcetype=splunkd
130+
<query>index=_internal `indexerhosts` "replication queue for " "full" OR "has room now" sourcetype=splunkd `splunkadmins_splunkd_source`
131131
| rename peer AS guid
132132
| join guid
133133
[| rest /services/search/distributed/peers

default/data/ui/views/splunk_forwarder_output_tuning.xml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,7 @@
105105
<body>
106106
<p>Purpose of destination count table? metrics.log only records the tcpout data *if* the connection is open at the time the metrics.log writes, so the count is to sanity-check that the numbers of connections matches the number of forwarders on the backend (this will happen with the below outputs.conf settings combined with regular data flow)</p>
107107
<br/>
108+
<p><a href="https://docs.splunk.com/Documentation/SVA/current/Architectures/Intermediaterouting#Asynchronous_load_balancing"> Asynchronous load balancing (docs.splunk.com) </a></p>
108109
<p><a href="https://www.linkedin.com/pulse/splunk-asynchronous-forwarding-lightning-fast-data-ingestor-rawat"> Splunk Asynchronous Forwarding (Lightning-fast data ingestor)</a></p>
109110
<p>Purpose of the data output per-second timechart? The current goal is to get close to switching indexers every second for an output group (per-pipeline), note that this will result in more open connections to indexers so only really works if this is deployed to a moderate number of intermediate forwarders (HF's or similar). Note that you want to do this with autoLBVolume, if you lower autoLBFrequency to a very short time period you may result in un-even data balance due to switching frequently when forwarding smaller volumes of data. In my testing so far it would appear that aiming above the average kb/s for the autoLBVolume appears to work well, going too low doesn't work well in my testing so far</p>
110111
<p>Please read the linked article for information on these settings, note that when using async forwarding the open file descriptor usage is higher than without async forwarding as the connections are held open by forwarders. So this works great on an intermediate forwarding tier, this may not work so well with a very large number of forwarders</p>

0 commit comments

Comments
 (0)