Merged pull request from jeffland-consist via github including various changes

New alerts:
- `IndexerLevel - replicationdatareceiverthread close to 100% utilisation`

New macros:
- `splunkadmins_metrics_source`
- `splunkadmins_hec_metrics_source`

New reports:
- `SearchHeadLevel - Accelerated DataModels Access Info`
- `SearchHeadLevel - Dashboards resulting in concurrency issues`
- `SearchHeadLevel - Dashboards that may benefit from base or post-process searches`
- `SearchHeadLevel - Searches by search type`

Updated macros:
- `splunkadmins_splunkd_source`
- `splunkadmins_splunkuf_source`
- `splunkadmins_mongo_source`
- `splunkadmins_license_usage_source`

To include a trailing wildcard (so splunkd.log.1 matches or similar)

Updated alerts:
- `AllSplunkEnterpriseLevel - Core Dumps Disabled` - updated matching criteria
- `AllSplunkEnterpriseLevel - Non-existent roles are assigned to users` - updated matching criteria
- `AllSplunkEnterpriseLevel - Splunk Servers throwing runScript errors` - updated matching criteria
- `AllSplunkEnterpriseLevel - sendmodalert errors` - updated matching criteria
- `AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only` - updated matching criteria
- `AllSplunkEnterpriseLevel - Splunk Servers with resource starvation` - updated to use `splunkadmins_splunkd_source` macro
- `AllSplunkLevel - No recent metrics.log data` - corrected comment to be after tstats, updated to use `splunkadmins_metrics_source` macro
- `AllSplunkLevel - DeploymentServer Application Installation Error` - updated matching criteria
- `DeploymentServer - Application Not Found On Deployment Server` - updated matching criteria
- `ForwarderLevel - Channel churn issues` - updated to use `splunkadmins_metrics_source` macro
- `ForwarderLevel - Forwarders connecting to a single endpoint for extended periods` - updated to use `splunkadmins_metrics_source` macro
- `ForwarderLevel - Forwarders connecting to a single endpoint for extended periods UF level` - updated to use `splunkadmins_metrics_source` macro
- `ForwarderLevel - Splunk HTTP Listener Overwhelmed` - updated matching criteria
- `ForwarderLevel - Splunk Universal Forwarders Exceeding the File Descriptor Cache` - updated matching criteria
- `ForwarderLevel - Splunk Universal Forwarders that are time shifting` - updated matching criteria
- `ForwarderLevel - Stopping all listening ports` - updated to use `splunkadmins_splunkd_source` macro
- `IndexerLevel - Buckets changes per day` - updated matching criteria, updated to use `splunkadmins_splunkd_source` macro
- `IndexerLevel - Indexer Queues May Have Issues` - updated to use `splunkadmins_metrics_source` macro
- `IndexerLevel - Knowledge bundle upload stats` - updated to use `splunkadmins_metrics_source` macro
- `IndexerLevel - platform_stats.indexers totalgb_thruput measurement` - updated to use `splunkadmins_metrics_source` macro
- `IndexerLevel - platform_stats.indexers stddev measurement` - updated to use `splunkadmins_metrics_source` macro
- `IndexerLevel - platform_stats.indexers stddev incoming measurement` - updated to use `splunkadmins_metrics_source` macro
- `IndexerLevel - Weekly Broken Events Report` - updated matching criteria
- `IndexerLevel - Time format has changed multiple log types in one sourcetype` - updated matching criteria
- `IndexerLevel - Buckets have being frozen due to index sizing` - updated matching criteria
- `IndexerLevel - Unclean Shutdown - Fsck` - updated matching criteria
- `IndexerLevel - Index not defined` - updated matching criteria
- `IndexerLevel - Timestamp parsing issues combined alert` - updated to use `splunkadmins_splunkd_source` macro
- `IndexerLevel - S2SFileReceiver Error` - updated matching criteria
- `MonitoringConsole - Core dumps have appeared on the filesystem` - corrected to use `indexer_cluster_name` macro
- `MonitoringConsole - Crash logs have appeared on the filesystem` - corrected description
- `SearchHeadLevel - LDAP users have been disabled or left the company cleanup required` - updated matching criteria
- `SearchHeadLevel - Long filenames may be causing issues` - updated matching criteria
- `SearchHeadLevel - SHCluster Artifact Replication Issues` - updated matching criteria
- `SearchHeadLevel - Captain Switchover Occurring` - updated matching criteria
- `SearchHeadLevel - Knowledge bundle replication times metrics.log` - updated to use `splunkadmins_metrics_source` macro
- `SearchHeadLevel - Detect bundle pushes no longer occurring` - updated to use `splunkadmins_metrics_source` macro
- `SearchHeadLevel - WLM aborted searches` - updated matching criteria
- `SearchHeadLevel - SHC Captain unable to establish common bundle` - updated to use `splunkadmins_splunkd_source` macro

Updated dashboards:
- `ClusterMasterJobs.xml`
- `heavyforwarders_max_data_queue_sizes_by_name.xml`
- `heavyforwarders_max_data_queue_sizes_by_name_v8.xml`
- `hec_performance.xml`
- `indexer_data_spread.xml`
- `indexer_max_data_queue_sizes_by_name.xml`
- `indexer_max_data_queue_sizes_by_name_v8.xml`
- `rolled_buckets_by_index.xml`
- `smartstore_stats.xml`
- `splunk_forwarder_data_balance_tuning.xml`
- `splunk_forwarder_output_tuning.xml`

To use `splunkadmins_splunkd_source` and/or `splunkadmins_metrics_source` macros

Author: gjanders

README.md

@@ -61,6 +61,8 @@ The macros are listed below, many expect a `host=A OR host=B` item to assist in
 The macros are used in various alerts which you can optionally enable, the alerts will raise a triggered alert only as emails are not allowed for Splunk app certification purposes
 The macros are also used in the dashboards for this application
 
+There are also other macros you might want to consider editing before enabling the alerts, for example `splunkadmins_replicationfactor`.
+
 The vast majority of the alerts also have a macro(s) which you can customise to tweak the search results, for example the macro `splunkadmins_weekly_truncated` allows the alert, `IndexerLevel - Weekly Truncated Logs Report`, to be customised without changing the alert itself. This will make upgrading to a new version of this app more straightforward
 I have attempted to provide an appropriate macro in any alert where I deemed it appropriate, feedback is welcome for any alert that you believe should have a macro or requires further improvement
 
@@ -207,6 +209,7 @@ The below list of alerts and reports are actively used since version 8.0.x and i
 - `IndexerLevel - platform_stats.indexers stddev incoming measurement`
 - `IndexerLevel - platform_stats.indexers totalgb measurement`
 - `IndexerLevel - platform_stats.indexers totalgb_thruput measurement`
+- `IndexerLevel - replicationdatareceiverthread close to 100% utilisation`
 - `IndexerLevel - RemoteSearches find datamodel acceleration with wildcards`
 - `IndexerLevel - RemoteSearches Indexes Stats`
 - `IndexerLevel - RemoteSearches Indexes Stats Wilcard`
@@ -303,6 +306,86 @@ The following ideas relate to this issue:
 Feel free to open an issue on github or use the contact author on the SplunkBase link and I will try to get back to you when possible, thanks!
 
 ## Release Notes
+### 3.0.2
+Merged pull request from jeffland-consist via github including various changes
+
+New alerts:
+- `IndexerLevel - replicationdatareceiverthread close to 100% utilisation`
+
+New macros:
+- `splunkadmins_metrics_source`
+- `splunkadmins_hec_metrics_source`
+
+New reports:
+- `SearchHeadLevel - Accelerated DataModels Access Info`
+- `SearchHeadLevel - Dashboards resulting in concurrency issues`
+- `SearchHeadLevel - Dashboards that may benefit from base or post-process searches`
+- `SearchHeadLevel - Searches by search type`
+
+Updated macros:
+- `splunkadmins_splunkd_source`
+- `splunkadmins_splunkuf_source`
+- `splunkadmins_mongo_source`
+- `splunkadmins_license_usage_source`
+
+To include a trailing wildcard (so splunkd.log.1 matches or similar)
+
+Updated alerts:
+- `AllSplunkEnterpriseLevel - Core Dumps Disabled` - updated matching criteria
+- `AllSplunkEnterpriseLevel - Non-existent roles are assigned to users` - updated matching criteria
+- `AllSplunkEnterpriseLevel - Splunk Servers throwing runScript errors` - updated matching criteria
+- `AllSplunkEnterpriseLevel - sendmodalert errors` - updated matching criteria
+- `AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only` - updated matching criteria
+- `AllSplunkEnterpriseLevel - Splunk Servers with resource starvation` - updated to use `splunkadmins_splunkd_source` macro
+- `AllSplunkLevel - No recent metrics.log data` - corrected comment to be after tstats, updated to use `splunkadmins_metrics_source` macro
+- `AllSplunkLevel - DeploymentServer Application Installation Error` - updated matching criteria
+- `DeploymentServer - Application Not Found On Deployment Server` - updated matching criteria
+- `ForwarderLevel - Channel churn issues` - updated to use `splunkadmins_metrics_source` macro
+- `ForwarderLevel - Forwarders connecting to a single endpoint for extended periods` - updated to use `splunkadmins_metrics_source` macro
+- `ForwarderLevel - Forwarders connecting to a single endpoint for extended periods UF level` - updated to use `splunkadmins_metrics_source` macro
+- `ForwarderLevel - Splunk HTTP Listener Overwhelmed` - updated matching criteria
+- `ForwarderLevel - Splunk Universal Forwarders Exceeding the File Descriptor Cache` - updated matching criteria
+- `ForwarderLevel - Splunk Universal Forwarders that are time shifting` - updated matching criteria
+- `ForwarderLevel - Stopping all listening ports` - updated to use `splunkadmins_splunkd_source` macro
+- `IndexerLevel - Buckets changes per day` - updated matching criteria, updated to use `splunkadmins_splunkd_source` macro
+- `IndexerLevel - Indexer Queues May Have Issues` - updated to use `splunkadmins_metrics_source` macro
+- `IndexerLevel - Knowledge bundle upload stats` - updated to use `splunkadmins_metrics_source` macro
+- `IndexerLevel - platform_stats.indexers totalgb_thruput measurement` - updated to use `splunkadmins_metrics_source` macro
+- `IndexerLevel - platform_stats.indexers stddev measurement` - updated to use `splunkadmins_metrics_source` macro
+- `IndexerLevel - platform_stats.indexers stddev incoming measurement` - updated to use `splunkadmins_metrics_source` macro
+- `IndexerLevel - Weekly Broken Events Report` - updated matching criteria
+- `IndexerLevel - Time format has changed multiple log types in one sourcetype` - updated matching criteria
+- `IndexerLevel - Buckets have being frozen due to index sizing` - updated matching criteria
+- `IndexerLevel - Unclean Shutdown - Fsck` - updated matching criteria
+- `IndexerLevel - Index not defined` - updated matching criteria
+- `IndexerLevel - Timestamp parsing issues combined alert` - updated to use `splunkadmins_splunkd_source` macro
+- `IndexerLevel - S2SFileReceiver Error` - updated matching criteria
+- `MonitoringConsole - Core dumps have appeared on the filesystem` - corrected to use `indexer_cluster_name` macro
+- `MonitoringConsole - Crash logs have appeared on the filesystem` - corrected description
+- `SearchHeadLevel - LDAP users have been disabled or left the company cleanup required` - updated matching criteria
+- `SearchHeadLevel - Long filenames may be causing issues` - updated matching criteria
+- `SearchHeadLevel - SHCluster Artifact Replication Issues` - updated matching criteria
+- `SearchHeadLevel - Captain Switchover Occurring` - updated matching criteria
+- `SearchHeadLevel - Knowledge bundle replication times metrics.log` - updated to use `splunkadmins_metrics_source` macro
+- `SearchHeadLevel - Detect bundle pushes no longer occurring` - updated to use `splunkadmins_metrics_source` macro
+- `SearchHeadLevel - WLM aborted searches` - updated matching criteria
+- `SearchHeadLevel - SHC Captain unable to establish common bundle` - updated to use `splunkadmins_splunkd_source` macro
+
+Updated dashboards:
+- `ClusterMasterJobs.xml`
+- `heavyforwarders_max_data_queue_sizes_by_name.xml`
+- `heavyforwarders_max_data_queue_sizes_by_name_v8.xml`
+- `hec_performance.xml`
+- `indexer_data_spread.xml`
+- `indexer_max_data_queue_sizes_by_name.xml`
+- `indexer_max_data_queue_sizes_by_name_v8.xml`
+- `rolled_buckets_by_index.xml`
+- `smartstore_stats.xml`
+- `splunk_forwarder_data_balance_tuning.xml`
+- `splunk_forwarder_output_tuning.xml`
+
+To use `splunkadmins_splunkd_source` and/or `splunkadmins_metrics_source` macros
+
 ### 3.0.1
 New macros:
 - `splunkadmins_shutdown_time_by_period`

app.manifest

@@ -5,7 +5,7 @@
     "id": {
       "group": null,
       "name": "SplunkAdmins",
-      "version": "3.0.1"
+      "version": "3.0.2"
     },
     "author": [
       {

default/app.conf

@@ -12,7 +12,7 @@ label = SplunkAdmins
 [launcher]
 author = Gareth Anderson
 description = Alerts and dashboards as described in the Splunk 2017 conf presentation How did you get so big?
-version = 3.0.1
+version = 3.0.2
 
 [package]
 id = SplunkAdmins

default/data/ui/nav/default.xml

@@ -161,6 +161,7 @@
 			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FIndexerLevel%20-%20IndexConfig%20Warnings%20from%20Splunk%20indexers">IndexConfig Warnings from Splunk indexers</a>
 			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FIndexerLevel%20-%20Index%20not%20defined">Index not defined</a>
 			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FForwarderLevel%20-%20Stopping%20all%20listening%20ports">ForwarderLevel - Stopping all listening ports</a>							
+			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FIndexerLevel%20-%20replicationdatareceiverthread%20close%20to%20100%25%20utilisation">IndexerLevel - replicationdatareceiverthread close to 100% utilisation</a>							
 		</collection>			
 		<collection label="Data Parsing">				
 			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FIndexerLevel%20-%20Failures%20To%20Parse%20Timestamp%20Correctly%20%28excluding%20breaking%20issues%29">Failures To Parse Timestamp Correctly (excluding breaking issues)</a>			
@@ -200,6 +201,7 @@
 				<view name="splunk_forwarder_data_balance_tuning" />
 				<view name="splunk_introspection_io_stats" />
   	                        <a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FForwarderLevel%20-%20Channel%20churn%20issues">ForwarderLevel - Channel churn issues</a>
+                                <a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FIndexerLevel%20-%20replicationdatareceiverthread%20close%20to%20100%25%20utilisation">IndexerLevel - replicationdatareceiverthread close to 100% utilisation</a>				
 			</collection>
 			<collection label="Other">
                                 <a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FIndexerLevel%20-%20Indexer%20not%20accepting%20TCP%20Connections">Indexer not accepting TCP Connections</a>		
@@ -288,13 +290,15 @@
                         <saved name="SearchHeadLevel - Search Queries summary exact match" />
                         <saved name="SearchHeadLevel - Search Queries summary exact match by user" />
                         <saved name="SearchHeadLevel - Search Queries summary exact match by index" />
+			<saved name="SearchHeadLevel - Searches by search type" />			
                         <saved name="SearchHeadLevel - IndexesPerUser Report" />			
                         <saved name="IndexerLevel - RemoteSearches Indexes Stats" />
-                        <saved name="IndexerLevel - RemoteSearches Indexes Stats Wilcard" />			
+                        <saved name="IndexerLevel - RemoteSearches Indexes Stats Wilcard" />
 		</collection>      
 		<collection label="Data Models">				
 			<saved name="SearchHeadLevel - Data Model Acceleration Completion Status" />
 			<saved name="SearchHeadLevel - DataModel Fields" />
+			<saved name="SearchHeadLevel - Accelerated DataModels Access Info" />
 			<saved name="IndexerLevel - DataModel Acceleration - Indexes in use" />
                         <a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FSearchHeadLevel%20-%20datamodel%20errors%20in%20splunkd">datamodel errors in splunkd</a>			
 			<view name="data_model_rebuild_monitor" />
@@ -339,6 +343,8 @@
 				<saved name="SearchHeadLevel - Dashboard refresh intervals" />
                                 <saved name="SearchHeadLevel - Dashboards using depends and running searches in the background" />
 				<saved name="SearchHeadLevel - Dashboards using special characters" />
+				<saved name="SearchHeadLevel - Dashboards resulting in concurrency issues" />
+				<saved name="SearchHeadLevel - Dashboards that may benefit from base or post-process searches" />				
 			</collection>			
 			<collection label="Scheduled Searches">
 				<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FSearchHeadLevel%20-%20Scheduled%20searches%20not%20specifying%20an%20index%20macro%20version">Scheduled searches not specifying an index macro version</a>

default/data/ui/views/ClusterMasterJobs.xml

@@ -18,7 +18,7 @@
       <title>Job Count</title>
       <chart>
         <search>
-          <query>index=_internal `splunkadmins_clustermaster_oshost` sourcetype=splunkd source=*splunkd.log *CMRepJob running job | timechart span=$span$ count by job</query>
+          <query>index=_internal `splunkadmins_clustermaster_oshost` sourcetype=splunkd `splunkadmins_splunkd_source` *CMRepJob running job | timechart span=$span$ count by job</query>
           <earliest>$time.earliest$</earliest>
           <latest>$time.latest$</latest>
           <sampleRatio>1</sampleRatio>
@@ -63,7 +63,7 @@
       <title>Fixup Jobs</title>
       <chart>
         <search>
-          <query>index=_internal source=*metrics.log sourcetype=splunkd name=cmmaster_service `splunkadmins_clustermaster_oshost` group=subtask_counts
+          <query>index=_internal `splunkadmins_metrics_source` sourcetype=splunkd name=cmmaster_service `splunkadmins_clustermaster_oshost` group=subtask_counts
 | timechart max(to_fix_gen), max(to_fix_rep_factor), max(to_fix_search_factor) span=$span$</query>
           <earliest>$time.earliest$</earliest>
           <latest>$time.latest$</latest>
@@ -104,4 +104,4 @@
       </chart>
     </panel>
   </row>
-</form>
+</form>

default/data/ui/views/heavyforwarders_max_data_queue_sizes_by_name.xml

@@ -22,7 +22,7 @@
       <title>Parsing Queue Fill Size</title>
       <chart>
         <search>
-          <query>index=_internal $hosts$ source=*metrics.log sourcetype=splunkd group=queue (name=parsingqueue)
+          <query>index=_internal $hosts$ `splunkadmins_metrics_source` sourcetype=splunkd group=queue (name=parsingqueue)
 | eval ingest_pipe = if(isnotnull(ingest_pipe), ingest_pipe, "none") | search ingest_pipe=* 
 | eval max=if(isnotnull(max_size_kb),max_size_kb,max_size) 
 | eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size) 
@@ -49,7 +49,7 @@
       <title>Aggregation Queue Fill Size</title>
       <chart>
         <search>
-          <query>index=_internal $hosts$ source=*metrics.log sourcetype=splunkd group=queue (name=aggqueue)
+          <query>index=_internal $hosts$ `splunkadmins_metrics_source` sourcetype=splunkd group=queue (name=aggqueue)
 | eval ingest_pipe = if(isnotnull(ingest_pipe), ingest_pipe, "none") | search ingest_pipe=* 
 | eval max=if(isnotnull(max_size_kb),max_size_kb,max_size) 
 | eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size) 
@@ -72,7 +72,7 @@
       <title>Typing Queue Fill Size</title>
       <chart>
         <search>
-          <query>index=_internal $hosts$ source=*metrics.log sourcetype=splunkd group=queue (name=typingqueue)
+          <query>index=_internal $hosts$ `splunkadmins_metrics_source` sourcetype=splunkd group=queue (name=typingqueue)
 | eval ingest_pipe = if(isnotnull(ingest_pipe), ingest_pipe, "none") | search ingest_pipe=* 
 | eval max=if(isnotnull(max_size_kb),max_size_kb,max_size) 
 | eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size) 
@@ -95,7 +95,7 @@
       <title>Index Queue Size</title>
       <chart>
         <search>
-          <query>index=_internal $hosts$ source=*metrics.log sourcetype=splunkd group=queue (name=indexqueue)
+          <query>index=_internal $hosts$ `splunkadmins_metrics_source` sourcetype=splunkd group=queue (name=indexqueue)
 | eval name=case(name=="aggqueue","2 - Aggregation Queue",
  name=="indexqueue", "4 - Indexing Queue",
  name=="parsingqueue", "1 - Parsing Queue",
@@ -143,7 +143,7 @@
       <title>TCPOut Queue Sizes</title>
       <chart>
         <search>
-          <query>index=_internal $hosts$ source=*metrics.log sourcetype=splunkd group=queue (name=tcpout_*)
+          <query>index=_internal $hosts$ `splunkadmins_metrics_source` sourcetype=splunkd group=queue (name=tcpout_*)
 | eval ingest_pipe = if(isnotnull(ingest_pipe), ingest_pipe, "none") | search ingest_pipe=* 
 | eval max=if(isnotnull(max_size_kb),max_size_kb,max_size) 
 | eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size) 
@@ -188,7 +188,7 @@
       <title>Blocked Forwarder Queues</title>
       <chart>
         <search>
-          <query>index=_internal $hosts$ source=*metrics.log sourcetype=splunkd group=queue max_size_kb&gt;0 | stats count(eval(isnotnull(blocked))) AS blockedCount, count by name, host, _time | eval percBlocked=(100/count)*blockedCount | eval hostQueue = host . "_" . name | where percBlocked&gt;0 | timechart limit=50 useOther=false span=$span$ avg(percBlocked) by hostQueue</query>
+          <query>index=_internal $hosts$ `splunkadmins_metrics_source` sourcetype=splunkd group=queue max_size_kb&gt;0 | stats count(eval(isnotnull(blocked))) AS blockedCount, count by name, host, _time | eval percBlocked=(100/count)*blockedCount | eval hostQueue = host . "_" . name | where percBlocked&gt;0 | timechart limit=50 useOther=false span=$span$ avg(percBlocked) by hostQueue</query>
           <earliest>$time.earliest$</earliest>
           <latest>$time.latest$</latest>
           <sampleRatio>1</sampleRatio>
@@ -226,7 +226,7 @@
       <title>TcpOut KB per second per forwarder</title>
       <chart>
         <search>
-          <query>index=_internal $hosts$ source=*metrics.log sourcetype=splunkd group=thruput name=cooked_output OR name=uncooked_output
+          <query>index=_internal $hosts$ `splunkadmins_metrics_source` sourcetype=splunkd group=thruput name=cooked_output OR name=uncooked_output
 		  | timechart useother=false span=$span$ limit=20 per_second(kb) by host 
           </query>
           <earliest>$time.earliest$</earliest>