New alerts:

- `SearchHeadLevel - summary indexing searches not using durable search`

New macros:
- `indexer_cluster_name` without any parameters created as per issue #19 (barrettnet)

New reports:
- `SearchHeadLevel - audit.log - lookup usage`
- `SearchHeadLevel - license usage per sourcetype per index`
- `SearchHeadLevel - Lookup file owners`
- `IndexerLevel - RemoteSearches - lookup usage`

Updated alerts:
- `AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only` - more matching criteria
- `SearchHeadLevel - Scheduled Searches That Cannot Run` - as per issue #18 (AHCL1)
- `SearchHeadLevel - SHC Captain unable to establish common bundle` - additional exclusion for Splunk 9.0.x

Updated reports:
- `IndexerLevel - platform_stats.indexers totalgb measurement` - added * to the end of `license_usage.log`, updated `indexer_cluster_name` with parameter as per issue #19 (barrettnet)
- `IndexerLevel - platform_stats.indexers totalgb_thruput measurement` - updated `indexer_cluster_name` with parameter as per issue #19 (barrettnet)
- `SearchHeadLevel - Search Queries summary exact match` - removed newlines to improve accuracy
- `SearchHeadLevel - Search Queries summary non-exact match` - removed newlines to improve accuracy

Updated recommended links in nav menu

Author: gjanders

README.md

@@ -173,6 +173,7 @@ There are many Splunk conf talks available on this subject in various conference
 - `SearchHeadLevel - Searches dispatched as owner by other users`
 - `SearchHeadLevel - Search Messages user level`
 - `SearchHeadLevel - audit logs showing all time searches`
+- `SearchHeadLevel - summary indexing searches not using durable search`
 
 Are all well suited to an automated email using the sendresults command or a similar function as they involve end user configuration which the individual can change/fix
 
@@ -215,12 +216,14 @@ The below list of alerts and reports are actively used since version 8.0.x and i
 - `IndexerLevel - RemoteSearches find datamodel acceleration with wildcards`
 - `IndexerLevel - RemoteSearches Indexes Stats`
 - `IndexerLevel - RemoteSearches Indexes Stats Wilcard`
+- `IndexerLevel - RemoteSearches - lookup usage`
 - `IndexerLevel - Search Failures`
 - `IndexerLevel - Slow peer from remote searches`
 - `IndexerLevel - strings_metadata triggering bucket rolling`
 - `MonitoringConsole - Check OS ulimits via REST`
 - `MonitoringConsole - Core dumps have appeared on the filesystem`
 - `MonitoringConsole - Crash logs have appeared on the filesystem`
+- `SearchHeadLevel - audit.log - lookup usage`
 - `SearchHeadLevel - authorize.conf settings will prevent some users from appearing in the UI`
 - `SearchHeadLevel - Captain Switchover Occurring`
 - `SearchHeadLevel - Dashboards invalid character in splunkd`
@@ -233,7 +236,9 @@ The below list of alerts and reports are actively used since version 8.0.x and i
 - `SearchHeadLevel - Detect searches hitting corrupt buckets`
 - `SearchHeadLevel - dispatch metadata files may need removal`
 - `SearchHeadLevel - Excessive REST API usage`
+- `SearchHeadLevel - Knowledge Bundle contents`
 - `SearchHeadLevel - KVStore Or Conf Replication Issues Are Occurring`
+- `SearchHeadLevel - license usage per sourcetype per index`
 - `SearchHeadLevel - platform_stats access summary`
 - `SearchHeadLevel - platform_stats.audit metrics api`
 - `SearchHeadLevel - platform_stats.audit metrics searches`
@@ -255,6 +260,7 @@ The below list of alerts and reports are actively used since version 8.0.x and i
 - `SearchHeadLevel - SHC Captain unable to establish common bundle`
 - `SearchHeadLevel - Splunk alert actions exceeding the max_action_results limit`
 - `SearchHeadLevel - Splunk Scheduler logs have not appeared in the last`
+- `SearchHeadLevel - summary indexing searches not using durable search`
 - `SearchHeadLevel - Users exceeding the disk quota`
 - `syslog-ng - cache statistics summary`
 
@@ -309,6 +315,33 @@ The following ideas relate to this issue:
 Feel free to open an issue on github or use the contact author on the SplunkBase link and I will try to get back to you when possible, thanks!
 
 ## Release Notes
+### 3.0.8
+New alerts:
+- `SearchHeadLevel - summary indexing searches not using durable search`
+
+New macros:
+- `indexer_cluster_name` without any parameters created as per issue #19 (barrettnet)
+
+New reports:
+- `SearchHeadLevel - audit.log - lookup usage`
+- `SearchHeadLevel - license usage per sourcetype per index`
+- `SearchHeadLevel - Lookup file owners`
+- `IndexerLevel - RemoteSearches - lookup usage`
+
+Updated alerts:
+- `AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only` - more matching criteria
+- `SearchHeadLevel - Scheduled Searches That Cannot Run` - as per issue #18 (AHCL1)
+- `SearchHeadLevel - SHC Captain unable to establish common bundle` - additional exclusion for Splunk 9.0.x
+
+
+Updated reports:
+- `IndexerLevel - platform_stats.indexers totalgb measurement` - added * to the end of `license_usage.log`, updated `indexer_cluster_name` with parameter as per issue #19 (barrettnet)
+- `IndexerLevel - platform_stats.indexers totalgb_thruput measurement` - updated `indexer_cluster_name` with parameter as per issue #19 (barrettnet)
+- `SearchHeadLevel - Search Queries summary exact match` - removed newlines to improve accuracy
+- `SearchHeadLevel - Search Queries summary non-exact match` - removed newlines to improve accuracy
+
+Updated recommended links in nav menu
+
 ### 3.0.7
 New macros:
 - `sysloghosts`

default/app.conf

@@ -12,7 +12,7 @@ label = SplunkAdmins
 [launcher]
 author = Gareth Anderson
 description = Alerts and dashboards as described in the Splunk 2017 conf presentation How did you get so big?
-version = 3.0.7
+version = 3.0.8
 
 [package]
 id = SplunkAdmins

default/data/ui/nav/default.xml

@@ -166,6 +166,7 @@
 			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FIndexerLevel%20-%20Index%20not%20defined">Index not defined</a>
 			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FForwarderLevel%20-%20Stopping%20all%20listening%20ports">ForwarderLevel - Stopping all listening ports</a>							
 			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FIndexerLevel%20-%20replicationdatareceiverthread%20close%20to%20100%25%20utilisation">IndexerLevel - replicationdatareceiverthread close to 100% utilisation</a>							
+			<saved name="SearchHeadLevel - license usage per sourcetype per index" />
 		</collection>			
 		<collection label="Data Parsing">				
 			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FIndexerLevel%20-%20Failures%20To%20Parse%20Timestamp%20Correctly%20%28excluding%20breaking%20issues%29">Failures To Parse Timestamp Correctly (excluding breaking issues)</a>			
@@ -238,6 +239,7 @@
 			<saved name="IndexerLevel - Maximum memory utilisation per search" />
                         <saved name="IndexerLevel - RemoteSearches find all time searches" />
 			<saved name="IndexerLevel - RemoteSearches find datamodel acceleration with wildcards" />
+			<saved name="IndexerLevel - RemoteSearches - lookup usage" />			
 			<collection label="SmartStore">
 				<saved name="SearchHeadLevel - SmartStore cache misses - savedsearches" />
 	                        <saved name="SearchHeadLevel - SmartStore cache misses - dashboards" />	
@@ -268,6 +270,7 @@
                     <saved name="SearchHeadLevel - platform_stats.user_stats.introspection metrics populating search" />
                     <saved name="SearchHeadLevel - platform_stats access summary" />
 		    <saved name="SearchHeadLevel - platform_stats.remote_searches metrics populating search" />
+		    <saved name="SearchHeadLevel - audit.log - lookup usage" />
                     <saved name="IndexerLevel - platform_stats.counters hosts" />
                     <saved name="IndexerLevel - platform_stats.counters hosts 24hour" />
                     <saved name="IndexerLevel - platform_stats.indexers totalgb measurement" />
@@ -276,6 +279,7 @@
 		    <saved name="IndexerLevel - platform_stats.indexers stddev incoming measurement" />	
                     <saved name="IndexerLevel - RemoteSearches Indexes Stats" />
 		    <saved name="IndexerLevel - RemoteSearches Indexes Stats Wilcard" />
+		    <saved name="IndexerLevel - RemoteSearches - lookup usage" />
                 </collection>
 		<collection label="External">
 			<a href="https://github.com/silkyrich/cluster_health_tools/">The cluster_health_tools git repository contains very useful dashboards for various indexer related performance stats</a>
@@ -286,7 +290,8 @@
 	</collection>
 	<collection label="SearchHeadLevel">
 		<collection label="Analytics">				
-             	        <saved name="SearchHeadLevel - Search Queries Per Day Audit Logs" />
+			<saved name="SearchHeadLevel - audit.log - lookup usage" />
+			<saved name="SearchHeadLevel - Search Queries Per Day Audit Logs" />
                         <saved name="SearchHeadLevel - Search Queries By Type Audit Logs" />         
                         <saved name="SearchHeadLevel - Search Queries By Type Audit Logs macro version" />                   
                         <saved name="SearchHeadLevel - Search Queries By Type Audit Logs macro version other" />                             
@@ -297,9 +302,12 @@
                         <saved name="SearchHeadLevel - Search Queries summary exact match by index" />
 			<saved name="SearchHeadLevel - Sourcetypes usage from search telemetry data" />
 			<saved name="SearchHeadLevel - Searches by search type" />			
-                        <saved name="SearchHeadLevel - IndexesPerUser Report" />			
+                        <saved name="SearchHeadLevel - IndexesPerUser Report" />
+			<saved name="SearchHeadLevel - license usage per sourcetype per index" />
+			<saved name="SearchHeadLevel - Lookup file owners" />
                         <saved name="IndexerLevel - RemoteSearches Indexes Stats" />
-                        <saved name="IndexerLevel - RemoteSearches Indexes Stats Wilcard" />
+                        <saved name="IndexerLevel - RemoteSearches Indexes Stats Wilcard" />			
+			<saved name="IndexerLevel - RemoteSearches - lookup usage" />
 		</collection>      
 		<collection label="Data Models">				
 			<saved name="SearchHeadLevel - Data Model Acceleration Completion Status" />
@@ -361,6 +369,7 @@
 				<saved name="SearchHeadLevel - SavedSearches using special characters" />
 				<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FSearchHeadLevel%20-%20Splunk%20alert%20actions%20exceeding%20the%20max_action_results%20limit">Splunk alert actions exceeding the max_action_results limit</a>
 				<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FSearchHeadLevel%20-%20Splunk%20Scheduler%20logs%20have%20not%20appeared%20in%20the%20last">Splunk Scheduler logs have not appeared in the last</a>
+				<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FSearchHeadLevel%20-%20summary%20indexing%20searches%20not%20using%20durable%20search">SearchHeadLevel - summary indexing searches not using durable search</a>
 	  		</collection>									
 			<collection label="Other">
 				<saved name="SearchHeadLevel - Knowledge bundle replication times metrics.log" />							
@@ -392,6 +401,7 @@
 			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FSearchHeadLevel%20-%20Splunk%20login%20attempts%20from%20users%20that%20do%20not%20have%20any%20LDAP%20roles">Splunk login attempts from users that do not have any LDAP roles</a>
 			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FSearchHeadLevel%20-%20authorize.conf%20settings%20will%20prevent%20some%20users%20from%20appearing%20in%20the%20UI">SearchHeadLevel - authorize.conf settings will prevent some users from appearing in the UI</a>
 			<saved name="SearchHeadLevel - Knowledge Bundle contents" />
+		        <a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FSearchHeadLevel%20-%20summary%20indexing%20searches%20not%20using%20durable%20search">SearchHeadLevel - summary indexing searches not using durable search</a>			
 		</collection>	
 		<collection label="Quotas">			
 			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FSearchHeadLevel%20-%20Splunk%20Max%20Historic%20Search%20Limits%20Reached">Splunk Max Historic Search Limits Reached</a>
@@ -425,8 +435,10 @@
 			<saved name="SearchHeadLevel - Searches dispatched as owner by other users" />
 			<saved name="SearchHeadLevel - Lookup CSV size" />
 			<saved name="SearchHeadLevel - audit logs showing all time searches" />
+			<saved name="SearchHeadLevel - audit.log - lookup usage" />
                         <saved name="IndexerLevel - RemoteSearches find all time searches" />
 			<saved name="IndexerLevel - RemoteSearches find datamodel acceleration with wildcards" />
+			<saved name="IndexerLevel - RemoteSearches - lookup usage" />			
 			<saved name="SearchHeadLevel - Search Messages field extractor slow" />
 			<saved name="SearchHeadLevel - SmartStore cache misses - savedsearches" />
                         <saved name="SearchHeadLevel - SmartStore cache misses - dashboards" />	
@@ -436,12 +448,16 @@
 			<view name="knowledge_objects_by_app" />
 			<view name="lookups_in_use_finder" />
 			<view name="lookup_audit" />
+			<saved name="SearchHeadLevel - Lookup file owners" />
 			<saved name="SearchHeadLevel - Knowledge bundle status on indexers" />
 			<saved name="SearchHeadLevel - Knowledge bundle replication times metrics.log" />		
 			<saved name="SearchHeadLevel - Knowledge Bundle contents" />
+			<saved name="SearchHeadLevel - license usage per sourcetype per index" />
 			<saved name="syslog-ng - cache statistics summary" />
 		</collection>	
 		<collection label="Summary_Reports">
+			<saved name="SearchHeadLevel - audit.log - lookup usage" />
+			<saved name="SearchHeadLevel - license usage per sourcetype per index" />
         		<saved name="SearchHeadLevel - platform_stats.audit metrics searches" />
 	                <saved name="SearchHeadLevel - platform_stats.audit metrics users" />
                         <saved name="SearchHeadLevel - platform_stats.audit metrics users 24hour" />
@@ -450,15 +466,16 @@
      			<saved name="SearchHeadLevel - platform_stats.audit metrics api" />			
                         <saved name="SearchHeadLevel - platform_stats.user_stats.introspection metrics populating search" />
                         <saved name="SearchHeadLevel - platform_stats access summary" />
-			<saved name="SearchHeadLevel - platform_stats.remote_searches metrics populating search" />
+			<saved name="SearchHeadLevel - platform_stats.remote_searches metrics populating search" />			
 			<saved name="IndexerLevel - platform_stats.counters hosts" />
 			<saved name="IndexerLevel - platform_stats.counters hosts 24hour" />
 			<saved name="IndexerLevel - platform_stats.indexers totalgb measurement" />
 	                <saved name="IndexerLevel - platform_stats.indexers totalgb_thruput measurement" />
 	                <saved name="IndexerLevel - platform_stats.indexers stddev measurement" />
 	                <saved name="IndexerLevel - platform_stats.indexers stddev incoming measurement" />			
 	                <saved name="IndexerLevel - RemoteSearches Indexes Stats" />
-			<saved name="IndexerLevel - RemoteSearches Indexes Stats Wilcard" />
+			<saved name="IndexerLevel - RemoteSearches Indexes Stats Wilcard" />			
+			<saved name="IndexerLevel - RemoteSearches - lookup usage" />			
 		</collection>
 		<collection label="Scheduled Search Failures">			
 			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FSearchHeadLevel%20-%20Scheduled%20searches%20failing%20in%20cluster%20with%20404%20error">Scheduled searches failing in cluster with 404 error</a>
@@ -477,16 +494,19 @@
 			<saved name="SearchHeadLevel - EventTypes report" />
 			<saved name="SearchHeadLevel - Users exceeding the disk quota introspection cleanup" />                      
                         <saved name="SearchHeadLevel - RMD5 to savedsearch_name lookupgen report" />
+			<saved name="SearchHeadLevel - Lookup file owners" />
 		</collection>	
 		<collection label="Recommended (externally hosted)">
 			<a href="https://github.com/dpaper-splunk/public/tree/master/dashboards" target="_blank">Extended Search Reporting (and others)</a>
 			<a href="https://github.com/nicovdw/splunk_concurrency_helper" target="_blank">Search Scheduler Tuning searches</a>
 			<a href="https://splunkbase.splunk.com/app/6449/" target="_blank">Sideview UI (User Activity details)</a>
 			<a href="https://splunkbase.splunk.com/app/6368/" target="_blank">Admins Little Helper for Splunk (btool, bundle utils and similar)</a>
 			<a href="https://splunkbase.splunk.com/app/4621/" target="_blank">TrackMe (Data Ingestion)</a>
+                        <a href="https://github.com/redvelociraptor/gettingsmarter/tree/main">Getting Smarter about Splunk SmartStore (including HEC dashboards)</a>
 		</collection>
 	</collection>	
         <collection label="Summary_Reports">
+		<saved name="SearchHeadLevel - audit.log - lookup usage" />		
 		<saved name="SearchHeadLevel - platform_stats.audit metrics searches" />
                 <saved name="SearchHeadLevel - platform_stats.audit metrics users" />
                 <saved name="SearchHeadLevel - platform_stats.audit metrics api" />
@@ -503,7 +523,8 @@
                 <saved name="IndexerLevel - platform_stats.indexers stddev measurement" />		
                 <saved name="IndexerLevel - platform_stats.indexers stddev incoming measurement" />		
 		<saved name="IndexerLevel - RemoteSearches Indexes Stats" />
-		<saved name="IndexerLevel - RemoteSearches Indexes Stats Wilcard" />
+		<saved name="IndexerLevel - RemoteSearches Indexes Stats Wilcard" />		
+		<saved name="IndexerLevel - RemoteSearches - lookup usage" />
         </collection>
         <collection label="Users">
             <saved name="What Access Do I Have Without REST?" />

default/macros.conf

@@ -667,6 +667,11 @@ args = indexer
 definition = "default"
 iseval = 0
 
+#Macro to define indexer cluster name
+[indexer_cluster_name]
+definition = "default"
+iseval = 0
+
 [forwarder_name(1)]
 args = hostname
 definition = "default"
@@ -884,3 +889,8 @@ iseval = 0
 [splunkadmins_hec_metrics_source]
 definition = source=*http_event_collector_metrics.log*
 iseval = 0
+
+[splunkadmins_summaryindex_durablesearch]
+definition = NOT title IN ("SearchHeadLevel - summary indexing searches not using durable search") next_scheduled_time!=""
+iseval = 0
+