New reports:

- `SearchHeadLevel - Datamodel access summary`

Updated alerts:
- `AllSplunkEnterpriseLevel - File integrity check failure` - removed wildcard, feedback from Gregg Woodcock
- `AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only` - removed extra "AND", feedback from Gregg Woodcock

Updated reports:
- `SearchHeadLevel - Accelerated DataModels Access Info - updated description`
- `SearchHeadLevel - Datamodel REST endpoint indexes in use` - correct indexin multivalued extraction
- `SearchHeadLevel - indexes per savedsearch` - correct indexin multivalued extraction
- `SearchHeadLevel - Indexes for savedsearch without subsearches` - correct indexin multivalued extraction
- `SearchHeadLevel - Lookups within savedsearches` - included the action.lookup.filename
- `SearchHeadLevel - Search Queries summary exact match` - correct indexin multivalued extraction
- `SearchHeadLevel - Search Queries summary non-exact match` - correct indexin multivalued extraction
- `SearchHeadLevel - SmartStore cache misses - dashboards` - correct indexin multivalued extraction
- `SearchHeadLevel - SmartStore cache misses - savedsearches` - correct indexin multivalued extraction
- `SearchHeadLevel - SmartStore cache misses - combined` - correct indexin multivalued extraction

Author: gjanders

README.md

@@ -359,6 +359,26 @@ These are appear to be from premium apps but it does imply that there is a mecha
 Feel free to open an issue on github or use the contact author on the SplunkBase link and I will try to get back to you when possible, thanks!
 
 ## Release Notes
+### 4.0.3
+New reports:
+- `SearchHeadLevel - Datamodel access summary`
+
+Updated alerts:
+- `AllSplunkEnterpriseLevel - File integrity check failure` - removed wildcard, feedback from Gregg Woodcock
+- `AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only` - removed extra "AND", feedback from Gregg Woodcock
+
+Updated reports:
+- `SearchHeadLevel - Accelerated DataModels Access Info - updated description`
+- `SearchHeadLevel - Datamodel REST endpoint indexes in use` - correct indexin multivalued extraction
+- `SearchHeadLevel - indexes per savedsearch` - correct indexin multivalued extraction
+- `SearchHeadLevel - Indexes for savedsearch without subsearches` - correct indexin multivalued extraction
+- `SearchHeadLevel - Lookups within savedsearches` - included the action.lookup.filename
+- `SearchHeadLevel - Search Queries summary exact match` - correct indexin multivalued extraction
+- `SearchHeadLevel - Search Queries summary non-exact match` - correct indexin multivalued extraction
+- `SearchHeadLevel - SmartStore cache misses - dashboards` - correct indexin multivalued extraction
+- `SearchHeadLevel - SmartStore cache misses - savedsearches` - correct indexin multivalued extraction
+- `SearchHeadLevel - SmartStore cache misses - combined` - correct indexin multivalued extraction
+
 ### 4.0.2
 Updated alerts:
 - `MonitoringConsole - one or more servers require configuration automated` - added missing \, issue #25 (thanks to barrettnet)

default/app.conf

@@ -14,7 +14,7 @@ supported_themes = light,dark
 [launcher]
 author = Gareth Anderson
 description = Alerts and dashboards as described in the Splunk 2017 conf presentation How did you get so big?
-version = 4.0.2
+version = 4.0.3
 
 [package]
 id = SplunkAdmins

default/data/ui/nav/default.xml

@@ -31,6 +31,7 @@
 			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FAllSplunkEnterpriseLevel%20-%20TCP%20or%20SSL%20Config%20Issue">TCP or SSL Config Issue</a> 
 			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FAllSplunkEnterpriseLevel%20-%20WARN%20iniFile%20Configuration%20Issues">WARN iniFile Configuration Issues</a> 
 			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FAllSplunkEnterpriseLevel%20-%20error%20in%20stdout">error in stdout.log</a>
+                        <a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FAllSplunkEnterpriseLevel%20-%20Splunkd%20Log%20Messages%20Admins%20Only">Splunkd Log Messages Admins Only</a>
 		</collection>	
 		<collection label="Splunk Level Failures">
 			<collection label="Deployment Server Related">
@@ -295,6 +296,7 @@
 			<saved name="SearchHeadLevel - audit.log - lookup usage" />
 			<saved name="SearchHeadLevel - Detect lookups that have not being accessed for a period of time" />
 			<saved name="SearchHeadLevel - Lookup Editor lookup updates" />
+			<saved name="SearchHeadLevel - Datamodel access summary" />
 			<saved name="SearchHeadLevel - indexes per savedsearch" />
 			<saved name="SearchHeadLevel - macros in use" />
 			<saved name="SearchHeadLevel - Search Queries Per Day Audit Logs" />
@@ -329,6 +331,7 @@
 			<saved name="SearchHeadLevel - Data Model Acceleration Completion Status" />
 			<saved name="SearchHeadLevel - DataModel Fields" />
 			<saved name="SearchHeadLevel - Accelerated DataModels Access Info" />
+			<saved name="SearchHeadLevel - Datamodel access summary" />
 			<saved name="SearchHeadLevel - Datamodel REST endpoint indexes in use" />			
 			<saved name="IndexerLevel - DataModel Acceleration - Indexes in use" />
                         <a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FSearchHeadLevel%20-%20datamodel%20errors%20in%20splunkd">datamodel errors in splunkd</a>			

default/savedsearches.conf

@@ -2443,7 +2443,7 @@ relation = greater than
 request.ui_dispatch_app = SplunkAdmins
 request.ui_dispatch_view = search
 search = ```One or more files did not pass the startup hash-check against the Splunk provided manifest, you can tune the limits.conf to control how the warning is logged or not logged```\
-index=_internal `splunkenterprisehosts` "An installed * did not pass hash-checking due to" (`splunkadmins_splunkd_source`) sourcetype=splunkd `splunkadmins_fileintegritycheck`\
+index=_internal `splunkenterprisehosts` "An installed" "did not pass hash-checking due to" (`splunkadmins_splunkd_source`) sourcetype=splunkd `splunkadmins_fileintegritycheck`\
 | eval message=coalesce(message,event_message)\
 | stats count, latest(_time) AS lastSeen by message, host\
 | eval lastSeen=strftime(lastSeen, "%+")
@@ -4714,8 +4714,7 @@ search = | multisearch \
   ```Extract out index= or index IN (a,b,c) but avoid NOT index in (...) and NOT index=... and also NOT (...anything) statements``` \
 | rex field=search "(?s)(NOT\s+index(\s*=\s*|::)[^ ]+)|(NOT\s+\([^\)]+\))|(index(\s*=\s*|::)\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
 | rex field=search "(?s)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexin>([^\)\"]+)|\"[^\)\"]+\"))" max_match=50 \
-| makemv delim="," indexin \
-| makemv delim=" " indexin \
+| makemv tokenizer="([^, ]+)" indexin \
 | eval indexes=mvappend(indexregex,indexin) \
 | eval indexes=if(isnotnull(esstylewildcard),mvfilter(NOT match(indexes,"^_?\*$")),indexes) \
 | eval wildcard=mvfilter(match(indexes,"\*")) \
@@ -4818,8 +4817,7 @@ search = | multisearch \
   ```Extract out index= or index IN (a,b,c) but avoid NOT index in (...) and NOT index=... and also NOT (...anything) statements``` \
 | rex field=search "(?s)(NOT\s+index(\s*=\s*|::)[^ ]+)|(NOT\s+\([^\)]+\))|(index(\s*=\s*|::)\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
 | rex field=search "(?s)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexin>([^\)\"]+)|\"[^\)\"]+\"))" max_match=50 \
-| makemv delim="," indexin \
-| makemv delim=" " indexin \
+| makemv tokenizer="([^, ]+)" indexin \
 | eval indexes=mvappend(indexregex,indexin) \
 | eval indexes=if(isnotnull(esstylewildcard),mvfilter(NOT match(indexes,"^_?\*$")),indexes) \
 | eval wildcard=mvfilter(match(indexes,"\*")) \
@@ -6507,7 +6505,7 @@ OR component=Saml OR component=FileClassifierManager OR component=HttpPubSubConn
 ```these may require more investigation. Ignoring for now Aug 2022``` NOT ("ERROR CacheManager" "No such file or directory") NOT ("ERROR BucketReplicator" "The bucket may have frozen") NOT ("BucketReplicator" "Failed to check the hotness of bucketId") \
  OR (sourcetype=scheduler source=*scheduler.log AlertNotifier WARN) \
  OR (sourcetype=splunkd (`splunkadmins_splunkd_source`) INFO (IndexWriter paused ```May relate to maxConcurrentOptimizes in indexes.conf or perhaps maxRunningProcessGroups or spikes in data-per indexer```) OR (component=HotDBManager "unflushed buckets") OR (TERM(event=reclaimMemory) IndexProcessor OR StreamingBucketBuilder ```May relate to memPoolMB / maxMemMB setting in indexes.conf or the IndexWriter getting paused. However data balance (too much MB/s of ingestion on a single indexer/uneven balance appears to cause this too)```)) \
-| search ```ignore shutdown times to remove errors that relate to shutdowns, note this may remove some legitimate alerts as well``` AND NOT [ `splunkadmins_shutdown_time_by_period(splunkenterprisehosts,60,60,10)` ] \
+| search ```ignore shutdown times to remove errors that relate to shutdowns, note this may remove some legitimate alerts as well``` NOT [ `splunkadmins_shutdown_time_by_period(splunkenterprisehosts,60,60,10)` ] \
 | eval search_head=host \
 | eval search_head_cluster=`search_head_cluster` \
 | search ```Exclude time periods where shutdowns were occurring. While this makes the alert less nosiy it removes some legitimate errors too``` NOT \
@@ -7023,7 +7021,7 @@ invocations_command_search_index_bucketcache_miss>0 OR invocations_command_searc
 | eval total_hours_searched=round(total_hours_searched,1) \
 | rex field=search "(?s)(NOT\s+index(\s*=\s*|::)[^ ]+)|(NOT\s+\([^\)]+\))|(index(\s*=\s*|::)\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
 | rex field=search "(?s)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexin>([^\)\"]+)|\"[^\)\"]+\"))" max_match=50 \
-| makemv delim="," indexin \
+| makemv tokenizer="([^, ]+)" indexin \
 | eval indexes=mvappend(indexregex,indexin) \
 | eval indexes=mvmap(indexes, replace(lower(indexes), "\"", "")) \
 | eval indexes=mvmap(indexes, trim(replace(indexes, "'", ""))) \
@@ -7078,7 +7076,7 @@ invocations_command_search_index_bucketcache_miss>0 OR invocations_command_searc
 | stats latest(mostRecent) AS mostRecent, count as number_of_runs values(host) as host values(total_hours_searched) AS total_hours_searched values(total_days_searched) AS total_days_searched max(run_time) AS max_run_time avg(run_time) AS avg_run_time sum(run_time) AS sum_run_time sum(total_cache_miss) as total_cache_miss max(result_count) AS result_count max(event_count) AS event_count max(searched_buckets) AS searched_buckets values(info) AS info values(numofsearchesinquery) AS numofsearchesinquery, values(app) AS app by users search \
 | rex field=search "(?s)(NOT\s+index(\s*=\s*|::[^ ]+)|(NOT\s+\([^\)]+\))|(index(\s*=\s*|::)\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
 | rex field=search "(?s)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexin>([^\)\"]+)|\"[^\)\"]+\"))" max_match=50 \
-| makemv delim="," indexin \
+| makemv tokenizer="([^, ]+)" indexin \
 | eval indexes=mvappend(indexregex,indexin) \
 | eval indexes=mvmap(indexes, replace(lower(indexes), "\"", "")) \
 | eval indexes=mvmap(indexes, trim(replace(indexes, "'", ""))) \
@@ -7133,7 +7131,7 @@ invocations_command_search_index_bucketcache_miss>0 OR invocations_command_searc
 | stats latest(mostRecent) AS mostRecent, count as number_of_runs, values(host) as host values(total_hours_searched) AS total_hours_searched values(total_days_searched) AS total_days_searched max(run_time) AS max_run_time avg(run_time) AS avg_run_time sum(run_time) AS sum_run_time sum(total_cache_miss) as total_cache_miss max(result_count) AS result_count max(event_count) AS event_count max(searched_buckets) AS searched_buckets values(info) AS info values(numofsearchesinquery) AS numofsearchesinquery, values(provenance) AS provenance, values(app) AS app by users search \
 | rex field=search "(?s)(NOT\s+index(\s*=\s*|::)[^ ]+)|(NOT\s+\([^\)]+\))|(index(\s*=\s*|::)\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
 | rex field=search "(?s)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexin>([^\)\"]+)|\"[^\)\"]+\"))" max_match=50 \
-| makemv delim="," indexin \
+| makemv tokenizer="([^, ]+)" indexin \
 | eval indexes=mvappend(indexregex,indexin) \
 | eval indexes=mvmap(indexes, replace(lower(indexes), "\"", "")) \
 | eval indexes=mvmap(indexes, trim(replace(indexes, "'", ""))) \
@@ -7595,7 +7593,7 @@ disabled = 1
 action.email.useNSSubject = 1
 alert.track = 0
 cron_schedule = 38 * * * *
-description = Report only? Yes. As found on Clara-Fication: Finding and Improving Expensive Searches, https://conf.splunk.com/files/2022/slides/PLA1162B.pdf / https://conf.splunk.com/files/2022/recordings/PLA1162B_1080.mp4. Run on the search head with the DMA
+description = Report only? Yes. As found on Clara-Fication: Finding and Improving Expensive Searches, https://conf.splunk.com/files/2022/slides/PLA1162B.pdf / https://conf.splunk.com/files/2022/recordings/PLA1162B_1080.mp4. Run on the search head with the DMA. Also refer to SearchHeadLevel - Datamodel access summary for a more detailed view.
 dispatch.earliest_time = -65m@m
 dispatch.latest_time = -5m@m
 display.events.fields = ["index","sourcetype","host"]
@@ -8112,7 +8110,8 @@ display.general.type = statistics
 enableSched = 0
 request.ui_dispatch_app = SplunkAdmins
 request.ui_dispatch_view = search
-search = | rest `splunkadmins_restmacro` timeout=900 /servicesNS/-/-/saved/searches f=search f=eai:acl* \
+search = | rest `splunkadmins_restmacro` timeout=900 /servicesNS/-/-/saved/searches f=search f=eai:acl* f=action.lookup.filename \
+| eval search=if(isnotnull('action.lookup.filename'),search . "| outputlookup " . 'action.lookup.filename',search) \
 | regex search="((input|output)?lookup)|(\|\s+apply\s+)" \
 | fields title, search eai:acl.app eai:acl.sharing \
 | rename eai:acl.app AS app, eai:acl.sharing AS sharing \
@@ -8228,7 +8227,7 @@ search = | rest `splunkadmins_restmacro` timeout=900 /servicesNS/-/-/data/models
 | rex field=eai:data "(?P<esstylewildcard>\(\s*index=\*\s+OR\s+index=_\*\s*\))" \
 | rex field=eai:data "(?sm)(NOT\s+index\s*(=|::)\s*[^ ]+)|(NOT\s+\([^\)]+\))|(index\s*(=|::)\s*(\\\)?\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
 | rex field=eai:data "(?sm)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexin>([^\)\"]+)|\"[^\)\"]+\"))" max_match=50 \
-| makemv delim="," indexin \
+| makemv tokenizer="([^, ]+)" indexin \
 | eval indexes=mvappend(indexregex,indexin) \
 | eval indexes=if(isnotnull(esstylewildcard),mvfilter(NOT match(indexes,"^_?\*$")),indexes) \
 | eval indexes=mvmap(indexes, replace(lower(indexes), "\"", "")) \
@@ -8475,8 +8474,7 @@ search = | rest /servicesNS/-/-/saved/searches f=next_scheduled_time f=search f=
 | eval prepipe = prepipe . " " . prepipe_subsearch \
 | rex field=prepipe "(?s)(NOT\s+index(\s*=\s*|::)[^ ]+)|(NOT\s+\([^\)]+\))|(index(\s*=\s*|::)\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
 | rex field=prepipe "(?s)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexin>([^\)\"]+)|\"[^\)\"]+\"))" max_match=50 \
-| makemv delim="," indexin \
-| makemv delim=" " indexin \
+| makemv tokenizer="([^, ]+)" indexin \
 | eval indexes=mvappend(indexregex,indexin) \
 | eval indexes=mvmap(indexes, replace(lower(indexes), "\"", "")) \
 | eval indexes=mvmap(indexes, trim(replace(indexes, "'", ""))) \
@@ -8575,7 +8573,7 @@ search = index=_audit savedsearch_name="$savedsearch_name$" host IN ($host$) \
 | rex field=search "(?s)^(?P<prepipe>\s*\|?([^\|]+))" \
 | rex field=prepipe "(?s)(NOT\s+index(\s*=\s*|::)[^ ]+)|(NOT\s+\([^\)]+\))|(index(\s*=\s*|::)\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
 | rex field=prepipe "(?s)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexin>([^\)\"]+)|\"[^\)\"]+\"))" max_match=50 \
-| makemv delim="," indexin \
+| makemv tokenizer="([^, ]+)" indexin \
 | eval indexes=mvappend(indexregex,indexin) \
 | eval indexes=if(isnotnull(esstylewildcard),mvfilter(NOT match(indexes,"^_?\*$")),indexes) \
 | eval wildcard=mvfilter(match(indexes,"\*")) \
@@ -8785,3 +8783,30 @@ request.ui_dispatch_view = search
 search = | rest /servicesNS/-/-/saved/searches count=0 search="disabled=0" search="is_scheduled=1" f=next_scheduled_time `splunkadmins_restmacro` f=title f=eai:* \
 | search next_scheduled_time="" \
 | table author, eai:acl.app, , title, next_scheduled_time 
+
+[SearchHeadLevel - Datamodel access summary]
+action.email.useNSSubject = 1
+alert.track = 0
+cron_schedule = 38 * * * *
+description = Report only? Yes. This report is based on the query in Splunk community slack provided by Ismo Soutamo. This query returns a summary of datamodels, acceleration status and if accelerated, access count and time. Similar to SearchHeadLevel - Accelerated DataModels Access Info. Run on the search head with DMA enabled.
+dispatch.earliest_time = -65m@m
+dispatch.latest_time = -5m@m
+display.events.fields = ["index","sourcetype","host"]
+display.general.type = statistics
+enableSched = 0
+request.ui_dispatch_app = SplunkAdmins
+request.ui_dispatch_view = search
+search =  | rest splunk_server=local timeout=60 /servicesNS/-/-/datamodel/model f=eai:* f=acceleration f=displayName \
+| fields title displayName author eai:acl.app eai:appName eai:acl.perms.read eai:acl.sharing splunk_server acceleration updated \
+| search acceleration = "*true*" \
+| eval DM="tstats:DM_" . 'eai:acl.app' . "_" . title \
+| join DM type=outer \
+    [| rest splunk_server=local timeout=60 /servicesNS/-/-/admin/summarization by_tstats=1 f=summary.access_count f=summary.access_time f=summary.size \
+    | search summary.access_count > 0 \
+    | table title summary.access_count summary.access_time summary.size \
+    | rename title as DM] \
+| spath input=acceleration \
+| rename eai:acl.* -> *\
+| rename enabled AS acceleration_enabled\
+| table title author app summary.access_count summary.access_time summary.size perms.read sharing updated acceleration_enabled earliest_time, cron_schedule, max_time, backfill_time, max_concurrent, allow_skew, allow_old_summaries\
+| eval summary.access_time=strftime('summary.access_time', "%+")