New macros:

- `indexes_extraction(1)` - to extract indexes from search logs

Updated reports/alerts:
- `AllSplunkEnterpriseLevel - Splunkd Crash Logs Have Appeared in Production` - updated based on email feedback to use sourcetype (as source matching needed wildcards)
- `IndexerLevel - Slow peer from remote searches` - corrected comment in search only
- `SearchHeadLevel - Search Queries summary exact match`
- `SearchHeadLevel - Search Queries summary non-exact match`
- `SearchHeadLevel - SmartStore cache misses - dashboards`
- `SearchHeadLevel - SmartStore cache misses - savedsearches`
- `SearchHeadLevel - SmartStore cache misses - combined`
- `SearchHeadLevel - Datamodel REST endpoint indexes in use`
- `SearchHeadLevel - indexes per savedsearch`
- `SearchHeadLevel - Indexes for savedsearch without subsearches`
- `SearchHeadLevel - indexes per dashboard`

Updated reports/alerts:
- `AllSplunkEnterpriseLevel - Splunk Scheduler excessive delays in executing search`
- `AllSplunkEnterpriseLevel - sendmodalert errors`
- `SearchHeadLevel - Alerts that have not fired an action in X days
- `SearchHeadLevel - Scheduled Search Efficiency`

To extract savedsearch_name (as I found you can have savedsearches with double quotes in the title).

Author: gjanders

README.md

@@ -359,6 +359,31 @@ These are appear to be from premium apps but it does imply that there is a mecha
 Feel free to open an issue on github or use the contact author on the SplunkBase link and I will try to get back to you when possible, thanks!
 
 ## Release Notes
+### 4.0.6
+New macros:
+- `indexes_extraction(1)` - to extract indexes from search logs
+
+Updated reports/alerts:
+- `AllSplunkEnterpriseLevel - Splunkd Crash Logs Have Appeared in Production` - updated based on email feedback to use sourcetype (as source matching needed wildcards)
+- `IndexerLevel - Slow peer from remote searches` - corrected comment in search only
+- `SearchHeadLevel - Search Queries summary exact match`
+- `SearchHeadLevel - Search Queries summary non-exact match`
+- `SearchHeadLevel - SmartStore cache misses - dashboards`
+- `SearchHeadLevel - SmartStore cache misses - savedsearches`
+- `SearchHeadLevel - SmartStore cache misses - combined`
+- `SearchHeadLevel - Datamodel REST endpoint indexes in use`
+- `SearchHeadLevel - indexes per savedsearch`
+- `SearchHeadLevel - Indexes for savedsearch without subsearches`
+- `SearchHeadLevel - indexes per dashboard`
+
+Updated reports/alerts:
+- `AllSplunkEnterpriseLevel - Splunk Scheduler excessive delays in executing search`
+- `AllSplunkEnterpriseLevel - sendmodalert errors`
+- `SearchHeadLevel - Alerts that have not fired an action in X days
+- `SearchHeadLevel - Scheduled Search Efficiency`
+
+To extract savedsearch_name (as I found you can have savedsearches with double quotes in the title).
+
 ### 4.0.5
 New alerts:
 - `AllSplunkEnterpriseLevel - Splunk servers with resource starvation v2`

default/app.conf

@@ -14,7 +14,7 @@ supported_themes = light,dark
 [launcher]
 author = Gareth Anderson
 description = Alerts and dashboards as described in the Splunk 2017 conf presentation How did you get so big?
-version = 4.0.5
+version = 4.0.6
 
 [package]
 id = SplunkAdmins

default/macros.conf

@@ -925,3 +925,19 @@ iseval = 0
 [splunkadmins_events_per_second]
 definition = desc.savedsearch_name IN ("Example")
 iseval = 0
+
+[indexes_extraction(1)]
+args = search
+definition = rex field=search "(?s)(NOT\s+index(\s*=\s*|::)[^ ]+)|(NOT\s+\([^\)]+\))|(index(\s*=\s*|::)\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
+| rex field=search "(?s)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexinall>[%\)]+))" max_match=50 \
+| rex field=indexinall "(?s)\s*(\"(?P<indexin>[^\", ]+))|(?P<indexin2>[^,\s\"]+)" max_match=50 \
+| makemv tokenizer="([^, ]+)" indexin \
+| eval indexes=mvappend(indexregex,indexin) \
+| eval indexes=if(isnotnull(esstylewildcard),mvfilter(NOT match(indexes,"^_?\*$")),indexes) \
+| eval wildcard=mvfilter(match(indexes,"\*")) \
+| where isnull(wildcard) \
+| eval indexes=mvmap(indexes, replace(lower(indexes), "\"", "")) \
+| eval indexes=mvmap(indexes, trim(replace(indexes, "'", ""))) \
+| eval indexes=mvdedup(indexes) \
+| makemv indexes tokenizer=(\S+)
+iseval = 0

default/savedsearches.conf

@@ -1604,6 +1604,7 @@ index=_internal `splunkenterprisehosts` sourcetype=scheduler app=* scheduled_tim
 | eval time=strftime(_time,"%+") \
 | eval delay_in_start = (dispatch_time - scheduled_time) \
 | where delay_in_start>100\
+| rex "savedsearch_id=\"[^;]+;[^;]+;(?P<savedsearch_name>.*?)\"," \
 | eval scheduled_time=strftime(scheduled_time,"%+") \
 | eval dispatch_time=strftime(dispatch_time,"%+") \
 | rename time AS endTime \
@@ -1970,6 +1971,7 @@ request.ui_dispatch_view = search
 search = ```Attempt to find alerts that are scheduled but not firing any actions, the alerts may need further review or may no longer be required. The app regex is in here because of some creative alert naming, X:app=Y is a real alert name in my environment!```\
 index=_internal source="*scheduler.log" sourcetype=scheduler `searchheadhosts` alert_actions!="" \
 | rex ", app=\"(?P<app>[^\"]+)\","\
+| rex "savedsearch_id=\"[^;]+;[^;]+;(?P<savedsearch_name>.*?)\"," \
 | stats count by savedsearch_name, app \
 | append \
     [| rest `splunkadmins_restmacro` /servicesNS/-/-/saved/searches \
@@ -2245,7 +2247,7 @@ OR "sendmodalert - Invoking modular alert action"\
 `splunkadmins_sendmodalert_errors`\
 | rex field=results_file "[/\\\]dispatch[/\\\](?P<sid>[^/]+)"\
 | eval sid=if(isnull(sid),"NOMATCH",sid)\
-| join sid type=outer [search index=_internal source="*scheduler.log" sourcetype=scheduler `splunkenterprisehosts` | table sid, savedsearch_name, app, user]\
+| join sid type=outer [search index=_internal source="*scheduler.log" sourcetype=scheduler `splunkenterprisehosts` | rex "savedsearch_id=\"[^;]+;[^;]+;(?P<savedsearch_name>.*?)\"," | table sid, savedsearch_name, app, user]\
 | cluster showcount=true\
 | table host, savedsearch_name, app, user, _raw, _time, cluster_count\
 | eval mostRecent = strftime(mostRecent, "%+")\
@@ -4067,6 +4069,7 @@ request.ui_dispatch_view = search
 search = ```This likely came from a Splunk conf presentation but I cannot remember which one so cannot attribute the original author!\
 Determine the length of time a scheduled search takes to run compared to how often it is configured to run, excluding acceleration jobs```\
 index=_internal `searchheadhosts` sourcetype=scheduler source=*scheduler.log (user=*) savedsearch_name!="_ACCELERATE_DM*"\
+| rex "savedsearch_id=\"[^;]+;[^;]+;(?P<savedsearch_name>.*?)\"," \
 | stats avg(run_time) as average_runtime_in_sec count(savedsearch_name) as num_times_per_week sum(run_time) as total_runtime_sec by savedsearch_name user app host\
 | eval ran_every_x_mins=round(60/(num_times_per_week/168))\
 | eval average_runtime_duration=tostring(round(average_runtime_in_sec/60,2), "duration")\
@@ -4712,16 +4715,10 @@ search = | multisearch \
  ```We now deal with cases where search earliest/latest times were not specified, assume all time is about 1 year in the past and latest time was the search run time``` \
 | eval search_lt=if(search_lt=="N/A",timestamp,search_lt), search_et=if(search_et=="N/A",now()-(365*24*60*60),search_et) \
   ```Extract out index= or index IN (a,b,c) but avoid NOT index in (...) and NOT index=... and also NOT (...anything) statements``` \
-| rex field=search "(?s)(NOT\s+index(\s*=\s*|::)[^ ]+)|(NOT\s+\([^\)]+\))|(index(\s*=\s*|::)\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
-| rex field=search "(?s)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexin>([^\)\"]+)|\"[^\)\"]+\"))" max_match=50 \
-| makemv tokenizer="([^, ]+)" indexin \
-| eval indexes=mvappend(indexregex,indexin) \
+| `indexes_extraction(search)` \
 | eval indexes=if(isnotnull(esstylewildcard),mvfilter(NOT match(indexes,"^_?\*$")),indexes) \
 | eval wildcard=mvfilter(match(indexes,"\*")) \
 | where isnull(wildcard) \
-| eval indexes=mvmap(indexes, replace(lower(indexes), "\"", "")) \
-| eval indexes=mvmap(indexes, trim(replace(indexes, "'", ""))) \
-| eval indexes=mvdedup(indexes) \
 | eval multi=if(mvcount(indexes)>1,"true","false") \
 | stats values(timestamp) AS _time, values(total_run_time) AS total_run_time, values(event_count) AS event_count, values(scan_count) AS scan_count, values(search_et) AS search_et, values(search_lt) AS search_lt, values(savedsearch_name) AS savedsearch_name, values(multi) AS multi, max(duration_index) AS duration_index, max(duration_rawdata) AS duration_rawdata, max(cache_index_hits) AS cache_index_hits, max(cache_index_miss) AS cache_index_miss, max(cache_index_hit_duration) AS cache_index_hit_duration, max(cache_index_miss_duration) AS cache_index_miss_duration, max(cache_rawdata_hits) AS cache_rawdata_hits, max(cache_rawdata_miss) AS cache_rawdata_miss, max(cache_rawdata_hit_duration) AS cache_rawdata_hit_duration, max(cache_rawdata_miss_duration) AS cache_rawdata_miss_duration, values(provenance) AS provenance by user, type, indexes, search_head_cluster, search_id, app_name \
 | eval period=search_lt-search_et \
@@ -4815,10 +4812,7 @@ search = | multisearch \
 | rex field=search "(?P<esstylewildcard>\(\s*index=\*\s+OR\s+index=_\*\s*\))" \
 | rex mode=sed field=search "s/search index=\s*\S+\s+index\s*=/search index=/g" \
   ```Extract out index= or index IN (a,b,c) but avoid NOT index in (...) and NOT index=... and also NOT (...anything) statements``` \
-| rex field=search "(?s)(NOT\s+index(\s*=\s*|::)[^ ]+)|(NOT\s+\([^\)]+\))|(index(\s*=\s*|::)\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
-| rex field=search "(?s)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexin>([^\)\"]+)|\"[^\)\"]+\"))" max_match=50 \
-| makemv tokenizer="([^, ]+)" indexin \
-| eval indexes=mvappend(indexregex,indexin) \
+| `indexes_extraction(search)` \
 | eval indexes=if(isnotnull(esstylewildcard),mvfilter(NOT match(indexes,"^_?\*$")),indexes) \
 | eval wildcard=mvfilter(match(indexes,"\*")) \
 | where isnotnull(wildcard) OR isnull(indexes) \
@@ -6204,7 +6198,7 @@ quantity = 0
 relation = greater than
 request.ui_dispatch_app = SplunkAdmins
 request.ui_dispatch_view = search
-search = ```This warning when occurring repetitively tends to indicate some kind of issue that will require the file to be manually removed. For example a zero sized metadata file that cannot be reaped by the dispatch reaper``` \
+search = ```This alert attempts to find peers that are relatively slow compared to other peers. I've used a hardcoded time rather than a variable as it appeared to work well. ``` \
 index=_internal `indexerhosts` source=*remote_searches.log terminated: OR closed: \
 | regex search!="^(pretypeahead|copybuckets)" \
 | rex "(?s) elapsedTime=(?P<elapsedTime>[0-9\.]+),( cpuTime=\S+,)? search='(?P<search>.*?)(', savedsearch_name|\", drop_count=\d+)" \
@@ -7021,13 +7015,7 @@ invocations_command_search_index_bucketcache_miss>0 OR invocations_command_searc
 | search total_cache_miss>0 \
 | search provenance=*Dashboard* \
 | eval total_hours_searched=round(total_hours_searched,1) \
-| rex field=search "(?s)(NOT\s+index(\s*=\s*|::)[^ ]+)|(NOT\s+\([^\)]+\))|(index(\s*=\s*|::)\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
-| rex field=search "(?s)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexin>([^\)\"]+)|\"[^\)\"]+\"))" max_match=50 \
-| makemv tokenizer="([^, ]+)" indexin \
-| eval indexes=mvappend(indexregex,indexin) \
-| eval indexes=mvmap(indexes, replace(lower(indexes), "\"", "")) \
-| eval indexes=mvmap(indexes, trim(replace(indexes, "'", ""))) \
-| eval indexes=mvdedup(indexes) \
+| `indexes_extraction(search)` \
 | eval has_pipe=if(match(search,"\|"),"true",null()) \
 | rex field=search "(?P<search>[^\|]+\|)" \
 | eval search = if(isnotnull(has_pipe),search . " ... (trimmed)",search)\
@@ -7076,13 +7064,7 @@ invocations_command_search_index_bucketcache_miss>0 OR invocations_command_searc
 | `base64decode(base64appname)` \
 | eval app3="N/A", app=coalesce(app,app2,base64appname,app3) \
 | stats latest(mostRecent) AS mostRecent, count as number_of_runs values(host) as host values(total_hours_searched) AS total_hours_searched values(total_days_searched) AS total_days_searched max(run_time) AS max_run_time avg(run_time) AS avg_run_time sum(run_time) AS sum_run_time sum(total_cache_miss) as total_cache_miss max(result_count) AS result_count max(event_count) AS event_count max(searched_buckets) AS searched_buckets values(info) AS info values(numofsearchesinquery) AS numofsearchesinquery, values(app) AS app by users search \
-| rex field=search "(?s)(NOT\s+index(\s*=\s*|::[^ ]+)|(NOT\s+\([^\)]+\))|(index(\s*=\s*|::)\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
-| rex field=search "(?s)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexin>([^\)\"]+)|\"[^\)\"]+\"))" max_match=50 \
-| makemv tokenizer="([^, ]+)" indexin \
-| eval indexes=mvappend(indexregex,indexin) \
-| eval indexes=mvmap(indexes, replace(lower(indexes), "\"", "")) \
-| eval indexes=mvmap(indexes, trim(replace(indexes, "'", ""))) \
-| eval indexes=mvdedup(indexes) \
+| `indexes_extraction(search)` \
 | rex max_match=100 field=search "tag=(?<tags>[^\s+\||\)]+)" \
 | rex max_match=100 field=search "eventtype=(?<eventtypes>[^\s+\||\)]+)" \
 | rex max_match=100 field=search "(?<macros>\`[^\s]+\`)" \
@@ -7131,13 +7113,7 @@ invocations_command_search_index_bucketcache_miss>0 OR invocations_command_searc
 | search total_cache_miss>0 \
 | eval total_hours_searched=round(total_hours_searched,1) \
 | stats latest(mostRecent) AS mostRecent, count as number_of_runs, values(host) as host values(total_hours_searched) AS total_hours_searched values(total_days_searched) AS total_days_searched max(run_time) AS max_run_time avg(run_time) AS avg_run_time sum(run_time) AS sum_run_time sum(total_cache_miss) as total_cache_miss max(result_count) AS result_count max(event_count) AS event_count max(searched_buckets) AS searched_buckets values(info) AS info values(numofsearchesinquery) AS numofsearchesinquery, values(provenance) AS provenance, values(app) AS app by users search \
-| rex field=search "(?s)(NOT\s+index(\s*=\s*|::)[^ ]+)|(NOT\s+\([^\)]+\))|(index(\s*=\s*|::)\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
-| rex field=search "(?s)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexin>([^\)\"]+)|\"[^\)\"]+\"))" max_match=50 \
-| makemv tokenizer="([^, ]+)" indexin \
-| eval indexes=mvappend(indexregex,indexin) \
-| eval indexes=mvmap(indexes, replace(lower(indexes), "\"", "")) \
-| eval indexes=mvmap(indexes, trim(replace(indexes, "'", ""))) \
-| eval indexes=mvdedup(indexes) \
+| `indexes_extraction(search)`
 | eval has_pipe=if(match(search,"\|"),"true",null())\
 | rex max_match=100 field=search "tag=(?<tags>[^\s+\||\)]+)" \
 | rex max_match=100 field=search "eventtype=(?<eventtypes>[^\s+\||\)]+)" \
@@ -7363,7 +7339,7 @@ quantity = 0
 relation = greater than
 request.ui_dispatch_app = SplunkAdmins
 request.ui_dispatch_view = search
-search = index=_internal source=*crash.log \
+search = index=_internal sourcetype=splunkd_crash_log \
 | stats count by source, host, sourcetype \
 | eval indexer_cluster=`indexer_cluster_name(host)` \
 | eval search_head=host \
@@ -8228,14 +8204,7 @@ search = | rest `splunkadmins_restmacro` timeout=900 /servicesNS/-/-/data/models
 | `splunkadmins_macro_sub('eai:data')` \
 | regex eai:data="index\s*(=|[iI][nN])" \
 | rex field=eai:data "(?P<esstylewildcard>\(\s*index=\*\s+OR\s+index=_\*\s*\))" \
-| rex field=eai:data "(?sm)(NOT\s+index\s*(=|::)\s*[^ ]+)|(NOT\s+\([^\)]+\))|(index\s*(=|::)\s*(\\\)?\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
-| rex field=eai:data "(?sm)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexin>([^\)\"]+)|\"[^\)\"]+\"))" max_match=50 \
-| makemv tokenizer="([^, ]+)" indexin \
-| eval indexes=mvappend(indexregex,indexin) \
-| eval indexes=if(isnotnull(esstylewildcard),mvfilter(NOT match(indexes,"^_?\*$")),indexes) \
-| eval indexes=mvmap(indexes, replace(lower(indexes), "\"", "")) \
-| eval indexes=mvmap(indexes, trim(replace(indexes, "'", ""))) \
-| eval indexes=mvdedup(indexes) \
+| `indexes_extraction(eai:data)` \
 | table title, indexes, eai:data, eai:acl.app
 
 [SearchHeadLevel - Job performance data per indexer]
@@ -8475,13 +8444,7 @@ search = | rest /servicesNS/-/-/saved/searches f=next_scheduled_time f=search f=
 | nomv prepipe_subsearch \
 | fillnull prepipe_subsearch value=" " \
 | eval prepipe = prepipe . " " . prepipe_subsearch \
-| rex field=prepipe "(?s)(NOT\s+index(\s*=\s*|::)[^ ]+)|(NOT\s+\([^\)]+\))|(index(\s*=\s*|::)\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
-| rex field=prepipe "(?s)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexin>([^\)\"]+)|\"[^\)\"]+\"))" max_match=50 \
-| makemv tokenizer="([^, ]+)" indexin \
-| eval indexes=mvappend(indexregex,indexin) \
-| eval indexes=mvmap(indexes, replace(lower(indexes), "\"", "")) \
-| eval indexes=mvmap(indexes, trim(replace(indexes, "'", ""))) \
-| eval indexes=mvdedup(indexes) \
+| `indexes_extraction(prepipe)` \
 | eval count=mvcount(indexes) \
 | rename eai:acl.app AS app, eai:acl.owner AS owner, eai:acl.sharing AS sharing \
 | table title, app, indexes, count, owner, sharing, updated
@@ -8574,16 +8537,10 @@ search = index=_audit savedsearch_name="$savedsearch_name$" host IN ($host$) \
 | regex search="^\s*(\|?)\s*(search|tstats|mstats|mcatalog|multisearch|union|set|summarize|datamodel|from\s*:?\s*datamodel|datamodelsimple)\s+" \
 | regex search!="(\||^)\s*(append|union|multisearch|set|appendcols|appendpipe|join|map)" \
 | rex field=search "(?s)^(?P<prepipe>\s*\|?([^\|]+))" \
-| rex field=prepipe "(?s)(NOT\s+index(\s*=\s*|::)[^ ]+)|(NOT\s+\([^\)]+\))|(index(\s*=\s*|::)\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
-| rex field=prepipe "(?s)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexin>([^\)\"]+)|\"[^\)\"]+\"))" max_match=50 \
-| makemv tokenizer="([^, ]+)" indexin \
-| eval indexes=mvappend(indexregex,indexin) \
+| `indexes_extraction(prepipe)` \
 | eval indexes=if(isnotnull(esstylewildcard),mvfilter(NOT match(indexes,"^_?\*$")),indexes) \
 | eval wildcard=mvfilter(match(indexes,"\*")) \
 | where isnull(wildcard) \
-| eval indexes=mvmap(indexes, replace(lower(indexes), "\"", "")) \
-| eval indexes=mvmap(indexes, trim(replace(indexes, "'", ""))) \
-| eval indexes=mvdedup(indexes) \
 | eval count=mvcount(indexes) \
 ```| where count==1 \
 | search indexes!=_* ```\
@@ -8881,13 +8838,7 @@ search = | rest `splunkadmins_restmacro` /servicesNS/-/-/data/ui/views f=eai:dat
 | nomv prepipe_subsearch \
 | fillnull prepipe_subsearch value=" " \
 | eval prepipe = prepipe . " " . prepipe_subsearch \
-| rex field=prepipe "(?s)(NOT\s+index(\s*=\s*|::)[^ ]+)|(NOT\s+\([^\)]+\))|(index(\s*=\s*|::)\"?(?P<indexregex>[\*A-Za-z0-9-_]+))" max_match=50 \
-| rex field=prepipe "(?s)(NOT\s+index\s+[iI][nN]\s*\([^\)]+)|(index\s+[iI][nN]\s*\((?P<indexin>([^\)\"]+)|\"[^\)\"]+\"))" max_match=50 \
-| makemv tokenizer="([^, ]+)" indexin \
-| eval indexes=mvappend(indexregex,indexin) \
-| eval indexes=mvmap(indexes, replace(lower(indexes), "\"", "")) \
-| eval indexes=mvmap(indexes, trim(replace(indexes, "'", ""))) \
-| eval indexes=mvdedup(indexes) \
+| `indexes_extraction(prepipe)` \
 | stats values(indexes) AS indexes by title
 
 [AllSplunkEnterpriseLevel - Splunk servers with resource starvation v2]