Updated alerts:

- `MonitoringConsole - one or more servers require configuration automated` - added missing \, issue #25 (thanks to barrettnet)
- `SearchHeadLevel - Detect MongoDB errors` - included warning level entries

Updated dashboards:
- `indexer_max_data_queue_sizes_by_name` - improved replication panel
- `indexer_max_data_queue_sizes_by_name_v8` - improved replication panel

Updated reports:
- `SearchHeadLevel - indexes per savedsearch` - updated regex for union/set/multisearch
- `SearchHeadLevel - Search Queries summary exact match` - updated regex for union/set/multisearch
- `SearchHeadLevel - Search Queries summary non-exact match` - updated regex for union/set/multisearch
- `SearchHeadLevel - Search Queries summary loadjob and savedsearch usage in audit logs` - updated rexgex, rewrote search to find map, join, appendcols and other commands

Author: gjanders

README.md

@@ -359,6 +359,21 @@ These are appear to be from premium apps but it does imply that there is a mecha
 Feel free to open an issue on github or use the contact author on the SplunkBase link and I will try to get back to you when possible, thanks!
 
 ## Release Notes
+### 4.0.2
+Updated alerts:
+- `MonitoringConsole - one or more servers require configuration automated` - added missing \, issue #25 (thanks to barrettnet)
+- `SearchHeadLevel - Detect MongoDB errors` - included warning level entries
+
+Updated dashboards:
+- `indexer_max_data_queue_sizes_by_name` - improved replication panel
+- `indexer_max_data_queue_sizes_by_name_v8` - improved replication panel
+
+Updated reports:
+- `SearchHeadLevel - indexes per savedsearch` - updated regex for union/set/multisearch
+- `SearchHeadLevel - Search Queries summary exact match` - updated regex for union/set/multisearch
+- `SearchHeadLevel - Search Queries summary non-exact match` - updated regex for union/set/multisearch
+- `SearchHeadLevel - Search Queries summary loadjob and savedsearch usage in audit logs` - updated rexgex, rewrote search to find map, join, appendcols and other commands
+
 ### 4.0.1
 New dashboard:
 -`heavy_forwarder_analysis` - as found in the conf24 presentation PLA1509B

default/app.conf

@@ -14,7 +14,7 @@ supported_themes = light,dark
 [launcher]
 author = Gareth Anderson
 description = Alerts and dashboards as described in the Splunk 2017 conf presentation How did you get so big?
-version = 4.0.1
+version = 4.0.2
 
 [package]
 id = SplunkAdmins

default/data/ui/views/indexer_max_data_queue_sizes_by_name.xml

@@ -132,8 +132,8 @@
 | join guid
     [| rest /services/search/distributed/peers
     | table guid peerName]
-| transaction bid guid endswith="has room now" keeporphans=true
-| timechart span=1m count, max(duration) AS duration by peerName</query>	  
+| transaction bid guid endswith="has room now" keeporphans=true keepevicted=true
+| timechart span=$span$ count, max(duration) AS duration by peerName</query>	  
           <earliest>-60m@m</earliest>
           <latest>now</latest>
           <sampleRatio>1</sampleRatio>

default/data/ui/views/indexer_max_data_queue_sizes_by_name_v8.xml

@@ -132,8 +132,8 @@
 | join guid
     [| rest /services/search/distributed/peers
     | table guid peerName]
-| transaction bid guid endswith="has room now" keeporphans=true
-| timechart span=1m count, max(duration) AS duration by peerName</query>
+| transaction bid guid endswith="has room now" keeporphans=true keepevicted=true
+| timechart span=$span$ count, max(duration) AS duration by peerName</query>
           <earliest>-60m@m</earliest>
           <latest>now</latest>
           <sampleRatio>1</sampleRatio>

default/savedsearches.conf

@@ -2814,8 +2814,8 @@ request.ui_dispatch_app = SplunkAdmins
 request.ui_dispatch_view = search
 search = ```The main goal of this alert errors which might not appear in splunkd.log but are critical to keeping the kvstore running on the search heads. Please check the mongod.log file for further information, the additional count field is simply determining that mongo is still logging...\
 Attempt to find errors in the mongod log and make sure the errors do not relate to shutdown events in the search head cluster. Since this does will ignore any events when either cluster shutsdown it might not be sensitive enough for some use cases...```\
-index=_internal `searchheadhosts` `splunkadmins_mongo_source` (" E " OR " F " OR " W ") ```https://jira.mongodb.org/browse/SERVER-42078 advises this is harmless``` NOT "update of non-mod failed" `splunkadmins_mongodb_errors`\
-| regex _raw="^\s+?\S+\s+[EF]" \
+index=_internal `searchheadhosts` `splunkadmins_mongo_source` (" E " OR " F " OR " W ") ```https://jira.mongodb.org/browse/SERVER-42078 advises this is harmless``` NOT "update of non-mod failed" NOT "is deprecated" NOT "No TransportLayer configured during NetworkInterface startup" NOT "interrupted at shutdown" `splunkadmins_mongodb_errors`\
+| regex _raw="^\s+?\S+\s+[EFW]" \
 | search ```Exclude time periods where shutdowns were occurring``` NOT [`splunkadmins_shutdown_time(searchheadhosts,60,60)`]\
 | eventstats max(_time) AS mostRecent, min(_time) AS firstSeen by host\
 | bin _time span=10m \
@@ -4693,7 +4693,7 @@ search = | multisearch \
     | rex field=search mode=sed "s/```.*?```/ /g" \
     | rex field=search max_match=50 "(?s)\|?\s*(append|appendcols|appendpipe|map|union)\s+\[(?P<subsearch>.*?)\]\s*(\||$)" \
     | rex field=search max_match=50 "(?s)\|?\s*(join)\s+.*?\[(?P<subsearch>.*?)\]\s*(\||$)" \
-    | rex field=search max_match=50 "(?s)\|?\s*(union|set|multisearch)\s+(?P<part1>\[.*?\](\s*\[.*?\])+\s*(`[^`]+`\s*)*(\||$|',\s+))" \
+    | rex field=search max_match=50 "(?s)\|?\s*(union|set|multisearch)[^\[]+(?P<part1>\[.*?\](\s*\[.*?\])+\s*(`[^`]+`\s*)*(\||$|',\s+))" \
     | rex field=part1 max_match=50 "(?s).*?\[(?P<subsearch>.*?)\]\s*(\||$|)" \
     | rex field=search max_match=50 "(?s)\|?\s*(map)\s+(maxsearches\s*=\s*\d+)?\s*search\s*=\s*\"(?P<subsearch>.*?)\"\s*(\||$)" \
     | rex field=search "^(?P<prepipe>\s*\|?([^\|]+))" \
@@ -4804,7 +4804,7 @@ search = | multisearch \
     | rex field=search mode=sed "s/```.*?```/ /g" \
     | rex field=search max_match=50 "(?s)\|?\s*(append|appendcols|appendpipe|map|union)\s+\[(?P<subsearch>.*?)\]\s*(\||$)" \
     | rex field=search max_match=50 "(?s)\|?\s*(join)\s+.*?\[(?P<subsearch>.*?)\]\s*(\||$)" \
-    | rex field=search max_match=50 "(?s)\|?\s*(union|set|multisearch)\s+(?P<part1>\[.*?\](\s*\[.*?\])+\s*(`[^`]+`\s*)*(\||$|',\s+))" \
+    | rex field=search max_match=50 "(?s)\|?\s*(union|set|multisearch)[^\[]+(?P<part1>\[.*?\](\s*\[.*?\])+\s*(`[^`]+`\s*)*(\||$|',\s+))" \
     | rex field=part1 max_match=50 "(?s).*?\[(?P<subsearch>.*?)\]\s*(\||$|)" \
     | rex field=search max_match=50 "(?s)\|?\s*(map)\s+(maxsearches\s*=\s*\d+)?\s*search\s*=\s*\"(?P<subsearch>.*?)\"\s*(\||$)" \
     | rex field=search "^(?P<prepipe>\s*\|?([^\|]+))" \
@@ -8408,7 +8408,7 @@ search = | rest splunk_server=local "/services/search/distributed/peers?output_m
         ] \
 | eval data="trigger_actions=1" \
     ``` 2023-12-12 , I've submitted docs feedback that this endpoint is undocumented but used by the MC, it appears to trigger the actions related to the savedsearch to populate a lookup ``` \
-``` remove the comments once you have Webtools Add-on, https://splunkbase.splunk.com/app/4146 installed to use the curl command
+``` remove the comments once you have Webtools Add-on, https://splunkbase.splunk.com/app/4146 installed to use the curl command \
 | curl splunkauth=true method=post uri="https://localhost:8089/servicesNS/nobody/splunk_monitoring_console/saved/searches/DMC+Asset+-+Build+Full/dispatch" datafield=data \
 ```
 disabled = 1
@@ -8465,7 +8465,7 @@ request.ui_dispatch_view = search
 search = | rest /servicesNS/-/-/saved/searches f=next_scheduled_time f=search f=qualifiedSearch count=0 search="disabled=0" search="is_scheduled=1" f=eai:acl* `splunkadmins_restmacro` timeout=900 \
 | rex field=qualifiedSearch mode=sed "s/```.*?```/ /g" \
 | regex qualifiedSearch="^\s*(\|?)\s*(search|tstats|mstats|mcatalog|multisearch|union|set|summarize|datamodel|from\s*:?\s*datamodel|datamodelsimple)\s+" \
-| rex field=search max_match=50 "(?s)\|?\s*(union|set|multisearch)\s+(?P<part1>\[.*?\](\s*\[.*?\])+\s*(`[^`]+`\s*)*(\||$|',\s+))" \
+| rex field=search max_match=50 "(?s)\|?\s*(union|set|multisearch)[^\[]+(?P<part1>\[.*?\](\s*\[.*?\])+\s*(`[^`]+`\s*)*(\||$|',\s+))" \
 | rex field=part1 max_match=50 "(?s).*?\[(?P<subsearch>.*?)\]\s*(\||$|)" \
 | rex field=search max_match=50 "(?s)\|?\s*(map)\s+(maxsearches\s*=\s*\d+)?\s*search\s*=\s*\"(?P<subsearch>.*?)\"\s*(\||$)" \
 | rex field=search "(?s)^(?P<prepipe>\s*\|?([^\|]+))" \
@@ -8652,7 +8652,7 @@ search = | rest "/servicesNS/-/-/storage/collections/config" count=0 timeout=900
 action.email.useNSSubject = 1
 alert.track = 0
 cron_schedule = 38 5 * * *
-description = Report only? Yes. This search attempts to search the audit logs to find any use of | loadjob of | savedsearch within the audit logs. macro substitution is not used but could be included (although I'm unsure how often someone calls a | savedsearch or | loadjob via a macro)
+description = Report only? Yes. This search attempts to search the audit logs to find any use of | loadjob of | savedsearch within the audit logs, additionally map, join, append, appendpipe, appendcols, set, union can make calls to savedsearches that are not easily findable in the audit.log file. macro substitution is not used but could be included (although I'm unsure how often someone calls a | savedsearch, | loadjob or | map via a macro)
 dispatch.earliest_time = -24h-5m@
 dispatch.latest_time = -5m@m
 display.events.fields = ["index","sourcetype","host"]
@@ -8667,15 +8667,29 @@ search = index=_audit "info=completed" search_id!="'SummaryDirector_*" search_id
 | rex field=search mode=sed "s/```.*?```/ /g" \
 | eval search=if(substr(search,len(search),len(search)-1)=="'",substr(search,0,len(search)-1),search) \
 | eval search_id=replace(search_id,"'","") \
-| regex search="\|\s*loadjob\s*savedsearch=|\|\s*savedsearch" \
-| rex field=search "\|\s*savedsearch\s+(\"(?P<identified_savedsearch_name>[^\"']+)\"|(?P<identified_savedsearch_name2>[^ ']+))" \
+| regex search="\|\s*loadjob\s*savedsearch=|\|\s*savedsearch|\|\s*map\s*(maxsearches\s*=\s*\S+\s*)?|\|\s*join\s+|\|\s*append(pipe|cols)?\s+|\|\s*set\s+|\|\s*union\s+" \
+| rex field=search "\|\s*savedsearch\s+(\"(?P<identified_savedsearch_name>[^\"']+)\"|(?P<identified_savedsearch_name2>[^ '\)]+))" \
 | rex field=search "\|\s*loadjob savedsearch=\"[^:]+:[^:]+:(?P<identified_savedsearch_name3>[^\"]+)" \
-| eval identified_savedsearch_name=coalesce(identified_savedsearch_name,identified_savedsearch_name2,identified_savedsearch_name3) \
-| where isnotnull(identified_savedsearch_name)\
-| search NOT identified_savedsearch_name IN ("instrumentation.topology*", "instrumentation.usage*", "instrumentation.upgrade*", "instrumentation.deployment*", "instrumentation.performance*", "instrumentation.app*", "instrumentation.licensing*", "instrumentation.authentication*")\
-| eval method=if(isnull(identified_savedsearch_name3),"savedsearch","loadjob")\
+| rex field=search "\|\s*map\s*(maxsearches\s*=\s*\S+\s*)?(?P<identified_savedsearch_name4>(\"[^\"]+)|\S+)" \
+| rex field=search "\|\s*join\s+[^\|]*?savedsearch:(?P<identified_savedsearch_name5>(\"[^\"]+)|\S+)" \
+| rex field=search "\|\s*(append|appendpipe|appendcols)\s+\[.*?(?P<identified_savedsearch_name6>savedsearch\s+(\"[^\"]+|\S+))" \
+| rex field=search max_match=50 "(?s)\|?\s*(union|set)[^\[]+(?P<part1>\[.*?\](\s*\[.*?\])+\s*(`[^`]+`\s*)*(\||$|',\s+))" \
+| rex field=part1 max_match=50 "(?s).*?\[(?P<subsearch>.*?)\]\s*(\||$|)" \
+| rex field=subsearch "(?s)^\s*\|?(?P<prepipe_subsearch>([^\|]+))" \
+| nomv prepipe_subsearch \
+| rex field=prepipe_subsearch max_match=50 "savedsearch\s+(?P<identified_savedsearch_name7>(\"[^\"]+|\S+))" \
+| rex mode=sed field=identified_savedsearch_name4 "s/([\",']|savedsearch:)//g" \
+| rex mode=sed field=identified_savedsearch_name5 "s/[\",']//g" \
+| rex mode=sed field=identified_savedsearch_name6 "s/(\"|savedsearch )//g" \
+| rex mode=sed field=identified_savedsearch_name7 "s/[\",']//g" \
+| eval identified_savedsearch_name4=if(match(identified_savedsearch_name4,"(search=|\)|\[|\[\|)"),null(),identified_savedsearch_name4) \
+| eval identified_savedsearch_name=coalesce(identified_savedsearch_name,identified_savedsearch_name2,identified_savedsearch_name3,identified_savedsearch_name4,identified_savedsearch_name5,identified_savedsearch_name6,identified_savedsearch_name7) \
+```| table identified_savedsearch_name*, search, subsearch, prepipe_subsearch    ``` \
+| where isnotnull(identified_savedsearch_name4) \
+| search NOT identified_savedsearch_name IN ("instrumentation.topology*", "instrumentation.usage*", "instrumentation.upgrade*", "instrumentation.deployment*", "instrumentation.performance*", "instrumentation.app*", "instrumentation.licensing*", "instrumentation.authentication*") \
+| eval method=if(isnull(identified_savedsearch_name3),"savedsearch","loadjob") \
 | eval search_head=host \
-| eval env=`search_head_cluster`\
+| eval env=`search_head_cluster` \
 | stats values(savedsearch_name) AS calling_savedsearch_name by _time, user, provenance, mode, app, identified_savedsearch_name, env, method
 
 [SearchHeadLevel - configtracker index example2]