New alerts:

- `MonitoringConsole - one or more servers require configuration`
- `MonitoringConsole - one or more servers require configuration automated`
- `SearchHeadLevel - Peer timeouts or authentication issues`

New macros:
- `splunkadmins_macro_sub`

New reports:
- `SearchHeadLevel - Datamodel REST endpoint indexes in use`
- `SearchHeadLevel - Job performance data per indexer`
- `SearchHeadLevel - Jobs endpoint example`
- `SearchHeadLevel - configtracker index example`

Updated alerts:
- `AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only` - more criteria
- `SearchHeadLevel - Search Messages user level` - more criteria
- `SearchHeadLevel - Search Messages admins only` - more criteria

Updated dashboards:
- `splunk_forwarder_output_tuning` - to reference NLB/load balanced version of asynchronous forwarding

Updated macros:
- `whataccessdoihave` - comments and added srchIndexesDisallowed

Updated reports:
- `SearchHeadLevel - IndexesPerRole Remote Report` - comment updates only
- `SearchHeadLevel - Lookup file owners` - comment updates only

Alerts added to future removal list:
- `ClusterMasterLevel - Per index status`

Updated to use `splunkadmins_macro_sub` macro:
- `SearchHeadLevel - Dashboards with all time searches set`
- `SearchHeadLevel - Scheduled searches not specifying an index macro version`
- `SearchHeadLevel - Search Queries By Type Audit Logs macro version`
- `SearchHeadLevel - Search Queries By Type Audit Logs macro version other`
- `SearchHeadLevel - Search Queries summary exact match`
- `SearchHeadLevel - Search Queries summary non-exact match`
- `SearchHeadLevel - User - Dashboards searching all indexes macro version`

Misc:
- Added supported themes settings in app.conf to allow the usage of dark theme (for 9.1 enterprise users and above)

Author: gjanders

README.md

@@ -181,7 +181,7 @@ Are all well suited to an automated email using the sendresults command or a sim
 ## Which alerts and reports have been tested on the newer Splunk versions such as 8.2 or 9.0?
 This application was first created in 2017 and both Splunk and the application have evolved during this time period. This application is a library of potential alerts that could be used in a Splunk environment so it would never be a good idea to turn on all alerts from this application.
 
-The below list of alerts and reports are actively used since version 8.0.x and in 8.2.x and eventually 9.0:
+The below list of alerts and reports are actively used since version 8.0.x and in 8.2.x and eventually 9.0.x:
 - `AllSplunkEnterpriseLevel - error in stdout.log`
 - `AllSplunkEnterpriseLevel - Email Sending Failures`
 - `AllSplunkEnterpriseLevel - Losing Contact With Master Node`
@@ -224,6 +224,8 @@ The below list of alerts and reports are actively used since version 8.0.x and i
 - `MonitoringConsole - Check OS ulimits via REST`
 - `MonitoringConsole - Core dumps have appeared on the filesystem`
 - `MonitoringConsole - Crash logs have appeared on the filesystem`
+- `MonitoringConsole - one or more servers require configuration`
+- `MonitoringConsole - one or more servers require configuration automated`
 - `SearchHeadLevel - audit.log - lookup usage`
 - `SearchHeadLevel - authorize.conf settings will prevent some users from appearing in the UI`
 - `SearchHeadLevel - Captain Switchover Occurring`
@@ -245,6 +247,7 @@ The below list of alerts and reports are actively used since version 8.0.x and i
 - `SearchHeadLevel - Lookup file owners`
 - `SearchHeadLevel - Lookups within dashboards`
 - `SearchHeadLevel - Lookups within savedsearches`
+- `SearchHeadLevel - Peer timeouts or authentication issues`
 - `SearchHeadLevel - platform_stats access summary`
 - `SearchHeadLevel - platform_stats.audit metrics api`
 - `SearchHeadLevel - platform_stats.audit metrics searches`
@@ -323,13 +326,49 @@ Feel free to open an issue on github or use the contact author on the SplunkBase
 
 ## Release Notes
 ### 3.0.12
+New alerts:
+- `MonitoringConsole - one or more servers require configuration`
+- `MonitoringConsole - one or more servers require configuration automated`
+- `SearchHeadLevel - Peer timeouts or authentication issues`
+
 New macros:
-- splunkadmins_macro_sub
+- `splunkadmins_macro_sub`
 
 New reports:
-- SearchHeadLevel - Datamodel REST endpoint indexes in use
-- SearchHeadLevel - Job performance data per indexer
-- SearchHeadLevel - Jobs endpoint example
+- `SearchHeadLevel - Datamodel REST endpoint indexes in use`
+- `SearchHeadLevel - Job performance data per indexer`
+- `SearchHeadLevel - Jobs endpoint example`
+- `SearchHeadLevel - configtracker index example`
+
+Updated alerts:
+- `AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only` - more criteria
+- `SearchHeadLevel - Search Messages user level` - more criteria
+- `SearchHeadLevel - Search Messages admins only` - more criteria
+
+Updated dashboards:
+- `splunk_forwarder_output_tuning` - to reference NLB/load balanced version of asynchronous forwarding
+
+Updated macros:
+- `whataccessdoihave` - comments and added srchIndexesDisallowed
+
+Updated reports:
+- `SearchHeadLevel - IndexesPerRole Remote Report` - comment updates only
+- `SearchHeadLevel - Lookup file owners` - comment updates only
+
+Alerts added to future removal list:
+- `ClusterMasterLevel - Per index status`
+
+Updated to use `splunkadmins_macro_sub` macro:
+- `SearchHeadLevel - Dashboards with all time searches set`
+- `SearchHeadLevel - Scheduled searches not specifying an index macro version`
+- `SearchHeadLevel - Search Queries By Type Audit Logs macro version`
+- `SearchHeadLevel - Search Queries By Type Audit Logs macro version other`
+- `SearchHeadLevel - Search Queries summary exact match`
+- `SearchHeadLevel - Search Queries summary non-exact match`
+- `SearchHeadLevel - User - Dashboards searching all indexes macro version`
+
+Misc:
+- Added supported themes settings in app.conf to allow the usage of dark theme (for 9.1 enterprise users and above)
 
 ### 3.0.11
 Updated alerts:

default/app.conf

@@ -8,11 +8,13 @@ is_configured = 0
 [ui]
 is_visible = 1
 label = SplunkAdmins
+# allow 9.1 and above to use themes
+supported_themes = light,dark
 
 [launcher]
 author = Gareth Anderson
 description = Alerts and dashboards as described in the Splunk 2017 conf presentation How did you get so big?
-version = 3.0.11
+version = 3.0.12
 
 [package]
 id = SplunkAdmins

default/data/ui/nav/default.xml

@@ -313,6 +313,7 @@
 			<saved name="SearchHeadLevel - Lookups within savedsearches" />
 			<saved name="SearchHeadLevel - Job performance data per indexer" />
 			<saved name="SearchHeadLevel - Jobs endpoint example" />
+			<saved name="SearchHeadLevel - configtracker index example" />
                         <saved name="IndexerLevel - RemoteSearches Indexes Stats" />
                         <saved name="IndexerLevel - RemoteSearches Indexes Stats Wilcard" />			
 			<saved name="IndexerLevel - RemoteSearches - lookup usage" />
@@ -321,7 +322,7 @@
 			<saved name="SearchHeadLevel - Data Model Acceleration Completion Status" />
 			<saved name="SearchHeadLevel - DataModel Fields" />
 			<saved name="SearchHeadLevel - Accelerated DataModels Access Info" />
-			<saved name="SearchHeadLevel - Datamodel REST endpoint indexes in use" />
+			<saved name="SearchHeadLevel - Datamodel REST endpoint indexes in use" />			
 			<saved name="IndexerLevel - DataModel Acceleration - Indexes in use" />
                         <a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FSearchHeadLevel%20-%20datamodel%20errors%20in%20splunkd">datamodel errors in splunkd</a>			
 			<view name="data_model_rebuild_monitor" />
@@ -343,7 +344,8 @@
 			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FSearchHeadLevel%20-%20datamodel%20errors%20in%20splunkd">datamodel errors in splunkd</a>
                         <a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FIndexerLevel%20-%20SmartStore%20-%20Bucket%20cache%20errors%20audit%20logs">IndexerLevel - SmartStore - Bucket cache errors audit logs</a>
                         <a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FAllSplunkLevel%20-%20No%20recent%20metrics.log%20data">AllSplunkLevel - No recent metrics.log data</a>
-			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FSearchHeadLevel%20-%20Detect%20bundle%20pushes%20no%20longer%20occurring">SearchHeadLevel - Detect bundle pushes no longer occurring</a>
+			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FSearchHeadLevel%20-%20Detect%20bundle%20pushes%20no%20longer%20occurring">Detect bundle pushes no longer occurring</a>
+			<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FSearchHeadLevel%20-%20Peer%20timeouts%20or%20authentication%20issues">Peer timeouts or authentication issues</a>
 	                <collection label="Generic">
 	                        <a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FAllSplunkEnterpriseLevel%20-%20Splunkd%20Log%20Messages%20Admins%20Only">Splunkd Log Messages Admins Only</a>
 	                        <a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FSearchHeadLevel%20-%20Search%20Messages%20user%20level">Search Messages user level</a>
@@ -548,5 +550,7 @@
 	<collection label="MonitoringConsole">
 		<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FMonitoringConsole%20-%20Core%20dumps%20have%20appeared%20on%20the%20filesystem">Core dumps have appeared on the filesystem</a>
 		<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FMonitoringConsole%20-%20Crash%20logs%20have%20appeared%20on%20the%20filesystem">Crash logs have appeared on the filesystem</a>		
+		<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FMonitoringConsole%20-%20one%20or%20more%20servers%20require%20configuration">one or more servers require configuration</a>
+		<a href="/app/SplunkAdmins/alert?s=%2FservicesNS%2Fnobody%2FSplunkAdmins%2Fsaved%2Fsearches%2FMonitoringConsole%20-%20one%20or%20more%20servers%20require%20configuration%20automated">one or more servers require configuration automated</a>
 	</collection>
 </nav>

default/data/ui/views/splunk_forwarder_output_tuning.xml

@@ -109,6 +109,7 @@
     <p>Purpose of the data output per-second timechart? The current goal is to get close to switching indexers every second for an output group (per-pipeline), note that this will result in more open connections to indexers so only really works if this is deployed to a moderate number of intermediate forwarders (HF's or similar). Note that you want to do this with autoLBVolume, if you lower autoLBFrequency to a very short time period you may result in un-even data balance due to switching frequently when forwarding smaller volumes of data. In my testing so far it would appear that aiming above the average kb/s for the autoLBVolume appears to work well, going too low doesn't work well in my testing so far</p>
     <p>Please read the linked article for information on these settings, note that when using async forwarding the open file descriptor usage is higher than without async forwarding as the connections are held open by forwarders. So this works great on an intermediate forwarding tier, this may not work so well with a very large number of forwarders</p>
     <p>Also note that the maxQueueSize should not be below 10MB (10MB minimium size)</p>
+    <p>If you are using an AWS NLB, you may wish to refer to this newer post <a href="https://www.linkedin.com/posts/harendra-rawat-b10b41_asynchronous-forwarding-with-nlb-activity-7112204069363933185-SYRv"> Asynchronous forwarding with NLB</a></p>
     <p>Finally while this also works on UF's, there are some reasons why you may want to consider HF's if you are running an intermediate tier, answers post <a href="https://community.splunk.com/t5/Getting-Data-In/Wrongly-merged-Events-permanently-blocked-tcpout-queue-with/m-p/508743">Wrongly merged Events/permanently blocked tcpout queue with Intermediate Universal Forwarder</a></p>
           <br/>
     <p>What config  is used to achieve the above?</p>

default/macros.conf

@@ -529,21 +529,21 @@ iseval = 0
 
 [whataccessdoihave]
 definition = rest /services/authentication/users splunk_server=local\
-| search `comment("REST query is limited to the current search head this is running on. If users have the dispatch REST to indexers capability then ise the 'What Access Do I Have' version' for more detail")`\
+| search `comment("REST query is limited to the current search head this is running on so we see the index access from this instances point of view")`\
     [| rest /services/authentication/current-context/context splunk_server=local\
     | head 1 \
     | fields username \
     | rename username AS title] \
 | table title roles | rename title as user | mvexpand roles\
 | join type=left roles \
     [rest /services/authorization/roles splunk_server=local\
-    | table title srchIndexesAllowed srchIndexesDefault imported_srchIndexesAllowed imported_srchIndexesDefault | rename title as roles]\
-| fillnull value="" srchIndexesAllowed, srchIndexesDefault, imported_srchIndexesAllowed, imported_srchIndexesDefault\
-| eval srchIndexesAllowed = srchIndexesAllowed . " " . imported_srchIndexesAllowed, srchIndexesDefault = srchIndexesDefault . " " . imported_srchIndexesDefault\
-| makemv srchIndexesAllowed tokenizer=(\S+) | makemv srchIndexesDefault tokenizer=(\S+)\
+    | table title srchIndexesAllowed srchIndexesDefault srchIndexesDisallowed imported_srchIndexesAllowed imported_srchIndexesDefault imported_srchIndexesDisallowed | rename title as roles]\
+| fillnull value="" srchIndexesAllowed, srchIndexesDefault, srchIndexesDisallowed, imported_srchIndexesAllowed, imported_srchIndexesDefault imported_srchIndexesDisallowed\
+| eval srchIndexesAllowed = srchIndexesAllowed . " " . imported_srchIndexesAllowed, srchIndexesDefault = srchIndexesDefault . " " . imported_srchIndexesDefault, srchIndexesDisallowed = srchIndexesDisallowed . " " . imported_srchIndexesDisallowed \
+| makemv srchIndexesAllowed tokenizer=(\S+) | makemv srchIndexesDefault tokenizer=(\S+) | makemv srchIndexesDisallowed tokenizer=(\S+) \
 | eval indexes= [ | eventcount summarize=false index=* index=_* | stats values(index) AS indexes | eval theindexes="\"" . mvjoin(indexes, " ") . "\"" | return $theindexes ]\
 | makemv indexes\
-| stats values(roles) AS roles, values(indexes) AS indexes, values(srchIndexesAllowed) AS srchIndexesAllowed, values(srchIndexesDefault) AS srchIndexesDefault by user
+| stats values(roles) AS roles, values(indexes) AS indexes, values(srchIndexesAllowed) AS srchIndexesAllowed, values(srchIndexesDefault) AS srchIndexesDefault, values(srchIndexesDisallowed) AS srchIndexesDisallowed by user
 
 [diskusage]
 definition = rest /services/authentication/current-context/context splunk_server=local \
@@ -646,7 +646,6 @@ definition = search `comment("Set all values to null() in case this macro is cal
 | eval search=if(isnotnull(macro_name),replace($fieldname$,mvindex(macro_name,0),definition),$fieldname$)
 iseval = 0
 
-
 #Note this macro requires TA-webtools
 #Alternatively the "Mothership app" on SplunkBase can be used for this purpose...
 [splunkadmins_remote_macros(3)]