3.0.2

Merged pull request from jeffland-consist via github including various changes

New alerts:

• IndexerLevel - replicationdatareceiverthread close to 100% utilisation

New macros:

• splunkadmins_metrics_source
• splunkadmins_hec_metrics_source

New reports:

• SearchHeadLevel - Accelerated DataModels Access Info
• SearchHeadLevel - Dashboards resulting in concurrency issues
• SearchHeadLevel - Dashboards that may benefit from base or post-process searches
• SearchHeadLevel - Searches by search type

Updated macros:

• splunkadmins_splunkd_source
• splunkadmins_splunkuf_source
• splunkadmins_mongo_source
• splunkadmins_license_usage_source

To include a trailing wildcard (so splunkd.log.1 matches or similar)

Updated alerts:

• AllSplunkEnterpriseLevel - Core Dumps Disabled - updated matching criteria
• AllSplunkEnterpriseLevel - Non-existent roles are assigned to users - updated matching criteria
• AllSplunkEnterpriseLevel - Splunk Servers throwing runScript errors - updated matching criteria
• AllSplunkEnterpriseLevel - sendmodalert errors - updated matching criteria
• AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only - updated matching criteria
• AllSplunkEnterpriseLevel - Splunk Servers with resource starvation - updated to use splunkadmins_splunkd_source macro
• AllSplunkLevel - No recent metrics.log data - corrected comment to be after tstats, updated to use splunkadmins_metrics_source macro
• AllSplunkLevel - DeploymentServer Application Installation Error - updated matching criteria
• DeploymentServer - Application Not Found On Deployment Server - updated matching criteria
• ForwarderLevel - Channel churn issues - updated to use splunkadmins_metrics_source macro
• ForwarderLevel - Forwarders connecting to a single endpoint for extended periods - updated to use splunkadmins_metrics_source macro
• ForwarderLevel - Forwarders connecting to a single endpoint for extended periods UF level - updated to use splunkadmins_metrics_source macro
• ForwarderLevel - Splunk HTTP Listener Overwhelmed - updated matching criteria
• ForwarderLevel - Splunk Universal Forwarders Exceeding the File Descriptor Cache - updated matching criteria
• ForwarderLevel - Splunk Universal Forwarders that are time shifting - updated matching criteria
• ForwarderLevel - Stopping all listening ports - updated to use splunkadmins_splunkd_source macro
• IndexerLevel - Buckets changes per day - updated matching criteria, updated to use splunkadmins_splunkd_source macro
• IndexerLevel - Indexer Queues May Have Issues - updated to use splunkadmins_metrics_source macro
• IndexerLevel - Knowledge bundle upload stats - updated to use splunkadmins_metrics_source macro
• IndexerLevel - platform_stats.indexers totalgb_thruput measurement - updated to use splunkadmins_metrics_source macro
• IndexerLevel - platform_stats.indexers stddev measurement - updated to use splunkadmins_metrics_source macro
• IndexerLevel - platform_stats.indexers stddev incoming measurement - updated to use splunkadmins_metrics_source macro
• IndexerLevel - Weekly Broken Events Report - updated matching criteria
• IndexerLevel - Time format has changed multiple log types in one sourcetype - updated matching criteria
• IndexerLevel - Buckets have being frozen due to index sizing - updated matching criteria
• IndexerLevel - Unclean Shutdown - Fsck - updated matching criteria
• IndexerLevel - Index not defined - updated matching criteria
• IndexerLevel - Timestamp parsing issues combined alert - updated to use splunkadmins_splunkd_source macro
• IndexerLevel - S2SFileReceiver Error - updated matching criteria
• MonitoringConsole - Core dumps have appeared on the filesystem - corrected to use indexer_cluster_name macro
• MonitoringConsole - Crash logs have appeared on the filesystem - corrected description
• SearchHeadLevel - LDAP users have been disabled or left the company cleanup required - updated matching criteria
• SearchHeadLevel - Long filenames may be causing issues - updated matching criteria
• SearchHeadLevel - SHCluster Artifact Replication Issues - updated matching criteria
• SearchHeadLevel - Captain Switchover Occurring - updated matching criteria
• SearchHeadLevel - Knowledge bundle replication times metrics.log - updated to use splunkadmins_metrics_source macro
• SearchHeadLevel - Detect bundle pushes no longer occurring - updated to use splunkadmins_metrics_source macro
• SearchHeadLevel - WLM aborted searches - updated matching criteria
• SearchHeadLevel - SHC Captain unable to establish common bundle - updated to use splunkadmins_splunkd_source macro

Updated dashboards:

• ClusterMasterJobs.xml
• heavyforwarders_max_data_queue_sizes_by_name.xml
• heavyforwarders_max_data_queue_sizes_by_name_v8.xml
• hec_performance.xml
• indexer_data_spread.xml
• indexer_max_data_queue_sizes_by_name.xml
• indexer_max_data_queue_sizes_by_name_v8.xml
• rolled_buckets_by_index.xml
• smartstore_stats.xml
• splunk_forwarder_data_balance_tuning.xml
• splunk_forwarder_output_tuning.xml

To use splunkadmins_splunkd_source and/or splunkadmins_metrics_source macros

3.0.1

New macros:

• splunkadmins_shutdown_time_by_period

New alerts:

• MonitoringConsole - Check OS ulimits via REST
• SearchHeadLevel - Detect bundle pushes no longer occurring

New reports:

• DeploymentServer - Count by application - contributed by @trex (radler)
• IndexerLevel - DataModel Acceleration - Indexes in use
• SearchHeadLevel - Knowledge bundle status on indexers
• SearchHeadLevel - Knowledge bundle replication times metrics.log

Updated alerts:

• AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only

Updated dashboards:

• splunk_introspection_io_stats - updated names/description of fields used
• indexer_max_data_queue_sizes_by_name - minor tweak to replication queue queries
• indexer_max_data_queue_sizes_by_name_v8 - minor tweak to replication queue queries
• splunk_forwarder_output_tuning - comment update only

Updated macros:

• splunkadmins_shutdown_time_by_period(4) to work as expected

Added link to Admins Little Helper for Splunk and TrackMe
README.md improvements

3.0.0

Due to the creation of TA-Alerts for SplunkAdmins, the following are removed in this release:

• bin directory
• README directory
• default/searchbnf.conf
• default/inputs.conf
• default/commands.conf

LookupWatcher and the custom commands streamfilter and streamfilterwildcard are now moved into the new TA-Alerts for SplunkAdmins application

New alerts:

• AllSplunkEnterpriseLevel - error in stdout.log
• IndexerLevel - platform_stats.indexers stddev incoming measurement
• MonitoringConsole - Core dumps have appeared on the filesystem
• MonitoringConsole - Crash logs have appeared on the filesystem
• SearchHeadLevel - Splunk Scheduler logs have not appeared in the last

Updated:

• AllSplunkEnterpriseLevel - Replication Failures - simplified criteria to match more issues
• AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only - corrected order of statements so this works as expected, added 1 more exclusion
• IndexerLevel - platform_stats.indexers stddev measurement - narrowed down to sourcetype/source
• IndexerLevel - Search Failures - changed criteria
• IndexerLevel - Indexer Queues May Have Issues - added server count
• IndexerLevel - RemoteSearches Indexes Stats Wilcard - description update as this requires TA-Alerts for SplunkAdmins
• SearchHeadLevel - Dashboards using depends and running searches in the background - description update as this requires TA-Alerts for SplunkAdmins
• SearchHeadLevel - Detect MongoDB errors - excluded 1 warning
• SearchHeadLevel - Search Queries summary exact match - comment update
• SearchHeadLevel - Search Queries summary non-exact match - comment and description update as this requires TA-Alerts for SplunkAdmins
• SearchHeadLevel - Search Messages user level - removed "DAG Execution Exception"
• SearchHeadLevel - Search Messages admins only - excluded "Found no results to append to collection"

2.6.13

Updated python SDK to 1.6.20

Updates to reports/alerts:
IndexerLevel - Future Dated Events that appeared in the last week - comment upate

IndexerLevel - IndexConfig Warnings from Splunk indexers - added wildcard to improve matching

Updated regex to handle index:: case:
IndexerLevel - RemoteSearches Indexes Stats

IndexerLevel - RemoteSearches Indexes Stats Wilcard

SearchHeadLevel - Determine query scan density

SearchHeadLevel - Search Queries By Type Audit Logs

SearchHeadLevel - Search Queries By Type Audit Logs macro version

SearchHeadLevel - Search Queries By Type Audit Logs macro version other

SearchHeadLevel - SmartStore cache misses - dashboards

SearchHeadLevel - SmartStore cache misses - savedsearches

SearchHeadLevel - SmartStore cache misses - combined

Updated regex to handle index:: case: and minor tweak to replace comments with spaces:
SearchHeadLevel - Search Queries summary exact match

SearchHeadLevel - Search Queries summary non-exact match

Updated links in nav menu:
SideView UI (user activity)

2.6.12

2.6.12 fixes a missing \ character in savedsearches.conf introduced in 2.6.11

Changed:
splunkadmins_userlist_indexinfo from kvstore collection into a csv file to prevent unncessary restarts related to updating this app (on standalone instances this triggers a restart due to collections.conf), collections.conf was removed from this appp

New dashboard:
splunk_introspection_io_stats - just an I/O focussed dashboard based on introspection data

New macros:
splunkadmins_shutdown_time_by_shc
cluster_masters

Updated alerts:
AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only - more criteria
IndexerLevel - IndexConfig Warnings from Splunk indexers - updated criteria
SearchHeadLevel - KVStore Or Conf Replication Issues Are Occurring - updated keywords for new instances, added more criteria to reduce false alarms
SearchHeadLevel - Lookup updates within SHC

Updated dashboards:
heavyforwarders_max_data_queue_sizes_by_name_v8
indexer_max_data_queue_sizes_by_name
smartstore_stats
splunk_forwarder_output_tuning - added attribution link

2.6.11

New dashboards:
splunk_introspection_io_stats - just an I/O focussed dashboard based on introspection data

New macro:
splunkadmins_shutdown_time_by_shc

cluster_masters

Updated alerts:
AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only - more criteria

IndexerLevel - IndexConfig Warnings from Splunk indexers - updated criteria, using stats instead of top

SearchHeadLevel - KVStore Or Conf Replication Issues Are Occurring - updated keywords for new instances, added more criteria to reduce false alarms

SearchHeadLevel - Lookup updates within SHC - changed to addCommit instead of acceptPush

Updated dashboards:
heavyforwarders_max_data_queue_sizes_by_name_v8 - corrected missing space in "TcpOut KB per second per forwarder" panel, (feedback from Vincent)

indexer_max_data_queue_sizes_by_name - updated comment on replication queue, replication queue issues now show duration

smartstore_stats - updated comment

splunk_forwarder_output_tuning - added attribution as the link is available via search engines and public, updated comments

Changed:
splunkadmins_userlist_indexinfo into a csv file to prevent unncessary restarts related to updating this app (on standalone instances this triggers a restart due to collections.conf), collections.conf was removed from this app

2.6.10

README.md update

New alert:
SearchHeadLevel - Excessive REST API usage

New dashboard:
splunk_forwarder_data_balance_tuning - new dashboard based on Brett Adam's work

New macro:
diskusage

Updated alert:
AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only - more criteria

ForwarderLevel - Channel churn issues - added another TERM to the search, added stats line to summarise the result, added to where so this fires only if channels added and removed

IndexerLevel - RemoteSearches Indexes Stats - updated comment and rename of fields

IndexerLevel - RemoteSearches Indexes Stats Wilcard - updated comment and rename of fields

SearchHeadLevel - Detect MongoDB errors - regex update to remove false positives

SearchHeadLevel - Indexer Peer Connection Failures - updated comment and sourcetype

SearchHeadLevel - platform_stats.user_stats.introspection metrics populating search - added rounding of fields, updated comment

SearchHeadLevel - platform_stats.users savedsearches - added time field

SearchHeadLevel - platform_stats.users dashboards - added time field

SearchHeadLevel - Scheduled Searches That Cannot Run - corrected failure count so it's accurate

SearchHeadLevel - Search Messages user level - more criteria and excluded some warnings

SearchHeadLevel - Search Queries summary exact match - updates to stats to include 1 more field, updated regex to match macros in multisearch commands, updated comment, removed extra ' character from search field

SearchHeadLevel - Search Queries summary non-exact match - updated comment, updated regex to match macros in multisearch commands, removed extra ' character from search field

Updated dashboards:
hec_performance - to include the additional num_of_requests_waiting_ack measurement from introspection data, if this is high it can stop data when tokens have useACK set to true

smartstore_stats - various new panels around queueing of downloads, and other potential smartstore issues

splunk_forwarder_output_tuning - update to include another measure of data balance

Updated comments on alerts:
AllSplunkLevel - Unable To Distribute to Peer

SearchHeadLevel - splunk_search_messages dispatch - description update

Updated metadata file to allow sc_admin role access

2.6.9

Updated alerts:
AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only - removed 1 log entry for consecutive date entries/unretrievable data

ForwarderLevel - Splunk HEC issues - added cluster command

New dashboards:
ForwarderLevel - Splunk HEC issues

New reports:
IndexerLevel - SmartStore cache misses - remote_searches

IndexerLevel - Buckets in cache

SearchHeadLevel - Detect searches hitting corrupt buckets

SearchHeadLevel - SmartStore cache misses - savedsearches

SearchHeadLevel - SmartStore cache misses - dashboards

SearchHeadLevel - SmartStore cache misses - combined

Updated SDK to 1.6.18

Updated alerts/reports to remove unncessary TERM() commands:
AllSplunkEnterpriseLevel - Losing Contact With Master Node

AllSplunkEnterpriseLevel - Replication Failures

AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only

ForwarderLevel - Splunk HEC issues - included lookup file to translate the HTTP code seen by client (based on the documentation version 8.2.3)

IndexerLevel - Data parsing error

IndexerLevel - IndexWriter pause duration

IndexerLevel - RemoteSearches Indexes Stats

IndexerLevel - RemoteSearches Indexes Stats Wilcard

IndexerLevel - RemoteSearches find all time searches

IndexerLevel - RemoteSearches find datamodel acceleration with wildcards

IndexerLevel - Slow peer from remote searches

SearchHeadLevel - Dashboards invalid character in splunkd

SearchHeadLevel - platform_stats.remote_searches metrics populating search

SearchHeadLevel - savedsearches invalid character in splunkd

SearchHeadLevel - Script failures in the last day

SearchHeadLevel - Search Messages field extractor slow

SearchHeadLevel - Search Messages user level

SearchHeadLevel - Search Messages admins only

2.6.8

New alerts:
AllSplunkLevel - No recent metrics.log data

New dashboards:
heavyforwarders_max_data_queue_sizes_by_name_v8 - this version uses tstats with PREFIX so only works with Splunk 8.0+

indexer_max_data_queue_sizes_by_name_v8 - this version uses tstats with PREFIX so only works with Splunk 8.0+

splunk_forwarder_output_tuning - using metrics.log to measure the TCP output/stdev per-name, includes example tuning parameters

New reports:
IndexerLevel - platform_stats.indexers stddev measurement - stdev per indexer cluster (useful for tuning the outputs.conf from incoming servers)

IndexerLevel - platform_stats.indexers totalgb_thruput measurement - index thruput measurements

Updated alerts:
AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only - more alert criteria

IndexerLevel - Cold data location approaching size limits - improvements to calculation of % used

IndexerLevel - Data parsing error - added macro splunkadmins_dataparsing_error as requested

SearchHeadLevel - Realtime Scheduled Searches are in use - updated timeout to 900 seconds, added context to description about potential use (as per feedback from Vincent)

SearchHeadLevel - Script failures in the last day - improved user id matching

SearchHeadLevel - Search Messages admins only - more alert criteria

SearchHeadLevel - Search Messages user level - more alert criteria

Updated macros:
splunkadmins_shutdown_keyword - updated keyword for shutdown state

splunkadmins_shutdown_list - updated keyword for shutdown state

splunkadmins_shutdown_time - updated keyword for shutdown state

Updated reports:
IndexerLevel - platform_stats.counters hosts - updated to use indexer_cluster_name macro

IndexerLevel - platform_stats.counters hosts 24hour - updated to use indexer_cluster_name macro

IndexerLevel - platform_stats.indexers totalgb measurement - updated to use indexer_cluster_name macro, comment update

IndexerLevel - RemoteSearches find datamodel acceleration with wildcards - handling the IN clause in remote_searches.log

IndexerLevel - RemoteSearches Indexes Stats - added short field

IndexerLevel - RemoteSearches Indexes Stats - added short field (set to False), to make queries easier

SearchHeadLevel - platform_stats.users dashboards - updated mcollect comment

SearchHeadLevel - Search Messages user level - added more error messages, limited the message to the first 30 messages

SearchHeadLevel - Search Messages admins only - added more error messages

SearchHeadLevel - Search Queries summary exact match - excluded Remote storage searches (no real difference)

SearchHeadLevel - Search Queries summary non-exact match - excluded Remote storage searches (no real difference)

2.6.7

New alerts:
IndexerLevel - SmartStore - Bucket cache errors audit logs

SearchHeadLevel - Accelerated DataModels with wildcard or no index specified

New reports:
IndexerLevel - IndexWriter pause duration

IndexerLevel - RemoteSearches find all time searches

IndexerLevel - RemoteSearches find datamodel acceleration with wildcards

SearchHeadLevel - platform_stats.audit metrics users 24hour

SearchHeadLevel - platform_stats.users dashboards

SearchHeadLevel - platform_stats.users savedsearches

Updated alerts:
AllSplunkEnterpriseLevel - sendmodalert errors - updated to refer to SearchHeadLevel - Script failures in the last day as it replaces most of this alerts functionality...

AllSplunkEnterpriseLevel - Splunkd Log Messages Admins Only - more alert criteria

DeploymentServer - Error Found On Deployment Server

SearchHeadLevel - audit logs showing all time searches - minor correction to display all searches without a savedsearch_name

SearchHeadLevel - Accelerated DataModels with All Time Searching Enabled - re-wrote the search to not use map

SearchHeadLevel - Script failures in the last day - updated to handle various webhook failures

Updated reports:
IndexerLevel - RemoteSearches Indexes Stats - updates to work with search heads with _ in the name, improved handling of "skipped" entries

IndexerLevel - RemoteSearches Indexes Stats Wilcard - updates to work with search heads with _ in the name, improved handling of "skipped" entries

SearchHeadLevel - Search Queries summary non-exact match - new field "short", updated regex

SearchHeadLevel - platform_stats.user_stats.introspection metrics populating search - updates to work with search heads with _ in the name

SearchHeadLevel - platform_stats.remote_searches metrics populating search - updates to work with search heads with _ in the name