Skip to content

Commit 2cf43ce

Browse files
Copilotdevinslick
Copilot
and
devinslick
committed
Add dashboard performance (load time) tracking functionality
Co-authored-by: devinslick <[email protected]>
1 parent b956511 commit 2cf43ce

File tree

6 files changed

+280
-12
lines changed

6 files changed

+280
-12
lines changed

‎README.md‎

Lines changed: 23 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ While the Splunk Monitoring Console (DMC) focuses on system-level health and per
2626
- **Usage Tracking**: Track how many times each dashboard is viewed, by which users, and when
2727
- **Edit History**: Monitor when dashboards are created or modified, and by whom
2828
- **Health Monitoring**: Detect and track dashboard errors and warnings from internal logs
29+
- **Performance Monitoring**: Track dashboard load times and identify slow-performing dashboards
2930
- **Stale Dashboard Detection**: Identify dashboards that haven't been accessed in 30+ days
3031

3132
#### Metrics & Analytics
@@ -56,13 +57,14 @@ While the Splunk Monitoring Console (DMC) focuses on system-level health and per
5657

5758
The app uses a three-stage pipeline for efficiency:
5859

59-
1. **Collect**: Scheduled searches analyze Splunk's internal logs (`_internal`, `_audit`) to track views, edits, and errors
60+
1. **Collect**: Scheduled searches analyze Splunk's internal logs (`_internal`, `_audit`) to track views, edits, errors, and performance
6061
2. **Store**: Metrics are written to a dedicated metrics index using `mcollect` for optimal performance
6162
3. **Query**: Fast retrieval via `mstats` command and reusable search macros
6263

6364
**Note on Scheduled Searches**: CACA is powered by lightweight scheduled searches that run at the following intervals:
6465
- Dashboard views: Every 5 minutes
65-
- Dashboard edits: Every 10 minutes
66+
- Dashboard edits: Every 10 minutes
67+
- Dashboard performance: Every 10 minutes
6668
- Dashboard health: Every 15 minutes
6769
- Registry updates: Daily at 2 AM
6870

@@ -140,6 +142,7 @@ Navigate to **Settings → Searches, reports, and alerts** and enable these sear
140142

141143
- **Dashboard Views - Metrics Collector** (runs every 5 minutes)
142144
- **Dashboard Edits - Metrics Collector** (runs every 10 minutes)
145+
- **Dashboard Performance - Metrics Collector** (runs every 10 minutes)
143146
- **Dashboard Health - Metrics Collector** (runs every 15 minutes)
144147
- **Dashboard Registry - Auto Update** (runs daily at 2 AM)
145148

@@ -161,20 +164,21 @@ Allow 15-30 minutes for the initial data collection to populate the metrics inde
161164

162165
Navigate to **CACA → Dashboard Leaderboard** to view:
163166

164-
- **High-Level KPIs**: Total dashboards, views, errors, and stale dashboards
165-
- **Activity Leaderboard Table**: Sortable list of all dashboards with metrics
166-
- **Trending Charts**: Views and errors over time
167-
- **Top Dashboards**: Most viewed and most edited dashboards
167+
- **High-Level KPIs**: Total dashboards, views, errors, average load time, and stale dashboards
168+
- **Activity Leaderboard Table**: Sortable list of all dashboards with usage, health, and performance metrics
169+
- **Trending Charts**: Views, errors, and load time trends over time
170+
- **Top Dashboards**: Most viewed, most edited, and slowest dashboards
168171

169172
### Dashboard Details
170173

171174
Click any dashboard in the leaderboard to view detailed metrics:
172175

173-
- Total views, edits, and errors
174-
- Activity trends over time
176+
- Total views, edits, errors, and average load time
177+
- Activity and performance trends over time
175178
- Top users by views
176179
- Edit history
177180
- Error details with severity
181+
- Load time analysis (average and maximum)
178182

179183
### Adding Badges to Your Dashboards
180184

@@ -223,6 +227,16 @@ CACA provides several search macros for easy querying:
223227
`get_top_dashboards(views)`
224228
```
225229

230+
#### Get dashboard performance:
231+
```spl
232+
`get_dashboard_performance("My Dashboard Name")`
233+
```
234+
235+
#### Get slow dashboards:
236+
```spl
237+
`get_slow_dashboards`
238+
```
239+
226240
#### Get last viewed time:
227241
```spl
228242
`get_dashboard_last_viewed("My Dashboard Name")`
@@ -236,6 +250,7 @@ Edit `default/savedsearches.conf` or use Splunk Web to modify:
236250

237251
- **View tracking frequency**: Default every 5 minutes
238252
- **Edit tracking frequency**: Default every 10 minutes
253+
- **Performance tracking frequency**: Default every 10 minutes
239254
- **Health tracking frequency**: Default every 15 minutes
240255
- **Registry update frequency**: Default daily at 2 AM
241256

‎default/data/ui/views/dashboard_details.xml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,25 @@
8585
</single>
8686
</panel>
8787

88+
<panel>
89+
<title>Avg Load Time</title>
90+
<single>
91+
<search>
92+
<query>| mstats avg(avg_load_time) as avg_load WHERE index=caca_metrics AND pretty_name="$dashboard_name$" AND metric_name="dashboard.load_time" span=1d
93+
| stats avg(avg_load) as avg_load_time
94+
| eval display=round(avg_load_time, 0)." ms"
95+
| table display</query>
96+
<earliest>$time_range.earliest$</earliest>
97+
<latest>$time_range.latest$</latest>
98+
</search>
99+
<option name="drilldown">none</option>
100+
<option name="colorMode">block</option>
101+
<option name="rangeColors">["0x53a051","0x0877a6","0xf8be34","0xdc4e41"]</option>
102+
<option name="rangeValues">[1000,3000,5000]</option>
103+
<option name="underLabel">Performance</option>
104+
</single>
105+
</panel>
106+
88107
<panel>
89108
<title>Last Viewed</title>
90109
<single>
@@ -119,6 +138,24 @@
119138
<option name="charting.drilldown">none</option>
120139
</chart>
121140
</panel>
141+
142+
<panel>
143+
<title>Load Time Trend Over Time</title>
144+
<chart>
145+
<search>
146+
<query>| mstats avg(avg_load_time) as avg_load, max(max_load_time) as max_load WHERE index=caca_metrics AND pretty_name="$dashboard_name$" AND metric_name="dashboard.load_time" span=1h
147+
| timechart avg(avg_load) as "Avg Load Time (ms)" max(max_load) as "Max Load Time (ms)" span=1h</query>
148+
<earliest>$time_range.earliest$</earliest>
149+
<latest>$time_range.latest$</latest>
150+
</search>
151+
<option name="charting.chart">line</option>
152+
<option name="charting.axisTitleX.text">Date</option>
153+
<option name="charting.axisTitleY.text">Load Time (ms)</option>
154+
<option name="charting.legend.placement">bottom</option>
155+
<option name="charting.seriesColors">[0xF8BE34,0xDC4E41]</option>
156+
<option name="charting.drilldown">none</option>
157+
</chart>
158+
</panel>
122159
</row>
123160

124161
<!-- User Activity Breakdown -->

‎default/data/ui/views/dashboard_leaderboard.xml‎

Lines changed: 81 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,24 @@
6666
</single>
6767
</panel>
6868

69+
<panel>
70+
<title>Avg Load Time (7d)</title>
71+
<single>
72+
<search>
73+
<query>| mstats avg(avg_load_time) as overall_avg WHERE index=caca_metrics AND metric_name="dashboard.load_time" span=1d
74+
| stats avg(overall_avg) as avg_load_time
75+
| eval avg_load_time=round(avg_load_time, 0)." ms"</query>
76+
<earliest>-7d@h</earliest>
77+
<latest>now</latest>
78+
</search>
79+
<option name="drilldown">none</option>
80+
<option name="colorMode">block</option>
81+
<option name="rangeColors">["0x53a051","0x0877a6","0xf8be34","0xf1813f","0xdc4e41"]</option>
82+
<option name="rangeValues">[1000,3000,5000,10000]</option>
83+
<option name="underLabel">Performance</option>
84+
</single>
85+
</panel>
86+
6987
<panel>
7088
<title>Stale Dashboards</title>
7189
<single>
@@ -94,14 +112,21 @@
94112
<search>
95113
<query>`get_all_dashboards_summary`
96114
| lookup dashboard_registry pretty_name OUTPUT owner description
115+
| eval avg_load_time_7d=round(avg_load_time_7d, 0)
116+
| eval perf_rating=case(
117+
avg_load_time_7d == 0, "-",
118+
avg_load_time_7d < 1000, "⚡ Fast",
119+
avg_load_time_7d < 3000, "✓ Good",
120+
avg_load_time_7d < 5000, "⚠ Slow",
121+
1=1, "✗ Very Slow")
97122
| eval health_badge=case(
98123
health_status=="healthy", "✓ Healthy",
99124
health_status=="warning", "⚠ Warning",
100125
health_status=="critical", "✗ Critical",
101126
health_status=="stale", "☾ Stale",
102127
1=1, health_status)
103-
| table pretty_name app views_7d edits_7d errors_7d health_badge owner description
104-
| rename pretty_name as "Dashboard Name", app as "App", views_7d as "Views (7d)", edits_7d as "Edits (7d)", errors_7d as "Errors (7d)", health_badge as "Health Status", owner as "Owner", description as "Description"
128+
| table pretty_name app views_7d edits_7d errors_7d avg_load_time_7d perf_rating health_badge owner description
129+
| rename pretty_name as "Dashboard Name", app as "App", views_7d as "Views (7d)", edits_7d as "Edits (7d)", errors_7d as "Errors (7d)", avg_load_time_7d as "Avg Load (ms)", perf_rating as "Performance", health_badge as "Health Status", owner as "Owner", description as "Description"
105130
| sort - "Views (7d)"</query>
106131
<earliest>$time_range.earliest$</earliest>
107132
<latest>$time_range.latest$</latest>
@@ -114,6 +139,9 @@
114139
<format type="color" field="Health Status">
115140
<colorPalette type="map">{"✓ Healthy":#53A051,"⚠ Warning":#F8BE34,"✗ Critical":#DC4E41,"☾ Stale":#708794}</colorPalette>
116141
</format>
142+
<format type="color" field="Performance">
143+
<colorPalette type="map">{"⚡ Fast":#53A051,"✓ Good":#0877a6,"⚠ Slow":#F8BE34,"✗ Very Slow":#DC4E41,"-":#708794}</colorPalette>
144+
</format>
117145
<format type="number" field="Views (7d)">
118146
<option name="precision">0</option>
119147
</format>
@@ -123,6 +151,9 @@
123151
<format type="number" field="Errors (7d)">
124152
<option name="precision">0</option>
125153
</format>
154+
<format type="number" field="Avg Load (ms)">
155+
<option name="precision">0</option>
156+
</format>
126157
<drilldown>
127158
<link target="_blank">/app/splunk-content-monitoring-console/dashboard_details?form.dashboard_name=$row.Dashboard Name$</link>
128159
</drilldown>
@@ -168,6 +199,54 @@
168199
</panel>
169200
</row>
170201

202+
<!-- Performance Trending -->
203+
<row>
204+
<panel>
205+
<title>Average Load Time Trend (Last 7 Days)</title>
206+
<chart>
207+
<search>
208+
<query>| mstats avg(avg_load_time) as avg_load WHERE index=caca_metrics AND metric_name="dashboard.load_time" span=1d
209+
| timechart avg(avg_load) as "Avg Load Time (ms)" span=1d</query>
210+
<earliest>-7d@h</earliest>
211+
<latest>now</latest>
212+
</search>
213+
<option name="charting.chart">line</option>
214+
<option name="charting.axisTitleX.text">Time</option>
215+
<option name="charting.axisTitleY.text">Load Time (ms)</option>
216+
<option name="charting.legend.placement">bottom</option>
217+
<option name="charting.seriesColors">[0xF8BE34]</option>
218+
<option name="charting.drilldown">none</option>
219+
</chart>
220+
</panel>
221+
222+
<panel>
223+
<title>Slowest Dashboards (7d Avg)</title>
224+
<table>
225+
<search>
226+
<query>`get_slow_dashboards`
227+
| head 10
228+
| rename pretty_name as "Dashboard", app as "App", avg_load_time_7d as "Avg Load (ms)", max_load_time_7d as "Max Load (ms)", performance_status as "Status"</query>
229+
<earliest>-7d@h</earliest>
230+
<latest>now</latest>
231+
</search>
232+
<option name="drilldown">row</option>
233+
<option name="count">10</option>
234+
<format type="color" field="Status">
235+
<colorPalette type="map">{"Critical":#DC4E41,"Poor":#F1813F,"Fair":#F8BE34,"Good":#53A051}</colorPalette>
236+
</format>
237+
<format type="number" field="Avg Load (ms)">
238+
<option name="precision">0</option>
239+
</format>
240+
<format type="number" field="Max Load (ms)">
241+
<option name="precision">0</option>
242+
</format>
243+
<drilldown>
244+
<link target="_blank">/app/splunk-content-monitoring-console/dashboard_details?form.dashboard_name=$row.Dashboard$</link>
245+
</drilldown>
246+
</table>
247+
</panel>
248+
</row>
249+
171250
<!-- Top Dashboards -->
172251
<row>
173252
<panel>

‎default/data/ui/views/poop_deck.xml‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,26 @@
8282
</single>
8383
</panel>
8484

85+
<panel>
86+
<title>🐌 Slow Dashboards</title>
87+
<single>
88+
<search>
89+
<query>| mstats avg(avg_load_time) as avg_load WHERE index=caca_metrics AND metric_name="dashboard.load_time" BY pretty_name span=1d
90+
| where _time >= relative_time(now(), "-7d")
91+
| stats avg(avg_load) as avg_load_time by pretty_name
92+
| where avg_load_time > 5000
93+
| stats count as slow_count</query>
94+
<earliest>-7d@d</earliest>
95+
<latest>now</latest>
96+
</search>
97+
<option name="drilldown">none</option>
98+
<option name="colorMode">block</option>
99+
<option name="rangeColors">["0x53a051","0xf8be34","0xdc4e41"]</option>
100+
<option name="rangeValues">[1,5]</option>
101+
<option name="underLabel">Performance Issues</option>
102+
</single>
103+
</panel>
104+
85105
<panel>
86106
<title>👻 Ghost Dashboards</title>
87107
<single>
@@ -259,6 +279,69 @@
259279
</panel>
260280
</row>
261281

282+
<!-- The Slow: Performance Issues -->
283+
<row>
284+
<panel>
285+
<title>🐌 The Slow - Dashboards with Performance Issues (Last 7 Days)</title>
286+
<table>
287+
<search>
288+
<query>| mstats avg(avg_load_time) as avg_load, max(max_load_time) as max_load WHERE index=caca_metrics AND metric_name="dashboard.load_time" BY pretty_name, app span=1d
289+
| where _time >= relative_time(now(), "-7d")
290+
| stats avg(avg_load) as avg_load_time, max(max_load) as max_load_time by pretty_name, app
291+
| where avg_load_time > 3000
292+
| eval avg_load_time=round(avg_load_time, 0)
293+
| eval max_load_time=round(max_load_time, 0)
294+
| lookup dashboard_registry pretty_name OUTPUT owner description
295+
| join type=left pretty_name
296+
[| mstats sum(_value) as views WHERE index=caca_metrics AND metric_name="dashboard.views" BY pretty_name span=1d
297+
| where _time >= relative_time(now(), "-7d")
298+
| stats sum(views) as views_7d by pretty_name]
299+
| fillnull value=0 views_7d
300+
| eval performance_level=case(
301+
avg_load_time > 10000, "🔥 Critical",
302+
avg_load_time > 7000, "❌ Very Slow",
303+
avg_load_time > 5000, "⚠️ Slow",
304+
1=1, "😐 Sluggish")
305+
| eval user_impact=case(
306+
views_7d > 100 AND avg_load_time > 5000, "HIGH - Poor user experience",
307+
views_7d > 10 AND avg_load_time > 5000, "MEDIUM - Affecting users",
308+
avg_load_time > 10000, "HIGH - Unacceptably slow",
309+
1=1, "LOW - Limited impact")
310+
| eval recommendation=case(
311+
avg_load_time > 10000, "URGENT: Optimize or disable",
312+
avg_load_time > 7000, "High priority optimization",
313+
avg_load_time > 5000, "Should optimize",
314+
1=1, "Consider optimization")
315+
| table pretty_name app avg_load_time max_load_time views_7d performance_level user_impact recommendation owner description
316+
| rename pretty_name as "Dashboard", app as "App", avg_load_time as "Avg Load (ms)", max_load_time as "Max Load (ms)", views_7d as "Views (7d)", performance_level as "Performance", user_impact as "Impact", recommendation as "Recommendation", owner as "Owner", description as "Description"
317+
| sort -"Avg Load (ms)"</query>
318+
<earliest>-7d@d</earliest>
319+
<latest>now</latest>
320+
</search>
321+
<option name="drilldown">row</option>
322+
<option name="count">30</option>
323+
<format type="color" field="Performance">
324+
<colorPalette type="map">{"🔥 Critical":#DC4E41,"❌ Very Slow":#F1813F,"⚠️ Slow":#F8BE34,"😐 Sluggish":#0877a6}</colorPalette>
325+
</format>
326+
<format type="number" field="Avg Load (ms)">
327+
<option name="precision">0</option>
328+
</format>
329+
<format type="number" field="Max Load (ms)">
330+
<option name="precision">0</option>
331+
</format>
332+
<format type="number" field="Views (7d)">
333+
<option name="precision">0</option>
334+
</format>
335+
<format type="number" field="Views (7d)">
336+
<option name="precision">0</option>
337+
</format>
338+
<drilldown>
339+
<link target="_blank">/app/splunk-content-monitoring-console/dashboard_details?form.dashboard_name=$row.Dashboard$</link>
340+
</drilldown>
341+
</table>
342+
</panel>
343+
</row>
344+
262345
<!-- The Neglected: Low Engagement -->
263346
<row>
264347
<panel>

0 commit comments

Comments
 (0)