Skip to content

Commit 035b869

Browse files
committed
Fix scheduled searches: correct mcollect syntax, escape field names, add dispatchAs owner
- Remove backslash line continuations (not supported in savedsearches.conf) - Fix mcollect syntax: create metric_name field, reference it instead of inline assignment - Escape REST API field names with single quotes ('eai:acl.app', 'eai:acl.owner', etc.) - Remove search='sharing=*' parameter from REST command (causing 0 results) - Add dispatchAs=owner to all searches for proper permissions (_internal, _audit access) - Use _value field for dashboard.load_time metric values - Add RELEASE-DESCRIPTION.md with Splunkbase listing content
1 parent 423c939 commit 035b869

File tree

3 files changed

+526
-59
lines changed

3 files changed

+526
-59
lines changed

RELEASE-DESCRIPTION.md

Lines changed: 309 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,309 @@
1+
# Splunkbase Release Description
2+
3+
This document contains the content for the Splunkbase listing tabs.
4+
5+
---
6+
7+
## Summary
8+
9+
**CACA (Content Activity Checking Application)** helps Splunk teams track the usage, health, and lifecycle of their dashboards and content. Get clear, metric-driven insights to answer critical questions: "Is anyone using this dashboard I built?" • "Which dashboards are most critical to our users?" • "Is this dashboard performing well?" • "Which dashboards are safe to archive?"
10+
11+
**Key Benefits:**
12+
- **Automated Discovery** - Automatically catalogs all dashboards across your environment
13+
- **Usage Analytics** - Track views, users, and engagement patterns over time
14+
- **Health Monitoring** - Detect and track dashboard errors and warnings
15+
- **Performance Tracking** - Identify slow-loading dashboards that need optimization
16+
- **Stale Detection** - Find unused dashboards that can be archived or deleted
17+
- **Efficient Storage** - Uses Splunk's native metrics index for low-impact data collection
18+
19+
**Why CACA?** While Splunk Monitoring Console (DMC) focuses on system health for admins, CACA answers a different question for content creators and users: "Is my content useful and working well?"
20+
21+
---
22+
23+
## Details
24+
25+
### Features
26+
27+
**Dashboard Monitoring**
28+
- Automated dashboard discovery across all apps (including private dashboards)
29+
- Real-time usage tracking (views by user and timestamp)
30+
- Edit history monitoring (creation and modification tracking)
31+
- Health monitoring from internal logs (errors and warnings)
32+
- Performance monitoring (load times and slow dashboard identification)
33+
- Stale dashboard detection (30+ days without access)
34+
35+
**Metrics & Analytics**
36+
- Efficient metrics storage using Splunk's native metrics index
37+
- Near real-time collection with 5-15 minute scheduled searches
38+
- Historical trending up to 1 year (configurable retention)
39+
- Aggregate statistics across all content
40+
41+
**Visualization & Reporting**
42+
- **Dashboard Leaderboard** - Centralized view of all monitored dashboards with sortable metrics
43+
- **CACA Admin Dashboard** - Multi-dimensional filtering, bulk management, and workflow guides
44+
- **Detailed Analytics** - Drill down into individual dashboard performance
45+
- **Health Status Indicators** - Color-coded badges (Healthy, Warning, Critical, Stale)
46+
- **Trending Charts** - Track views, errors, and performance over time
47+
48+
**Embeddable Badges**
49+
- GitHub-style badges showing usage and health metrics
50+
- Customizable XML templates for quick integration
51+
- Multiple badge styles (single-stat or multi-metric)
52+
- Self-service implementation for dashboard owners
53+
54+
**Search Macros**
55+
Pre-built macros for common queries:
56+
- `get_dashboards_with_errors` - Find dashboards with health issues
57+
- `get_slow_dashboards` - Identify performance problems
58+
- `get_problematic_dashboards` - All dashboards needing attention
59+
- `get_dashboard_stats("name")` - Comprehensive stats for specific dashboard
60+
- `get_all_dashboards_summary` - Overview of all dashboard metrics
61+
- `get_top_dashboards(metric)` - Top 10 dashboards by views or edits
62+
63+
### Architecture
64+
65+
Three-stage pipeline for efficiency:
66+
1. **Collect** - Scheduled searches analyze `_internal` and `_audit` logs
67+
2. **Store** - Metrics written to dedicated metrics index using `mcollect`
68+
3. **Query** - Fast retrieval via `mstats` and reusable search macros
69+
70+
**Resource Impact:** Lightweight scheduled searches run at low priority to avoid impacting user searches or system performance.
71+
72+
### Use Cases
73+
74+
- **Content Governance** - Identify and archive unused dashboards
75+
- **Performance Optimization** - Find and fix slow-loading dashboards
76+
- **Health Monitoring** - Proactively detect dashboard errors
77+
- **Usage Analytics** - Understand which content delivers the most value
78+
- **Team Metrics** - Demonstrate content creation impact with real data
79+
- **Compliance** - Maintain audit trails for dashboard access and changes
80+
81+
---
82+
83+
## Installation
84+
85+
### Requirements
86+
87+
- Splunk Enterprise 8.0 or later
88+
- Access to `_internal` and `_audit` indexes
89+
- Permissions to create metrics indexes and scheduled searches
90+
- Admin or appropriate capabilities to discover private dashboards (optional)
91+
92+
### Installation Steps
93+
94+
**Method 1: Via Splunk Web (Recommended)**
95+
96+
1. Download the latest release package from Splunkbase
97+
2. In Splunk Web, navigate to **Apps → Manage Apps**
98+
3. Click **Install app from file**
99+
4. Upload the package file
100+
5. Click **Upload**
101+
6. Restart Splunk if prompted
102+
103+
**Method 2: Manual Installation**
104+
105+
1. Extract the app package to `$SPLUNK_HOME/etc/apps/`
106+
2. Ensure the directory is named `caca`
107+
3. Restart Splunk:
108+
```bash
109+
$SPLUNK_HOME/bin/splunk restart
110+
```
111+
112+
### Initial Setup (Required)
113+
114+
After installation, complete these steps to activate CACA:
115+
116+
**Step 1: Verify Index Creation**
117+
118+
The `caca_metrics` index is created automatically. Verify:
119+
```spl
120+
| eventcount summarize=false index=caca_metrics
121+
```
122+
123+
**Step 2: Populate Dashboard Registry**
124+
125+
Navigate to the **CACA app** in Splunk Web and run this search:
126+
```spl
127+
| rest /services/data/ui/views splunk_server=local count=0 search="sharing=*"
128+
| search isDashboard=1 OR isVisible=1
129+
| eval dashboard_uri="/app/".eai:acl.app."/".title
130+
| eval pretty_name=coalesce(label, title)
131+
| eval app=eai:acl.app, owner=eai:acl.owner, sharing=eai:acl.sharing
132+
| eval description=coalesce(eai:data, ""), status="active"
133+
| table dashboard_uri pretty_name app owner sharing description status
134+
| outputlookup dashboard_registry.csv
135+
```
136+
137+
Verify the registry:
138+
```spl
139+
| inputlookup dashboard_registry | stats count
140+
```
141+
142+
**Step 3: Enable Scheduled Searches**
143+
144+
Navigate to **Settings → Searches, reports, and alerts** and enable:
145+
- Dashboard Views - Metrics Collector (every 5 minutes)
146+
- Dashboard Edits - Metrics Collector (every 10 minutes)
147+
- Dashboard Performance - Metrics Collector (every 10 minutes)
148+
- Dashboard Health - Metrics Collector (every 15 minutes)
149+
- Dashboard Registry - Auto Update (daily at 2 AM)
150+
151+
**Important:** These searches are disabled by default. Enable them only after completing Steps 1 and 2.
152+
153+
**Step 4: Wait for Data Collection**
154+
155+
Allow 15-30 minutes for initial metrics to populate. Verify:
156+
```spl
157+
| mstats count WHERE index=caca_metrics BY metric_name
158+
```
159+
160+
### Configuration (Optional)
161+
162+
**Filter Apps for Monitoring**
163+
Edit `lookups/app_filter.csv` to include/exclude specific apps:
164+
```csv
165+
app,include
166+
production_app,true
167+
splunk_monitoring_console,false
168+
```
169+
170+
**Adjust Metrics Retention**
171+
Edit `local/indexes.conf`:
172+
```ini
173+
[caca_metrics]
174+
frozenTimePeriodInSecs = 31536000 # 1 year (default)
175+
```
176+
177+
**Customize Collection Schedules**
178+
Modify scheduled search frequencies in **Settings → Searches, reports, and alerts**.
179+
180+
---
181+
182+
## Troubleshooting
183+
184+
### No Data Appearing in Dashboards
185+
186+
**Symptom:** CACA dashboards show zero metrics or "No results found"
187+
188+
**Solutions:**
189+
190+
1. **Verify scheduled searches are running:**
191+
```spl
192+
index=_internal source=*scheduler.log savedsearch_name="Dashboard*Metrics*"
193+
| stats count by savedsearch_name status
194+
```
195+
- Ensure searches show `status=success`
196+
- If searches aren't running, verify they are enabled in **Settings → Searches, reports, and alerts**
197+
198+
2. **Check metrics index exists and has data:**
199+
```spl
200+
| eventcount summarize=false index=caca_metrics
201+
| where count > 0
202+
```
203+
- If count is 0, wait 15-30 minutes for initial collection
204+
- Verify dashboards have been accessed since CACA was installed
205+
206+
3. **Verify dashboard registry is populated:**
207+
```spl
208+
| inputlookup dashboard_registry | stats count
209+
```
210+
- If count is 0, re-run the registry population search from Step 2 of Initial Setup
211+
- Ensure you run the search **from within the CACA app** in Splunk Web
212+
213+
### Dashboard Not Appearing in Registry
214+
215+
**Symptom:** Specific dashboard is missing from the registry or metrics
216+
217+
**Solutions:**
218+
219+
1. **Re-run registry update manually:**
220+
- Navigate to the **CACA app** in Splunk Web (important!)
221+
- Run the registry population search from Installation Step 2
222+
- Verify the dashboard appears: `| inputlookup dashboard_registry | search pretty_name="Your Dashboard"`
223+
224+
2. **Check app filter configuration:**
225+
- Verify `lookups/app_filter.csv` isn't excluding the dashboard's app
226+
- If app is listed with `include=false`, change to `true` or remove the entry
227+
228+
3. **Private dashboard visibility:**
229+
- Private dashboards require admin privileges to discover via REST API
230+
- Ensure the registry update search runs with sufficient permissions
231+
- Alternatively, manually add to `lookups/dashboard_registry.csv`
232+
233+
### Metrics Showing Zero Despite Dashboard Usage
234+
235+
**Symptom:** Dashboard appears in registry but shows 0 views/edits/errors
236+
237+
**Solutions:**
238+
239+
1. **Check scheduled search permissions:**
240+
```spl
241+
index=_internal source=*scheduler.log savedsearch_name="Dashboard Views - Metrics Collector"
242+
| table _time status message
243+
```
244+
- Look for permission errors or failed executions
245+
- Ensure search runs with role that has access to `_internal` and `_audit` indexes
246+
247+
2. **Verify internal logs are accessible:**
248+
```spl
249+
index=_internal sourcetype=splunkd_ui_access "/app/*" earliest=-1h
250+
| stats count
251+
```
252+
- If count is 0, check that `_internal` index is available
253+
- Verify audit logging is enabled in Splunk
254+
255+
3. **Check for dashboard activity:**
256+
- Dashboards must be accessed **after** CACA is installed for metrics to appear
257+
- Open the dashboard manually to generate initial view event
258+
- Wait 5-15 minutes for collection searches to run
259+
260+
### Error in 'outputlookup': Could not find all of the specified destination fields
261+
262+
**Symptom:** Registry update search fails with outputlookup error
263+
264+
**Solution:**
265+
- Ensure you are running the search **from within the CACA app context** in Splunk Web
266+
- Navigate to **CACA app** first, then run the search
267+
- Alternatively, use the `caca:` prefix: `| outputlookup caca:dashboard_registry.csv`
268+
269+
### High Scheduler Load or Performance Impact
270+
271+
**Symptom:** CACA searches impacting system performance
272+
273+
**Solutions:**
274+
275+
1. **Adjust collection frequency:**
276+
- Reduce scheduled search frequency in **Settings → Searches, reports, and alerts**
277+
- Example: Change view collection from 5 minutes to 15 minutes
278+
279+
2. **Filter monitored apps:**
280+
- Edit `lookups/app_filter.csv` to exclude non-critical apps
281+
- Focus monitoring on production apps only
282+
283+
3. **Reduce metrics retention:**
284+
- Edit `local/indexes.conf` and decrease `frozenTimePeriodInSecs`
285+
- Default is 1 year; consider 90 or 180 days for less storage
286+
287+
### Dashboard Health Status Incorrect
288+
289+
**Symptom:** Dashboard shows errors but appears healthy (or vice versa)
290+
291+
**Solutions:**
292+
293+
1. **Check error thresholds:**
294+
- Health status is based on 7-day error counts
295+
- Review dashboard detail view to see actual error counts
296+
- Recent errors may not impact 7-day average immediately
297+
298+
2. **Manually verify errors:**
299+
```spl
300+
index=_internal source=*splunkd.log dashboard_id="*your_dashboard*" (ERROR OR WARN)
301+
| stats count by log_level
302+
```
303+
304+
### Need Additional Help?
305+
306+
- Review the built-in **CACA Admin Dashboard** for dashboard management guidance
307+
- Check `README.md` in the app directory for detailed documentation
308+
- Visit the GitHub repository for issues and community support
309+
- Contact Splunk support for platform-level issues

0 commit comments

Comments
 (0)