|
| 1 | +# Splunkbase Release Description |
| 2 | + |
| 3 | +This document contains the content for the Splunkbase listing tabs. |
| 4 | + |
| 5 | +--- |
| 6 | + |
| 7 | +## Summary |
| 8 | + |
| 9 | +**CACA (Content Activity Checking Application)** helps Splunk teams track the usage, health, and lifecycle of their dashboards and content. Get clear, metric-driven insights to answer critical questions: "Is anyone using this dashboard I built?" • "Which dashboards are most critical to our users?" • "Is this dashboard performing well?" • "Which dashboards are safe to archive?" |
| 10 | + |
| 11 | +**Key Benefits:** |
| 12 | +- **Automated Discovery** - Automatically catalogs all dashboards across your environment |
| 13 | +- **Usage Analytics** - Track views, users, and engagement patterns over time |
| 14 | +- **Health Monitoring** - Detect and track dashboard errors and warnings |
| 15 | +- **Performance Tracking** - Identify slow-loading dashboards that need optimization |
| 16 | +- **Stale Detection** - Find unused dashboards that can be archived or deleted |
| 17 | +- **Efficient Storage** - Uses Splunk's native metrics index for low-impact data collection |
| 18 | + |
| 19 | +**Why CACA?** While Splunk Monitoring Console (DMC) focuses on system health for admins, CACA answers a different question for content creators and users: "Is my content useful and working well?" |
| 20 | + |
| 21 | +--- |
| 22 | + |
| 23 | +## Details |
| 24 | + |
| 25 | +### Features |
| 26 | + |
| 27 | +**Dashboard Monitoring** |
| 28 | +- Automated dashboard discovery across all apps (including private dashboards) |
| 29 | +- Real-time usage tracking (views by user and timestamp) |
| 30 | +- Edit history monitoring (creation and modification tracking) |
| 31 | +- Health monitoring from internal logs (errors and warnings) |
| 32 | +- Performance monitoring (load times and slow dashboard identification) |
| 33 | +- Stale dashboard detection (30+ days without access) |
| 34 | + |
| 35 | +**Metrics & Analytics** |
| 36 | +- Efficient metrics storage using Splunk's native metrics index |
| 37 | +- Near real-time collection with 5-15 minute scheduled searches |
| 38 | +- Historical trending up to 1 year (configurable retention) |
| 39 | +- Aggregate statistics across all content |
| 40 | + |
| 41 | +**Visualization & Reporting** |
| 42 | +- **Dashboard Leaderboard** - Centralized view of all monitored dashboards with sortable metrics |
| 43 | +- **CACA Admin Dashboard** - Multi-dimensional filtering, bulk management, and workflow guides |
| 44 | +- **Detailed Analytics** - Drill down into individual dashboard performance |
| 45 | +- **Health Status Indicators** - Color-coded badges (Healthy, Warning, Critical, Stale) |
| 46 | +- **Trending Charts** - Track views, errors, and performance over time |
| 47 | + |
| 48 | +**Embeddable Badges** |
| 49 | +- GitHub-style badges showing usage and health metrics |
| 50 | +- Customizable XML templates for quick integration |
| 51 | +- Multiple badge styles (single-stat or multi-metric) |
| 52 | +- Self-service implementation for dashboard owners |
| 53 | + |
| 54 | +**Search Macros** |
| 55 | +Pre-built macros for common queries: |
| 56 | +- `get_dashboards_with_errors` - Find dashboards with health issues |
| 57 | +- `get_slow_dashboards` - Identify performance problems |
| 58 | +- `get_problematic_dashboards` - All dashboards needing attention |
| 59 | +- `get_dashboard_stats("name")` - Comprehensive stats for specific dashboard |
| 60 | +- `get_all_dashboards_summary` - Overview of all dashboard metrics |
| 61 | +- `get_top_dashboards(metric)` - Top 10 dashboards by views or edits |
| 62 | + |
| 63 | +### Architecture |
| 64 | + |
| 65 | +Three-stage pipeline for efficiency: |
| 66 | +1. **Collect** - Scheduled searches analyze `_internal` and `_audit` logs |
| 67 | +2. **Store** - Metrics written to dedicated metrics index using `mcollect` |
| 68 | +3. **Query** - Fast retrieval via `mstats` and reusable search macros |
| 69 | + |
| 70 | +**Resource Impact:** Lightweight scheduled searches run at low priority to avoid impacting user searches or system performance. |
| 71 | + |
| 72 | +### Use Cases |
| 73 | + |
| 74 | +- **Content Governance** - Identify and archive unused dashboards |
| 75 | +- **Performance Optimization** - Find and fix slow-loading dashboards |
| 76 | +- **Health Monitoring** - Proactively detect dashboard errors |
| 77 | +- **Usage Analytics** - Understand which content delivers the most value |
| 78 | +- **Team Metrics** - Demonstrate content creation impact with real data |
| 79 | +- **Compliance** - Maintain audit trails for dashboard access and changes |
| 80 | + |
| 81 | +--- |
| 82 | + |
| 83 | +## Installation |
| 84 | + |
| 85 | +### Requirements |
| 86 | + |
| 87 | +- Splunk Enterprise 8.0 or later |
| 88 | +- Access to `_internal` and `_audit` indexes |
| 89 | +- Permissions to create metrics indexes and scheduled searches |
| 90 | +- Admin or appropriate capabilities to discover private dashboards (optional) |
| 91 | + |
| 92 | +### Installation Steps |
| 93 | + |
| 94 | +**Method 1: Via Splunk Web (Recommended)** |
| 95 | + |
| 96 | +1. Download the latest release package from Splunkbase |
| 97 | +2. In Splunk Web, navigate to **Apps → Manage Apps** |
| 98 | +3. Click **Install app from file** |
| 99 | +4. Upload the package file |
| 100 | +5. Click **Upload** |
| 101 | +6. Restart Splunk if prompted |
| 102 | + |
| 103 | +**Method 2: Manual Installation** |
| 104 | + |
| 105 | +1. Extract the app package to `$SPLUNK_HOME/etc/apps/` |
| 106 | +2. Ensure the directory is named `caca` |
| 107 | +3. Restart Splunk: |
| 108 | + ```bash |
| 109 | + $SPLUNK_HOME/bin/splunk restart |
| 110 | + ``` |
| 111 | + |
| 112 | +### Initial Setup (Required) |
| 113 | + |
| 114 | +After installation, complete these steps to activate CACA: |
| 115 | + |
| 116 | +**Step 1: Verify Index Creation** |
| 117 | + |
| 118 | +The `caca_metrics` index is created automatically. Verify: |
| 119 | +```spl |
| 120 | +| eventcount summarize=false index=caca_metrics |
| 121 | +``` |
| 122 | + |
| 123 | +**Step 2: Populate Dashboard Registry** |
| 124 | + |
| 125 | +Navigate to the **CACA app** in Splunk Web and run this search: |
| 126 | +```spl |
| 127 | +| rest /services/data/ui/views splunk_server=local count=0 search="sharing=*" |
| 128 | +| search isDashboard=1 OR isVisible=1 |
| 129 | +| eval dashboard_uri="/app/".eai:acl.app."/".title |
| 130 | +| eval pretty_name=coalesce(label, title) |
| 131 | +| eval app=eai:acl.app, owner=eai:acl.owner, sharing=eai:acl.sharing |
| 132 | +| eval description=coalesce(eai:data, ""), status="active" |
| 133 | +| table dashboard_uri pretty_name app owner sharing description status |
| 134 | +| outputlookup dashboard_registry.csv |
| 135 | +``` |
| 136 | + |
| 137 | +Verify the registry: |
| 138 | +```spl |
| 139 | +| inputlookup dashboard_registry | stats count |
| 140 | +``` |
| 141 | + |
| 142 | +**Step 3: Enable Scheduled Searches** |
| 143 | + |
| 144 | +Navigate to **Settings → Searches, reports, and alerts** and enable: |
| 145 | +- Dashboard Views - Metrics Collector (every 5 minutes) |
| 146 | +- Dashboard Edits - Metrics Collector (every 10 minutes) |
| 147 | +- Dashboard Performance - Metrics Collector (every 10 minutes) |
| 148 | +- Dashboard Health - Metrics Collector (every 15 minutes) |
| 149 | +- Dashboard Registry - Auto Update (daily at 2 AM) |
| 150 | + |
| 151 | +**Important:** These searches are disabled by default. Enable them only after completing Steps 1 and 2. |
| 152 | + |
| 153 | +**Step 4: Wait for Data Collection** |
| 154 | + |
| 155 | +Allow 15-30 minutes for initial metrics to populate. Verify: |
| 156 | +```spl |
| 157 | +| mstats count WHERE index=caca_metrics BY metric_name |
| 158 | +``` |
| 159 | + |
| 160 | +### Configuration (Optional) |
| 161 | + |
| 162 | +**Filter Apps for Monitoring** |
| 163 | +Edit `lookups/app_filter.csv` to include/exclude specific apps: |
| 164 | +```csv |
| 165 | +app,include |
| 166 | +production_app,true |
| 167 | +splunk_monitoring_console,false |
| 168 | +``` |
| 169 | + |
| 170 | +**Adjust Metrics Retention** |
| 171 | +Edit `local/indexes.conf`: |
| 172 | +```ini |
| 173 | +[caca_metrics] |
| 174 | +frozenTimePeriodInSecs = 31536000 # 1 year (default) |
| 175 | +``` |
| 176 | + |
| 177 | +**Customize Collection Schedules** |
| 178 | +Modify scheduled search frequencies in **Settings → Searches, reports, and alerts**. |
| 179 | + |
| 180 | +--- |
| 181 | + |
| 182 | +## Troubleshooting |
| 183 | + |
| 184 | +### No Data Appearing in Dashboards |
| 185 | + |
| 186 | +**Symptom:** CACA dashboards show zero metrics or "No results found" |
| 187 | + |
| 188 | +**Solutions:** |
| 189 | + |
| 190 | +1. **Verify scheduled searches are running:** |
| 191 | + ```spl |
| 192 | + index=_internal source=*scheduler.log savedsearch_name="Dashboard*Metrics*" |
| 193 | + | stats count by savedsearch_name status |
| 194 | + ``` |
| 195 | + - Ensure searches show `status=success` |
| 196 | + - If searches aren't running, verify they are enabled in **Settings → Searches, reports, and alerts** |
| 197 | + |
| 198 | +2. **Check metrics index exists and has data:** |
| 199 | + ```spl |
| 200 | + | eventcount summarize=false index=caca_metrics |
| 201 | + | where count > 0 |
| 202 | + ``` |
| 203 | + - If count is 0, wait 15-30 minutes for initial collection |
| 204 | + - Verify dashboards have been accessed since CACA was installed |
| 205 | + |
| 206 | +3. **Verify dashboard registry is populated:** |
| 207 | + ```spl |
| 208 | + | inputlookup dashboard_registry | stats count |
| 209 | + ``` |
| 210 | + - If count is 0, re-run the registry population search from Step 2 of Initial Setup |
| 211 | + - Ensure you run the search **from within the CACA app** in Splunk Web |
| 212 | + |
| 213 | +### Dashboard Not Appearing in Registry |
| 214 | + |
| 215 | +**Symptom:** Specific dashboard is missing from the registry or metrics |
| 216 | + |
| 217 | +**Solutions:** |
| 218 | + |
| 219 | +1. **Re-run registry update manually:** |
| 220 | + - Navigate to the **CACA app** in Splunk Web (important!) |
| 221 | + - Run the registry population search from Installation Step 2 |
| 222 | + - Verify the dashboard appears: `| inputlookup dashboard_registry | search pretty_name="Your Dashboard"` |
| 223 | + |
| 224 | +2. **Check app filter configuration:** |
| 225 | + - Verify `lookups/app_filter.csv` isn't excluding the dashboard's app |
| 226 | + - If app is listed with `include=false`, change to `true` or remove the entry |
| 227 | + |
| 228 | +3. **Private dashboard visibility:** |
| 229 | + - Private dashboards require admin privileges to discover via REST API |
| 230 | + - Ensure the registry update search runs with sufficient permissions |
| 231 | + - Alternatively, manually add to `lookups/dashboard_registry.csv` |
| 232 | + |
| 233 | +### Metrics Showing Zero Despite Dashboard Usage |
| 234 | + |
| 235 | +**Symptom:** Dashboard appears in registry but shows 0 views/edits/errors |
| 236 | + |
| 237 | +**Solutions:** |
| 238 | + |
| 239 | +1. **Check scheduled search permissions:** |
| 240 | + ```spl |
| 241 | + index=_internal source=*scheduler.log savedsearch_name="Dashboard Views - Metrics Collector" |
| 242 | + | table _time status message |
| 243 | + ``` |
| 244 | + - Look for permission errors or failed executions |
| 245 | + - Ensure search runs with role that has access to `_internal` and `_audit` indexes |
| 246 | + |
| 247 | +2. **Verify internal logs are accessible:** |
| 248 | + ```spl |
| 249 | + index=_internal sourcetype=splunkd_ui_access "/app/*" earliest=-1h |
| 250 | + | stats count |
| 251 | + ``` |
| 252 | + - If count is 0, check that `_internal` index is available |
| 253 | + - Verify audit logging is enabled in Splunk |
| 254 | + |
| 255 | +3. **Check for dashboard activity:** |
| 256 | + - Dashboards must be accessed **after** CACA is installed for metrics to appear |
| 257 | + - Open the dashboard manually to generate initial view event |
| 258 | + - Wait 5-15 minutes for collection searches to run |
| 259 | + |
| 260 | +### Error in 'outputlookup': Could not find all of the specified destination fields |
| 261 | + |
| 262 | +**Symptom:** Registry update search fails with outputlookup error |
| 263 | + |
| 264 | +**Solution:** |
| 265 | +- Ensure you are running the search **from within the CACA app context** in Splunk Web |
| 266 | +- Navigate to **CACA app** first, then run the search |
| 267 | +- Alternatively, use the `caca:` prefix: `| outputlookup caca:dashboard_registry.csv` |
| 268 | + |
| 269 | +### High Scheduler Load or Performance Impact |
| 270 | + |
| 271 | +**Symptom:** CACA searches impacting system performance |
| 272 | + |
| 273 | +**Solutions:** |
| 274 | + |
| 275 | +1. **Adjust collection frequency:** |
| 276 | + - Reduce scheduled search frequency in **Settings → Searches, reports, and alerts** |
| 277 | + - Example: Change view collection from 5 minutes to 15 minutes |
| 278 | + |
| 279 | +2. **Filter monitored apps:** |
| 280 | + - Edit `lookups/app_filter.csv` to exclude non-critical apps |
| 281 | + - Focus monitoring on production apps only |
| 282 | + |
| 283 | +3. **Reduce metrics retention:** |
| 284 | + - Edit `local/indexes.conf` and decrease `frozenTimePeriodInSecs` |
| 285 | + - Default is 1 year; consider 90 or 180 days for less storage |
| 286 | + |
| 287 | +### Dashboard Health Status Incorrect |
| 288 | + |
| 289 | +**Symptom:** Dashboard shows errors but appears healthy (or vice versa) |
| 290 | + |
| 291 | +**Solutions:** |
| 292 | + |
| 293 | +1. **Check error thresholds:** |
| 294 | + - Health status is based on 7-day error counts |
| 295 | + - Review dashboard detail view to see actual error counts |
| 296 | + - Recent errors may not impact 7-day average immediately |
| 297 | + |
| 298 | +2. **Manually verify errors:** |
| 299 | + ```spl |
| 300 | + index=_internal source=*splunkd.log dashboard_id="*your_dashboard*" (ERROR OR WARN) |
| 301 | + | stats count by log_level |
| 302 | + ``` |
| 303 | + |
| 304 | +### Need Additional Help? |
| 305 | + |
| 306 | +- Review the built-in **CACA Admin Dashboard** for dashboard management guidance |
| 307 | +- Check `README.md` in the app directory for detailed documentation |
| 308 | +- Visit the GitHub repository for issues and community support |
| 309 | +- Contact Splunk support for platform-level issues |
0 commit comments