Merge pull request #350 from derkkila-splunk/OBS-122-workshop

Obs 122 workshop

Author: rcastley

content/en/ninja-workshops/12-alerting_monitoring_with_itsi/1-getting-started/1-access-cloud-instances.md

@@ -4,6 +4,14 @@ linkTitle: 1.1 How to connect to your workshop environment
 weight: 2
 ---
 
-Access Show and sping up Workshop
+# Starting up your Workshop
 
-TBD
+This workshop is available on Splunk Show and will take some time to start up all of your resources. It contains a Splunk environment with IT Service Intelligence, the Splunk Infrastructure Monitoring Add-On, as well as the recently updated AppDynamics Add-on all preconfigured.
+
+The Workshop is titled **"Tech Summit 2025: OBS-122"** or you can go directly to it's entry on [Splunk Show](https://show.splunk.com/template/646/). It takes approximately 15 minutes to start up however data generation and ingestion will take up to a half hour.
+
+![show-entry](../images/show_entry.png?classes=inline)
+
+# Splunk Observability Cloud Access
+
+Creating an alert in Observability Cloud should be done in the Observability Cloud US1 [Show Playground](https://app.us1.signalfx.com/#/home) Org.

content/en/ninja-workshops/12-alerting_monitoring_with_itsi/2-creating-basic-alerts/_index.md

@@ -12,66 +12,56 @@ This section covers the creation of basic alerts in Splunk Enterprise, AppDynami
 
 Splunk alerts are triggered by search results that match specific criteria. We'll create a real-time alert that notifies us when a certain condition is met.
 
-**Scenario:**  Alert when the number of "error" events in the "application_logs" index exceeds 10 in the last 5 minutes.
+**Scenario:**  Alert when the number of "Invalid user" events in the "main" index exceeds 100 in the last 5 minutes.
 
 **Steps:**
 
 1. **Create a Search:** Start by creating a Splunk search that identifies the events you want to alert on. For example:
 
    ```splunk
-   index=application_logs level=error
+   index=main "Invalid user"
    ```
-    Use the time picker to select "Relative" and set the timespan to 10.
+    Use the time picker to select "Last 15 minutes"".
 
 2. **Configure the Alert:**
    * Click "Save As" and select "Alert."
-   * Give your alert a descriptive name (e.g., "Application Error Alert").
-   * **Trigger Condition:**
-      * **Scheduled:** Choose "Scheduled" to evaluate the search on a set schedule. Below scheduled will be the button to select the frequency, select "Run on Cron Schedule". If the time Range below that is different than 10 minutes, update it.
-      * **Triggered when:** Select "Number of results" "is greater than" "10."
-      * **Time Range:** Set to "5 minutes."
+   * Give your alert a descriptive name (e.g., "Numerous Invalid User Logins Attempted").
+   * **Alert type:**
+      * **Scheduled:** Choose "Scheduled" to evaluate the search on a set schedule. Below scheduled will be the button to select the frequency, select "Run on Cron Schedule". 
+      * **Cron Expression:** */15 * * * *
+      * **Triggered when:** Select "Number of results" "is greater than" "100."
+      * **Time Range:** Set to "15 minutes."
    * **Trigger Actions:**
       * For this basic example, choose "Add to Triggered Alerts."  In a real-world scenario, you'd configure email notifications, Slack integrations, or other actions.
    * **Save:** Save the alert.
 
-**Explanation:** This alert runs the search every 10 minutes and triggers if the search returns more than 10 results. The "Add to Triggered Alerts" action simply adds a Alert to the Splunk Triggered Alerts list.
+![show-entry](../images/save_as_alert.png?classes=inline)
 
-**Time Ranges and Frequency:** Since everything in Splunk core is a search, you need to consider the search timespan and frequency so that you are not a) searching the same data multiple times with an overlap timespan, b) missing events because of a gap between timespan and frequency, c) running too frequently and adding overhead or d) running too infrequently and experiencing delays in alerting.
-
-
-## 2. AppDynamics Alerts (Health Rule Violations)
+**Explanation:** This alert runs the search every 15 minutes and triggers if the search returns more than 100 results. The "Add to Triggered Alerts" action simply adds a Alert to the Splunk Triggered Alerts list.
 
-**2. Create a Health Rule (or modify an existing one):**
-   * Click "Create Rule" (or edit an existing rule that applies to your application).
-   * Give the health rule a descriptive name (e.g., "Order Service Response Time Alert").
-   * **Scope:**  Select the application and tier (e.g., "OrderService").
-   * **Conditions:**
-      * Choose the metric "Average Response Time."
-      * Set the threshold: "is greater than" "500" "milliseconds."
-      * Configure the evaluation frequency (how often AppDynamics checks the metric).
-   * **Actions:**
-      * For this basic example, choose "Log to console."  In a real-world scenario, you would configure email, SMS, or other notification channels.
-   * **Save:** Save the health rule.
-
-**Explanation:** This health rule continuously monitors the average response time of the "OrderService." If the response time exceeds 500ms, the health rule is violated, triggering the alert and the configured actions.
+**Time Ranges and Frequency:** Since everything in Splunk core is a search, you need to consider the search timespan and frequency so that you are not a) searching the same data multiple times with an overlap timespan, b) missing events because of a gap between timespan and frequency, c) running too frequently and adding overhead or d) running too infrequently and experiencing delays in alerting.
 
 
-## 3. Splunk Observability Cloud Alerts (Detectors) (Continued)
+## 2. Splunk Observability Cloud Alerts (Detectors)
 
-**2. Create a Detector:**
-   * Click "Create Detector."
-   * Give the detector a descriptive name (e.g., "High CPU Utilization Alert").
+**Create a Detector:**
+   * Click "Detectors & SLOs" in the lefthand menu
+   * Click "Create Detector -> Custom Detector"
+   * Give the detector a descriptive name (e.g., "High CPU Utilization Alert - INITIALS").
    * **Signal:**
-      * Select the metric you want to monitor (e.g., "host.cpu.utilization").  Use the metric finder to locate the correct metric.
-      * Add any necessary filters to specify the host (e.g., `host:my-hostname`).
+      * Select the metric you want to monitor ("cpu.utilization"). 
+      * Add any necessary filters to specify the host (`service.name:otelshop-loadgenerator`).
+      * Click "Proceed to Alert Condition"
    * **Condition:**
-      * Set the threshold: "is above" "80" "%."
-      * Configure the evaluation frequency and the "for" duration (how long the condition must be true before the alert triggers).
+      * Select Static Threshold
+      * Set the threshold: "is above" "90"
    * **Notifications:**
       * For this example, choose a simple notification method (e.g., a test webhook). In a real-world scenario, you would configure integrations with PagerDuty, Slack, or other notification systems.
    * **Save:** Save the detector.
 
-**Explanation:** This detector monitors the CPU utilization metric for the specified host.  If the CPU utilization exceeds 80% for the configured "for" duration, the detector triggers the alert and sends a notification.
+![show-entry](../images/detector_preview.png?classes=inline)
+
+**Explanation:** This detector monitors the CPU utilization metric for the specified service.  If the CPU utilization exceeds 90% for the configured "for" duration, the detector triggers the alert and sends a notification.
 
 **Important Considerations for All Platforms:**
 

content/en/ninja-workshops/12-alerting_monitoring_with_itsi/3-creating-services-in-itsi/1-creating-o11y-service.md

@@ -0,0 +1,47 @@
+---
+title: Creating an O11y Based Service
+linkTitle: 3.1 Creating an O11y Based Service
+weight: 2
+---
+
+# Starting with an Observability Cloud Based Service
+
+1. **Access Services:** In ITSI click "Configuration", click on "Services".
+
+2. **Create New Service: PaymentService2:** Click "Create New Service".
+
+3. **Service Details (PaymentService2):**
+    * **Title:** "PaymentService2"
+    * **Description (Optional):**  e.g., "Payment Service for Hipster Shop - version 2"
+
+4. **Select Template:** Choose "Link service to a service template" and search for "Splunk APM Business Workflow KPIs" from the template dropdown. Click **Create** to save the new service.
+
+6. **Entity Assignment:**
+    * The page will load and display the new Service and you will be on the Entities page. This demo defaults to selecting the *paymentservice:grpc.hipstershop.PaymentService/Charge* entity. In a real world situation you would need to match the workflow to the entity name manually.
+    * **Direct Entity Selection (If Available):** Search for the entity using `sf_workflow="paymentservice:grpc.hipstershop.PaymentService/Charge"` and select it.
+
+7. **Save Service (PaymentService2):** Click "Save" to create "PaymentService2".
+
+8. **Settings:** Click the "Settings" tab, enable *Backfill* and keep that standard 7 days. Enable the Service, and click "Save"
+
+## Setting PaymentService2's Service Health as a Dependency for Online-Boutique-US
+
+1. **Locate Online-Boutique-US:** Find the "Online-Boutique-US" service in the service list.
+
+2. **Edit Online-Boutique-US:** Click "Edit".
+
+3. **Service Dependencies:** Look for the "Service Dependencies" section. 
+
+4. **Add Dependency:**  There should be an option to add a dependent service.  Search for "PaymentService2".
+
+5. **Select KPI:** Check the box next to ServiceHealthScore for PaymentService2.
+
+6. **Save Changes:** Save the changes to the "Online-Boutique-US" service.
+
+## Verification
+
+* Click on "Service Analyzer" and select the "Default Analyzer"
+* Filter the service to just "Buttercup Business Health"
+* Verify that *PaymentService2* is now present below *Online-Boutique-US* and should be in a grey status.
+
+![show-entry](../images/service_tree_o11y.png?classes=inline)

content/en/ninja-workshops/12-alerting_monitoring_with_itsi/3-creating-services-in-itsi/2-creating-appd-service.md

@@ -0,0 +1,47 @@
+---
+title: Creating an AppD Based Service
+linkTitle: 3.2 Creating an AppD Based Service
+weight: 3
+---
+
+# Starting with an AppDynamics Based Service
+
+1. **Access Services:** In ITSI click "Configuration", click on "Services".
+
+2. **Create Service: AD-Ecommerce2:** Click "Create Service -> Create Service".
+
+3. **Service Details (AD-Ecommerce2):**
+    * **Title:** "AD-Ecommerce2"
+    * **Description (Optional):**  e.g., "Ecommerce Service - version 2"
+
+4. **Select Template:** Choose "Link service to a service template" and search for "AppDynamics App Performance Monitoring" from the template dropdown. Click **Create** to save the new service.
+
+5. **Entity Assignment:**
+    * The page will load and display the new Service and you will be on the Entities page. This demo defaults to selecting the *AD-Ecommerce:18112:demo1.saas.appdynamics.com* entity. In a real world situation you would need to match the entity_name to the entity name manually.
+    * **Direct Entity Selection (If Available):** Search for the entity using `entity_name="AD-Ecommerce:18112:demo1.saas.appdynamics.com"` and select it.
+
+7. **Settings:** Click the "Settings" tab, enable *Backfill* and keep that standard 7 days. Enable the Service, and click "Save"
+
+## Setting AD-Ecommerce2's Service Health as a Dependency for AD.Ecommerce
+
+1. **Navigate back to Services page:** Click "Configuration -> Services"
+
+2. **Locate AD.Ecommerce:** Find the "AD.Ecommerce" service in the service list.
+
+3. **Edit AD.Ecommerce:** Click "Edit".
+
+4. **Service Dependencies:** Look for the "Service Dependencies" section. 
+
+5. **Add Dependency:**  There should be an option to add a dependent service.  Search for "AD-Ecommerce2".
+
+6. **Select KPI:** Check the box next to ServiceHealthScore for AD-Ecommerce2.
+
+7. **Save Changes:** Save the changes to the "AD.Ecommerce" service.
+
+## Verification
+
+* Click on "Service Analyzer" and select the "Default Analyzer"
+* Filter the service to just "Buttercup Business Health"
+* Verify that *AD-Ecommerce2* is now present below *AD.Ecommerce* and should be in a grey status.
+
+![show-entry](../images/service_tree_appd.png?classes=inline)

content/en/ninja-workshops/12-alerting_monitoring_with_itsi/3-creating-services-in-itsi/_index.md

@@ -1,5 +1,5 @@
 ---
-title: Creating Basic Alerts
+title: Creating Services in ITSI
 linkTitle: 3. Creating Services in ITSI
 weight: 1
 ---
@@ -10,5 +10,10 @@ This workshop outlines how to create a service in Splunk IT Service Intelligence
 
 **Scenario:**
 
-We have two existing services: "Astronomy Shop" (representing an application running in Kubernetes and being monitored by Splunk Observability Cloud) and "AD.ECommerce" (representing an application monitored by AppDynamics). We want to create a new service and add it as a dependent of one of those services. It is not necessary to create a service for both during your first run through this workshop so pick one that you are more interested in to start with.
+We have two existing services: "Online-Boutique-US" (representing an application running in Kubernetes and being monitored by Splunk Observability Cloud) and "AD.ECommerce" (representing an application monitored by AppDynamics). We want to create a new service and add it as a dependent of one of those services. It is not necessary to create a service for both during your first run through this workshop so pick one that you are more interested in to start with.
 
+![show-entry](../images/service_tree_start.png?classes=inline)
+
+**Return to your Splunk Environment and under Apps, select IT Service Intelligence**
+
+In the Default Analyzer update the Filter to "Buttercup Business Health"

content/en/ninja-workshops/12-alerting_monitoring_with_itsi/4-creating-alerts-in-itsi/_index.md

@@ -0,0 +1,46 @@
+---
+title: Creating Alerts in ITSI
+linkTitle: 4. Creating Alerts in ITSI
+weight: 1
+---
+
+# Configuring a Basic Alert in Splunk ITSI
+
+This section guides you through configuring a basic alert in Splunk IT Service Intelligence (ITSI).  We'll set up an alert that triggers when our previously created Service breaches a KPI threshold.
+
+**Depending on the Service You Created, the KPI we use for this alert will change. In the instruction steps below replace Service Name and KPI appropriately**
+
+* **PaymentService2:** Business Workflow Error Rate
+* **AD-Ecommerce2:** Availability
+
+**Steps:**
+
+1. **Navigate to the KPI:**
+   * In ITSI, go to "Configuration" -> "Correlation Searches"
+   * Click "Create New Search"
+
+2. **Configure the new search:**
+   * **Search Title:** *Service Name* *KPI* Critical
+   * **Description:** *Service Name* *KPI* Critical
+   * **Search:** 
+   ```
+   index=itsi_summary kpi="*KPI*" alert_severity=critical
+   ```
+    * **Time Range:** Last 15 minutes
+    * **Service:** *Service Name*
+    * **Entity Lookup Field:** itsi_service_id
+    * **Run Every:** minutes
+    * **Notable Event Title:** *Service Name* *KPI* Critical
+    * **Severity:** Critical
+    * **Notable Event Identified Fields:** source
+
+**After Creating the Alert:**
+
+* You will need to wait 5-10 minutes for the alert to run
+* The alert will be listed in the "Alerts and Episodes" Pane in ITSI.
+
+![show-entry](../images/alerts.png?classes=inline)
+
+**Important Considerations:**
+
+* **Alert Fatigue:** Avoid setting up too many alerts or alerts with overly sensitive thresholds.  This can lead to alert fatigue, where people become desensitized to alerts and might miss critical issues.

content/en/ninja-workshops/12-alerting_monitoring_with_itsi/5-episodes-in-itsi/_index.md

@@ -0,0 +1,51 @@
+---
+title: Creating Episodes in ITSI
+linkTitle: 5. Creating Episodes in ITSI
+weight: 1
+---
+
+# Creating an Aggregation Policy in Splunk ITSI 
+
+This section outlines the steps to create an aggregation policy in Splunk ITSI that matches the alerts we just set up. This policy will group related alerts, reducing noise and improving incident management.
+
+**Depending on the Alert You Created, the title we use for this alert will change. In the instruction steps below replace AlertName with the Service Name used**
+
+* **PaymentService2** or
+* **AD-Ecommerce2** 
+
+## Steps
+
+1. **Navigate to Notable Event Aggregation Policies:** In Splunk, go to "Configuration" -> "Notable Event Aggregation Policies".
+
+2. **Create New Policy:** click the green "Create Notable Event Aggregation Policy" button in the upper right corner.
+
+3. **Filtering Criteria:** This is the most important part.  You'll define the criteria for alerts to be grouped by this policy. Click "Add Rule (OR)"
+
+    * **Field:** Select "title" from the dropdown menu.
+    * **Operator:** Choose "matches".
+    * **Value:** Enter the string "*Service Name**". (make sure to include the *)
+
+4. **Splitting Events:** Remove the "hosts" field that is provided by default and update it to use the "service" field. We want this generating new episodes for each Service that is found. In our example, it should only be 1.
+
+5. **Breaking Criteria:** Configure how Episodes are broken or ended. We'll leave it as the default *"If an event occurs for which severity = Normal"*. Click Preview on the right to confirm it is picking up our Alert
+
+6. **Click Next**
+
+7. **Actions (Optional):** Define actions to be taken on aggregated alerts.  For example, you can automatically create a ticket in ServiceNow or send an email notification. We're going to skip this part.
+
+8. **Click Next**
+
+9. **Policy Title and Description:**
+    * **Policy Title:** *Service Name* Alert Grouping
+    * **Description:** Grouping *Service Name* alerts together.
+
+8. **Save Policy:** Click the "Next" button to create the aggregation policy.
+
+## Verification
+
+After saving the policy, navigate to the "Go to Episode Review" page and filter alerts for last 15 minutes and add a filter to status=New and search for our Service Name in the search box.
+
+There may already be an episode named after our specific alert already, if so,  close it out and wait for a new one to be generated with our new Title.
+
+![show-entry](../images/episode.png?classes=inline)
+