Merge pull request #3084 from hpe-dev-incubator/cms/blog/integrating-hpe-greenlake-webhooks-with-splunk

Create Blog “integrating-hpe-greenlake-webhooks-with-splunk”

Author: Didier-Lalli

content/blog/integrating-hpe-greenlake-webhooks-with-splunk.md

@@ -0,0 +1,387 @@
+---
+title: "Integrating HPE GreenLake webhooks with Splunk "
+date: 2025-06-24T11:34:04.977Z
+author: Vandewilly Silva
+authorimage: https://ca.slack-edge.com/E01LD9FH0JZ-U05UA6N7KH6-6b8619b87acd-512
+disable: false
+tags:
+  - hpe_greenlake_cloud
+  - webhooks
+---
+<style>
+li {
+   font-size: 27px;
+   line-height: 33px;
+   max-width: none;
+}
+</style>
+
+## Overview
+
+This guide shows you how to connect HPE GreenLake webhooks with [Splunk](https://www.splunk.com/). Splunk is a data platform that collects, indexes, and analyzes machine-generated data to provide insights for various purposes, including security monitoring, IT operations, and business analytics. When the two are connected, you will be able to see your HPE GreenLake events through Splunk for improved data monitoring and analysis.
+
+## What you’ll learn
+
+* How to set up Splunk to receive data from HPE GreenLake
+* How to handle HPE GreenLake's security requirements
+* How to build a helper app that makes everything work together
+* How to test and monitor your setup
+
+## Overview of Splunk HTTP Event Collector (HEC)
+
+The [HTTP Event Collector (HEC)](https://dev.splunk.com/enterprise/docs/devtools/httpeventcollector/) is a Splunk feature that lets you send data and application events to a Splunk deployment over the HTTP and Secure HTTP (HTTPS) protocols. HEC uses a token-based authentication model. You can generate a token and then configure a logging library or HTTP client with the token to send data to HEC in a specific format.
+
+### Key features of HEC
+
+* Token-based authentication: Each token has a unique value, which is a 128-bit number that is represented as a 32-character globally unique identifier (GUID)
+* Secure communication: Supports both HTTP and HTTPS protocols for data transmission
+* API key support: Provides secure authentication mechanisms that align perfectly with HPE GreenLake's security requirements
+* Flexible data formats: Accepts both JSON-formatted events and raw text data
+
+## Overview of HPE GreenLake webhooks
+
+[HPE GreenLake webhooks](https://developer.greenlake.hpe.com/docs/greenlake/services/event/public/webhooks/) facilitate automated, real-time communication between HPE GreenLake cloud services and an external service of your choosing. For example, a webhook could notify your IT Operation Management platform when a new audit log is created, or when subscriptions are about to expire. A getting started guide to HPE GreenLake webhooks is available [here](https://developer.hpe.com/blog/getting-started-with-the-hpe-greenlake-cloud-eventing-framework/).
+
+### HPE GreenLake webhook security features:
+
+HPE GreenLake implements robust security measures to ensure webhook authenticity:
+
+* Challenge Request Validation: After registering a webhook, a verification challenge is sent to the destination (the webhook URL). The event type is `hpe.greenlake.events.v1beta1.webhooks.verification`. The challenge request includes a unique, random string generated by the server that is sent in the body as a payload.
+* HMAC SHA-256 Signature Verification: HPE GreenLake webhooks use a verification challenge process to ensure that webhook connections are legitimate and secure. HPE GreenLake secures webhook notifications through HMAC (Hash-based Message Authentication Code) with SHA-256, a cryptographic hashing algorithm.
+* Shared Secret Management: The platform supports dual secret rotation for zero-downtime security updates.
+
+Challenge Request Example:
+
+```json
+{
+  "specversion" : "1.0",
+  "type" : "hpe.greenlake.events.v1beta1.webhooks.verification",
+  "source" : "//global.api.greenlake.hpe.com/events",
+  "id" : "C234-1234-1234",
+  "time" : "2018-04-05T17:31:00Z",
+  "datacontenttype" : "application/json",
+  "data" : {
+	"challengeRequest" : "<TOKEN>"
+  }
+}
+```
+
+## Challenges and solutions
+
+The primary challenge in integrating HPE GreenLake webhooks with Splunk HEC lies in the webhook verification process. The destination must read the value from the challengeRequest field and create an HMAC SHA-256 hash, using the webhook secret as salt and the challengeRequest value as a string to hash. When successful, the destination responds with a JSON object with the format `{"verification": "CREATED_HASH"}` and a HTTP 200 OK status.
+
+### A challenge
+
+Splunk's HEC endpoint is designed for data ingestion and doesn't natively support the  challenge-response mechanism required by HPE GreenLake webhooks. HEC expects to receive event data directly and cannot handle the initial verification handshake.
+
+### The solution
+
+This is where [Splunk's custom REST endpoints](https://dev.splunk.com/enterprise/docs/devtools/customrestendpoints/) capability becomes invaluable. A custom REST endpoint is a developer-defined endpoint and associated handler that lets you build out the Splunk REST API to meet your specific needs. We can create a custom endpoint handler that:
+
+1. Intercepts the initial challenge request from HPE GreenLake
+2. Validates the challenge using HMAC SHA-256
+3. Responds appropriately to complete the verification
+4. Forwards validated event data to HEC for ingestion
+
+## Custom REST endpoints in Splunk
+
+Splunk's custom REST endpoints provide powerful extensibility for scenarios exactly like ours. You use a custom endpoint to add a special feature that Splunk doesn't have built-in, like, in our case, handling the unique secret handshake from HPE GreenLake.
+
+Key benefits of our integration:
+
+* Flexible request handling: Can process both challenge requests and event data
+* Custom logic implementation: Handler code implements HPE GreenLake's specific validation requirements
+* Centralized management: Provides a single endpoint for webhook management
+
+## Where to configure the endpoint handler: Splunk Enterprise vs Splunk Cloud
+
+Splunk Enterprise is the self-hosted version that an organization deploys and manages on its own infrastructure, either on-premises (on-prem) or in a private cloud.
+
+Splunk Cloud Platform is the Software as a Service (SaaS) offering, where the Splunk platform is hosted, managed, and maintained by Splunk.
+
+### For Splunk Enterprise
+
+You can install and configure the endpoint handler directly on your Splunk Enterprise instance by placing it in the etc/apps/ directory and following the steps in this guide. Splunk Enterprise supports custom REST endpoints out of the box.
+
+### For Splunk Cloud
+
+[Splunk Cloud has extra security controls](https://docs.splunk.com/Documentation/SplunkCloud/latest/RESTTUT/RESTandCloud), so you might need to take additional steps to allow your helper to communicate with the Splunk REST API.
+
+## Generate an Splunk REST API key
+
+In order to use the custom REST endpoint in Splunk, you need to get an API key which HPE GreenLake will use to authenticate against Splunk when initially setting up the webhook and sending events. You can use the following cURL command to generate a key.
+
+```shell
+curl -k https://your-splunk-instance:8089/services/auth/login \
+     --data-urlencode username=YOUR_USERNAME \
+     --data-urlencode 'password=YOUR_PASSWORD’
+```
+
+The response will be:
+
+```json
+<response>
+  <sessionKey>YOUR_SESSION_KEY</sessionKey>
+  <messages>
+    <msg code=""></msg>
+  </messages>
+</response>
+```
+
+## Sample Python handler
+
+Let's create a custom REST endpoint handler in Python to handle the HPE GreenLake webhook validation and forwards events to Splunk HEC, once validated.
+
+### Directory structure
+
+```markdown
+splunk_hpe_webhook_app/
+├── bin/
+│   └── hpe_webhook_handler.py
+├── default/
+│   ├── restmap.conf
+│   └── web.conf
+└── metadata/
+└── default.meta
+```
+
+### Python handler (bin/hpe_webhook_handler.py)
+
+```python
+import os
+import sys
+import json
+import hmac
+import hashlib
+import urllib.request
+import urllib.parse
+from splunk.rest import BaseRestHandler
+class HPEWebhookHandler(BaseRestHandler):
+	def __init__(self, command_line, command_arg):
+    	super(HPEWebhookHandler, self).__init__(command_line, command_arg)
+    	# Configure your HEC endpoint and token
+    	self.hec_url = "<https://your-splunk-instance:8088/services/collector/event>"
+    	self.hec_token = "YOUR_HEC_TOKEN_HERE"
+    	self.webhook_secret = "YOUR_WEBHOOK_SECRET_HERE"
+	def handle_POST(self):
+    	"""Handle POST requests from HPE GreenLake webhooks"""
+    	try:
+        	# Parse the incoming request body
+        	request_body = json.loads(self.request['payload'])
+        	event_type = request_body.get('type', '')
+        	# Check if this is a verification challenge
+        	if event_type == 'hpe.greenlake.events.v1beta1.webhooks.verification':
+            	return self._handle_challenge(request_body)
+        	else:
+            	# Validate signature for regular events
+            	if self._validate_signature():
+                	return self._forward_to_hec(request_body)
+            	else:
+                	return self._return_error("Invalid signature", 401)
+    	except Exception as e:
+        	return self._return_error(f"Error processing request: {str(e)}", 500)
+	def _handle_challenge(self, request_body):
+    	"""Handle HPE GreenLake webhook verification challenge"""
+    	try:
+        	# Extract challenge request token
+        	challenge_request = request_body['data']['challengeRequest']
+        	# Create HMAC SHA-256 hash
+        	hmac_hash = hmac.new(
+            	key=self.webhook_secret.encode('utf-8'),
+            	msg=challenge_request.encode('utf-8'),
+            	digestmod=hashlib.sha256
+        	)
+        	# Create verification response
+        	verification_response = {
+            	"verification": hmac_hash.hexdigest()
+        	}
+        	# Return successful verification
+        	self.response.setHeader('content-type', 'application/json')
+        	self.response.write(json.dumps(verification_response))
+        	return
+    	except Exception as e:
+        	return self._return_error(f"Challenge validation failed: {str(e)}", 400)
+	def _validate_signature(self):
+    	"""Validate HMAC signature for regular webhook events"""
+    	try:
+        	# Get signature from headers
+        	signature_header = self.request.get('headers', {}).get('hpe-webhook-signature', '')
+        	if not signature_header.startswith('sha256='):
+            	return False
+        	expected_signature = signature_header[7:]  # Remove 'sha256=' prefix
+        	# Calculate expected signature
+        	payload = self.request['payload']
+        	calculated_signature = hmac.new(
+            	key=self.webhook_secret.encode('utf-8'),
+            	msg=payload.encode('utf-8'),
+            	digestmod=hashlib.sha256
+        	).hexdigest()
+        	# Compare signatures
+        	return hmac.compare_digest(expected_signature, calculated_signature)
+    	except Exception:
+        	return False
+	def _forward_to_hec(self, event_data):
+    	"""Forward validated event data to Splunk HEC"""
+    	try:
+        	# Prepare HEC request
+        	hec_data = {
+            	"event": event_data,
+            	"sourcetype": "hpe:greenlake:webhook",
+            	"source": "hpe_greenlake",
+            	"index": "main"  # Configure as needed
+        	}
+        	# Create HTTP request to HEC
+        	req = urllib.request.Request(
+            	url=self.hec_url,
+            	data=json.dumps(hec_data).encode('utf-8'),
+            	headers={
+                	'Authorization': f'Splunk {self.hec_token}',
+                	'Content-Type': 'application/json'
+            	}
+        	)
+        	# Send to HEC
+        	with urllib.request.urlopen(req) as response:
+            	if response.status == 200:
+                	self.response.setHeader('content-type', 'application/json')
+                	self.response.write(json.dumps({"status": "success"}))
+                	return
+            	else:
+                	return self._return_error("Failed to forward to HEC", 500)
+    	except Exception as e:
+        	return self._return_error(f"HEC forwarding failed: {str(e)}", 500)
+	def _return_error(self, message, status_code):
+    	"""Return error response"""
+    	self.response.setStatus(status_code)
+    	self.response.setHeader('content-type', 'application/json')
+    	self.response.write(json.dumps({"error": message}))
+```
+
+### Configuration files
+
+#### default/restmap.conf
+
+```markdown
+[script:hpe_webhook_handler]
+match = /hpe/webhook
+script = hpe_webhook_handler.py
+scripttype = persist
+handler = hpe_webhook_handler.HPEWebhookHandler
+requireAuthentication = false
+output_modes = json
+passPayload = true
+passHttpHeaders = true
+passHttpCookies = false
+```
+
+#### default/web.conf
+
+```markdown
+[expose:hpe_webhook_handler]
+pattern = hpe/webhook
+methods = POST`
+```
+
+#### metadata/default.meta
+
+```markdown
+[restmap/hpe_webhook_handler]
+export = system
+[views]
+export = system`
+```
+
+## Configuring Splunk HTTP Event Collector (HEC)
+
+You need to create an API token to use HEC via its API. You can do this from:
+
+1.    **Settings > Data Inputs > HTTP Event Collector**
+
+2.    Select **New token.** Use this token to update the Python handler script line
+
+![Data inputs settings ](/img/dccf7431-b83b-4799-a795-25bccc7637db.png "Data inputs settings ")
+
+Your final configuration should look like this:
+
+![HEC final configuration ](/img/03af2f59-ab50-443b-ae93-90d274738317.png "HEC final configuration ")
+
+Verify your global settings so that they match the following:
+
+![Global settings](/img/310bb0be-5363-428a-99a2-5f64bef2ca8a.png "Global settings")
+
+This allows you to get your HEC endpoint, which is used in the Python handler to create an incident based on the HPE GreenLake event received via the webhook. The URL of the endpoint should look like this:
+
+`<https://<splunk-host>>:8088/services/collector/event`
+
+## Final integration flow
+
+The complete integration flow works as follows:
+
+1. Initial setup
+
+   * Deploy the custom Splunk endpoint handler using the above HPE webhook handler Python script.
+   * Make sure to set the HEC endpoint in the Python script (line 13)
+   * Make sure to set the Splunk HEC API token in the Python script (line 14)
+   * Make sure to set the HPE GreenLake webhook secret in the Python script (line 15)
+   * Register the webhook handler URL with HPE GreenLake:
+
+      * URL of handler: [`https://your-splunk-instance:8089/servicesNS/-/your_app/hpe/webhook`](https://your-splunk-instance:8089/servicesNS/-/your_app/hpe/webhook)
+      * Set the same secret key as we setup in the Python handler (line 15)
+      * Use API as Authentication type and set the API key to the Splunk REST API Key generated in the section above. 
+
+> Note: See [this blog](https://developer.hpe.com/blog/getting-started-with-the-hpe-greenlake-cloud-eventing-framework/) to learn how to register a new webhook handler in HPE GreenLake
+
+2. Webhook handler verification process
+
+   * HPE GreenLake sends a verification challenge to your custom endpoint.
+   * The custom REST handler receives the challenge and validates it using HMAC SHA-256.
+   * The handler responds with the computed verification hash.
+   * HPE GreenLake confirms the webhook and marks it as active.
+
+3. Event processing flow
+
+   * HPE GreenLake sends event data to the custom handler endpoint.
+   * The custom REST handler validates the HMAC signature. 
+   * The handler forwards validated events to HEC.
+   * Splunk HEC ingests the data for analysis and visualization.
+
+4. Data flow diagram
+ 
+![Data flow diagram](/img/diagram.jpg "Data flow diagram")
+
+## Benefits of this architecture
+
+Security: The custom endpoint handler ensures only validated, authentic events reach your Splunk environment.
+
+Reliability: If there are more than 20 failures in a 12-hour period, the webhook enters the critical state in HPE GreenLake. If there are no new failures in 12 hours, the webhook returns to the active state. The custom handler can implement robust error handling to maintain webhook health.
+
+Scalability: The solution can handle multiple webhook types and route them to different HEC endpoints or indexes as needed.
+
+Monitoring: All webhook interactions are logged within Splunk for troubleshooting and monitoring.
+
+## Testing and deployment
+
+Testing the integration
+
+1. Verify custom endpoint: Test your custom REST endpoint using curl:
+
+```shell
+`curl -X POST`[`https://your-splunk-instance:8089/servicesNS/-/your_app/hpe/webhook`](https://your-splunk-instance:8089/servicesNS/-/your_app/hpe/webhook)`\`
+`-H "Content-Type: application/json" \`
+`-d '{"type": "test.event", "data": {"message": "Hello Splunk"}}'`
+```
+
+2. Webhook registration: Register your webhook with HPE GreenLake using the custom endpoint URL.
+3. Challenge validation: Monitor Splunk logs to ensure the challenge request is handled correctly.
+4. Event flow testing: Trigger test events from HPE GreenLake and verify they appear in your Splunk index.
+
+## Conclusion
+
+Integrating HPE GreenLake webhooks with Splunk via HTTP Event Collector presents unique challenges due to the webhook verification requirements, but Splunk's custom REST endpoints capabilities provide an elegant solution. Such integration offers several key benefits:
+
+Enhanced security: The custom REST endpoint handler ensures that only validated, authentic events from HPE GreenLake reach your Splunk environment, maintaining the security standards required by both platforms.
+
+Seamless event flow: Once configured, events flow automatically from HPE GreenLake to Splunk, enabling real-time monitoring and analysis of your cloud infrastructure.
+
+Extensible architecture: The custom REST handler can be extended to support multiple webhook types, different routing logic, and additional validation mechanisms as your requirements evolve.
+
+Whether you're monitoring subscription changes, or audit events, this integration ensures that your HPE GreenLake data becomes a valuable part of your Splunk analytics ecosystem, empowering your organization with comprehensive, real-time insights into your cloud infrastructure.