Skip to content

Commit 7ba8c9e

Browse files
KorayErkanKorayErkan
authored
Merge pull request #382 from VirtualMetric:DT-486-write-content-hub-integration
DT-486-Add Content Hub Integration documentation
2 parents 769c674 + 3babc2d commit 7ba8c9e

File tree

6 files changed

+273
-0
lines changed

6 files changed

+273
-0
lines changed

‎docs/configuration/pipelines/overview.mdx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,8 @@ Divide and conquer your data processing by composing targeted pipelines:
207207
* **Conditional Processing**: Use conditional statements to selectively apply processors
208208
* **Error Handling**: Implement robust error handling with `on_failure` and `ignore_failure`
209209

210+
For pre-built pipeline templates with dependency management, see <Topic id="content-hub-integration-dependencies">Content Hub dependency integration</Topic>.
211+
210212
### Conditional Execution
211213

212214
Pipelines and processors support conditional execution using the `if` parameter:

‎docs/configuration/routes/content-routing.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -165,5 +165,6 @@ Content routes reference their associated content pack pipelines. The route proc
165165
## Related Documentation
166166

167167
- <Topic id="content-hub-overview">Content Hub Overview</Topic> — Template library and installation
168+
- <Topic id="content-hub-integration">Content Hub Integration</Topic> — Routes, dependencies, pipelines, and API
168169
- <Topic id="routes-overview">Routes Overview</Topic> — YAML configuration reference
169170
- <Topic id="routes-config">Route Configuration</Topic> — Configuration field reference

‎docs/configuration/routes/overview.mdx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -309,3 +309,5 @@ The following aspects of your telemetry stream must be considered in the impleme
309309
- ${TARGET_SYSTEM}
310310
```
311311

312+
For pre-built route configurations bundled with pipeline templates, see <Topic id="content-hub-integration-routes">Content Hub route integration</Topic>.
313+

‎docs/content-hub/integration.mdx‎

Lines changed: 261 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,261 @@
1+
---
2+
sidebar_label: Integration
3+
---
4+
5+
# Content Hub: Integration
6+
7+
Content Hub templates integrate with multiple **DataStream** platform components. Templates can include route configurations for automatic pipeline-to-target connections, define dependencies on other templates, transform into fully editable pipelines after installation, and expose REST API endpoints for programmatic management.
8+
9+
## Route Integration
10+
11+
Templates can include pre-configured route definitions that automatically connect pipelines to devices and targets upon installation.
12+
13+
### Route Configuration Structure
14+
15+
A template's route configuration specifies how the installed pipeline connects to data sources and destinations:
16+
17+
- **Route name and description**: Identifier and purpose of the route
18+
- **Pipeline references**: Which pipelines the route uses for processing
19+
- **Filter expressions**: Conditional routing based on data content
20+
- **Scheduling**: Cron or interval-based execution (only available from <Topic id="routes-advanced-scheduling">Advanced Routes</Topic>)
21+
- **Defined targets**: Abstract target types the route expects (e.g., "SIEM", "Storage")
22+
23+
### Target Mapping
24+
25+
When installing a template that includes route configuration, the installation modal presents target mapping options:
26+
27+
- **Target selection toggles**: Enable or disable each target defined in the route
28+
- **Target dropdown**: Map abstract targets to actual configured datasources in your environment
29+
- **Configuration inheritance**: Target mapping determines which destinations receive processed data
30+
31+
The target mapping step connects template-defined abstract targets (like "primary_siem" or "archive_storage") to your organization's actual target configurations.
32+
33+
### Installing Routes
34+
35+
Routes from templates can be installed through two workflows:
36+
37+
**During initial template installation (combined flow)**:
38+
1. Click <gui>Install Template</gui> on the template detail page
39+
2. Complete dependency selection if applicable
40+
3. Configure target mappings in the route configuration step
41+
4. Confirm installation to create both pipeline and route
42+
43+
**After pipeline installation (separate route installation)**:
44+
1. Navigate to the installed template in Content Hub
45+
2. Select <gui>Install Route</gui> from the <gui>Actions</gui> menu
46+
3. Configure target mappings
47+
4. Confirm to create the route
48+
49+
After installation, the route appears in the <gui>Advanced Routes</gui> section of Routes management and can be edited like any other route.
50+
51+
## Dependency Integration
52+
53+
Templates can reference other templates as dependencies, enabling modular pipeline composition.
54+
55+
### Dependency Types
56+
57+
**Required dependencies** are essential for template functionality:
58+
- Must be installed for the template to function correctly
59+
- Include core processing libraries and shared modules
60+
- System enforces installation before the dependent template
61+
62+
**Optional dependencies** provide enhanced features:
63+
- Can be skipped during installation
64+
- Include advanced transformations and integrations
65+
- Can be added later through <gui>Manage Dependencies</gui>
66+
67+
### Dependency Detection
68+
69+
The system analyzes template content to detect dependencies:
70+
71+
- **Pipeline references**: `{{ IngestPipeline "pipeline-name" }}` calls in template YAML
72+
- **Required vs optional**: The `ignore_missing_pipeline` flag determines classification
73+
- `ignore_missing_pipeline: false` (or absent): Required dependency
74+
- `ignore_missing_pipeline: true`: Optional dependency
75+
76+
### Managing Dependencies
77+
78+
**Pre-installation dependency check**:
79+
When clicking <gui>Install Template</gui>, the system automatically checks for dependencies and displays them in the installation modal.
80+
81+
**Selecting optional dependencies during installation**:
82+
- Required dependencies are pre-selected and cannot be deselected
83+
- Optional dependencies appear as checkboxes that can be enabled or skipped
84+
- Already-installed dependencies show an <gui>Installed</gui> badge and disabled checkbox
85+
86+
**Adding optional dependencies after installation**:
87+
1. Navigate to the installed template in Content Hub
88+
2. Select <gui>Manage Dependencies</gui> from the <gui>Actions</gui> menu
89+
3. Enable additional optional dependencies
90+
4. Confirm to install selected dependencies
91+
92+
**Dependency status indicators**:
93+
- <gui>Installed</gui> badge on dependencies already present in your pipeline library
94+
- Clickable dependency names open dependency detail pages
95+
96+
## Pipeline Integration
97+
98+
Installed templates become fully editable pipelines within **DataStream**.
99+
100+
### Template-to-Pipeline Transformation
101+
102+
When a template is installed:
103+
104+
1. **Content copy**: Template YAML content is copied to the pipeline table
105+
2. **Hash preservation**: A content hash tracks the original template version for update detection
106+
3. **Child pipeline handling**: Child pipelines maintain parent relationships
107+
4. **Name collision handling**: If a pipeline with the same name exists, an automatic suffix is appended
108+
109+
The installed pipeline is independent from the template source. Modifications to the installed pipeline do not affect the original template.
110+
111+
### Post-Installation Workflow
112+
113+
After installation:
114+
115+
1. **Navigate to pipeline**: Click <gui>See Installed Pipeline</gui> from the template <gui>Actions</gui> menu
116+
2. **Full editing access**: Modify processing logic, field mappings, and transformation rules
117+
3. **Customization preservation**: Template updates preserve your modifications through the merge workflow
118+
4. **Child pipeline management**: Create, modify, or delete child pipelines as needed
119+
120+
Installed pipelines appear in the Pipelines management interface with full editing capabilities.
121+
122+
## API Integration
123+
124+
REST API endpoints enable programmatic template management for automation and integration scenarios.
125+
126+
:::tip
127+
Templates installed via API remain accessible through the Content Hub web interface for subsequent management, configuration changes, and deployment operations.
128+
:::
129+
130+
### Endpoint Reference
131+
132+
| Method | Endpoint | Description |
133+
|-------:|:---------|-------------|
134+
| GET | `/api/templates` | List templates (paginated, filterable) |
135+
| GET | `/api/templates/{id}` | Get template details |
136+
| GET | `/api/templates/dependencies/{id}` | Check template dependencies |
137+
| POST | `/api/templates/install/{id}` | Install template as pipeline |
138+
| PUT | `/api/templates/install/{id}` | Add optional dependencies to installed template |
139+
| POST | `/api/templates/install-route/{id}` | Install route configuration |
140+
| GET | `/api/templates/deviceTypes` | Get device type filter options |
141+
| GET | `/api/templates/deviceVendors` | Get vendor filter options |
142+
| GET | `/api/templates/deviceFamilies` | Get family filter options |
143+
144+
### Filtering Parameters
145+
146+
When listing templates (`GET /api/templates`):
147+
148+
| Parameter | Type | Description |
149+
|----------:|:-----|-------------|
150+
| `searchQuery` | string | Case-insensitive name search |
151+
| `deviceType` | string | Filter by device category |
152+
| `deviceVendor` | string[] | Filter by manufacturer (supports multiple) |
153+
| `deviceFamily` | string[] | Filter by device family (supports multiple) |
154+
| `pageNumber` | integer | Page number for pagination |
155+
| `itemCount` | integer | Items per page |
156+
157+
### Authentication and Permissions
158+
159+
All template endpoints require valid authentication. Specific operations require corresponding permissions:
160+
161+
| Operation | Required Permission |
162+
|----------:|:--------------------|
163+
| Browse templates | `ContentHubRead` |
164+
| Install template | `PipelineCreate` |
165+
| Install route | `AdvancedRouteCreate` |
166+
| Manage dependencies | `PipelineCreate` |
167+
168+
### Response Codes
169+
170+
| Code | Description |
171+
|-----:|:------------|
172+
| 200 | Success |
173+
| 400 | Invalid request or template already installed |
174+
| 404 | Template not found |
175+
| 500 | Server error |
176+
177+
## Examples
178+
179+
The following examples demonstrate programmatic Content Hub integration using the REST API.
180+
181+
### List Templates with Filters
182+
183+
Retrieve templates filtered by device type and vendor with pagination. Replace `<token>` with a valid authentication token.
184+
185+
<Tabs>
186+
<TabItem value="powershell" label="PowerShell" default>
187+
```powershell
188+
Invoke-RestMethod -Uri "https://api.example.com/api/templates?deviceType=firewall&deviceVendor=Cisco&pageNumber=1&itemCount=25" `
189+
-Headers @{ Authorization = "Bearer <token>" }
190+
```
191+
</TabItem>
192+
<TabItem value="bash" label="Bash">
193+
```bash
194+
curl -X GET "https://api.example.com/api/templates?deviceType=firewall&deviceVendor=Cisco&pageNumber=1&itemCount=25" \
195+
-H "Authorization: Bearer <token>"
196+
```
197+
</TabItem>
198+
</Tabs>
199+
200+
### Install Template
201+
202+
Install a template as a pipeline, including all required and optional dependencies. The `installDependencies` flag controls whether dependencies are automatically installed.
203+
204+
<Tabs>
205+
<TabItem value="powershell" label="PowerShell" default>
206+
```powershell
207+
Invoke-RestMethod -Uri "https://api.example.com/api/templates/install/12345" `
208+
-Method Post `
209+
-Headers @{ Authorization = "Bearer <token>"; "Content-Type" = "application/json" } `
210+
-Body '{"installDependencies": true}'
211+
```
212+
</TabItem>
213+
<TabItem value="bash" label="Bash">
214+
```bash
215+
curl -X POST "https://api.example.com/api/templates/install/12345" \
216+
-H "Authorization: Bearer <token>" \
217+
-H "Content-Type: application/json" \
218+
-d '{"installDependencies": true}'
219+
```
220+
</TabItem>
221+
</Tabs>
222+
223+
### Install Route Configuration
224+
225+
Install a route configuration for an already-installed template, mapping abstract target types to actual target instances in your environment.
226+
227+
<Tabs>
228+
<TabItem value="powershell" label="PowerShell" default>
229+
```powershell
230+
$body = @{
231+
targetMappings = @(
232+
@{ abstractTarget = "primary_siem"; targetId = "67890" }
233+
)
234+
} | ConvertTo-Json
235+
236+
Invoke-RestMethod -Uri "https://api.example.com/api/templates/install-route/12345" `
237+
-Method Post `
238+
-Headers @{ Authorization = "Bearer <token>"; "Content-Type" = "application/json" } `
239+
-Body $body
240+
```
241+
</TabItem>
242+
<TabItem value="bash" label="Bash">
243+
```bash
244+
curl -X POST "https://api.example.com/api/templates/install-route/12345" \
245+
-H "Authorization: Bearer <token>" \
246+
-H "Content-Type: application/json" \
247+
-d '{
248+
"targetMappings": [
249+
{"abstractTarget": "primary_siem", "targetId": "67890"}
250+
]
251+
}'
252+
```
253+
</TabItem>
254+
</Tabs>
255+
256+
## Related Documentation
257+
258+
- <Topic id="content-hub-overview">Content Hub Overview</Topic> — Template library, browsing, and installation
259+
- <Topic id="pipelines-overview">Pipelines Overview</Topic> — Pipeline configuration and processor chains
260+
- <Topic id="routes-overview">Routes Overview</Topic> — Route configuration reference
261+
- <Topic id="routes-content-routing">Content Routing</Topic> — Installing routes from Content Hub templates

‎sidebars.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -440,6 +440,7 @@ const sidebars: SidebarsConfig = {
440440
label: "Content Hub",
441441
items: [
442442
"content-hub/overview",
443+
"content-hub/integration",
443444
"content-hub/licensing",
444445
]
445446
},

‎topics.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,11 @@
4343
"targets-google-chronicle": "/configuration/targets/gcp/google-chronicle",
4444

4545
"content-hub-overview": "/content-hub/overview",
46+
"content-hub-integration": "/content-hub/integration",
47+
"content-hub-integration-routes": "/content-hub/integration#route-integration",
48+
"content-hub-integration-dependencies": "/content-hub/integration#dependency-integration",
49+
"content-hub-integration-pipelines": "/content-hub/integration#pipeline-integration",
50+
"content-hub-integration-api": "/content-hub/integration#api-integration",
4651

4752
"pipelines-overview": "/configuration/pipelines/overview",
4853
"pipelines-overview-config": "/configuration/pipelines/overview#configuration",
@@ -88,6 +93,7 @@
8893
"routes-config": "/configuration/routes/overview#configuration",
8994
"routes-implementation-strategies": "/configuration/routes/overview#implementation-strategies",
9095
"routes-content-routing": "/configuration/routes/content-routing",
96+
"routes-advanced-scheduling": "/configuration/routes/management#execution-settings",
9197

9298
"scheduling-cron-based": "/configuration/scheduling/cron",
9399
"scheduling-interval-based": "/configuration/scheduling/interval",

0 commit comments

Comments
 (0)