Skip to content

Commit 160d027

Browse files
KorayErkanKorayErkan
authored
Merge pull request #391 from VirtualMetric:DT-576-audit-getting-started-section
DT-576-Audit and fix Getting Started section (9 files, 42 errors)
2 parents 31efeb0 + fd3b15a commit 160d027

15 files changed

+570
-1194
lines changed

docs/getting-started/add-first-device.mdx renamed to docs/getting-started/add-your-first-device.mdx

Lines changed: 28 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,8 @@
1+
---
2+
title: Add Your First Device
3+
description: Create a Syslog device to start collecting data in your DataStream pipeline.
4+
---
5+
16
# Add Your First Device
27

38
## What is a Device?
@@ -9,8 +14,6 @@ A Device represents your data source in DataStream. It defines how DataStream re
914
- **TCP/UDP streams** - Custom protocols and data feeds
1015
- **Cisco eStreamer** - Real-time security intelligence from Cisco devices
1116

12-
Devices are categorized as either **Push** (they listen for incoming data) or **Pull** (they actively collect data using agents).
13-
1417
## Choosing Your Device Type
1518

1619
For this getting started guide, we'll create a **Syslog Device** because:
@@ -33,25 +36,25 @@ For this getting started guide, we'll create a **Syslog Device** because:
3336
- **Name**: "My First Syslog Device"
3437
- **Description**: "Learning syslog collection"
3538
- **Tags**: Leave blank for now
36-
- **Device Status**: Ensure the toggle is **enabled** (blue)
39+
- **Device Status**: Ensure the toggle is set to **Enabled**
3740
- **Director(s) for the Device**: Select the Director you created earlier
38-
- **Preprocessing pipeline**: Leave blank for now
41+
- **Pre-processing pipeline (Optional)**: Leave blank for now
3942
- Click <gui>Next step</gui>
4043

4144
4. **Configure Protocol Settings** (Second Tab)
4245
- **Authentication Protocol**: UDP (default - most common)
4346
- **IP Address**: 0.0.0.0 (listens on all network interfaces)
44-
- **Port**: 514 (standard syslog port)
47+
- **Port**: 514 (standard syslog port - you can assign any port you want)
4548
- Click <gui>Next step</gui>
4649

47-
:::info Important
48-
Make sure port 514 is open for inbound traffic on your network and firewall. This allows syslog sources to send data to your DataStream device.
50+
:::note
51+
The device listener runs on your **Director host**. The IP address `0.0.0.0` means it accepts connections on all network interfaces of that server. Configure your syslog sources to send data to your Director's IP address on port **514**.
4952
:::
5053

5154
5. **Advanced Configuration** (Third Tab)
5255
- You can accept all the default values for now:
5356
- **Socket address reuse**: Enabled
54-
- **Max Connections**: 1000
57+
- **Max Connections**: 10000
5558
- **Timeout**: 300 seconds
5659
- **Max Message Size**: 20 MB
5760
- **Buffer Size**: 9000 bytes
@@ -69,15 +72,22 @@ Your device should now appear in the Syslog devices table with:
6972

7073
Want to verify your device is working? You can send a test syslog message:
7174

72-
**Linux/macOS:**
73-
```bash
74-
logger -n <your-server-ip> -P 514 "Test message from DataStream setup"
75-
```
76-
77-
**Windows PowerShell (requires syslog client):**
78-
```powershell
79-
# Use your preferred syslog testing tool or install one like "Posh-Syslog"
80-
```
75+
<Tabs>
76+
<TabItem value="powershell" label="PowerShell" default>
77+
```powershell
78+
Send-SyslogMessage -Server <your-server-ip> -Message "Test message from DataStream setup" -Severity Informational -Facility User
79+
```
80+
81+
:::note
82+
Requires a syslog module such as [Posh-Syslog](https://www.powershellgallery.com/packages/Posh-SYSLOG). Install with `Install-Module Posh-SYSLOG`.
83+
:::
84+
</TabItem>
85+
<TabItem value="bash" label="Bash">
86+
```bash
87+
logger -n <your-server-ip> -P 514 "Test message from DataStream setup"
88+
```
89+
</TabItem>
90+
</Tabs>
8191

8292
You won't see the processed data yet because we haven't set up a Target or Route, but your Director logs should show the message was received.
8393

@@ -106,4 +116,4 @@ You won't see the processed data yet because we haven't set up a Target or Route
106116

107117
Your device is now listening for data. Next, we need to configure where that data should go by creating a Target.
108118

109-
**Next:** [Configure Your First Target](configure-first-target) to define where your processed data will be stored.
119+
**Next:** <Topic id="getting-started-configure-your-first-target">Configure Your First Target</Topic> to define where your processed data will be stored.

‎docs/getting-started/configure-first-target.mdx‎

Lines changed: 0 additions & 108 deletions
This file was deleted.

‎docs/getting-started/configure-your-first-target.mdx‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
---
2+
title: Configure Your First Target
3+
description: Create a File target to store processed data from your DataStream pipeline.
4+
---
5+
6+
# Configure Your First Target
7+
8+
## What is a Target?
9+
10+
A Target defines where your processed data goes after **DataStream** transforms it. Common target types include:
11+
12+
- **File** - Saves data to JSON, Avro, Parquet, or MultiJSON file formats
13+
- **Microsoft Sentinel** - Sends security data to Microsoft's cloud SIEM platform
14+
- **Azure Blob Storage** - Archives data for long-term retention and compliance
15+
- **Azure Data Explorer** - Enables real-time analytics and dashboard creation
16+
17+
For this guide, we create a **File Target** because it is simple to verify and understand.
18+
19+
## Create Your File Target
20+
21+
1. **Navigate to Targets**
22+
- From the sidebar: **Fleet Management** > **Targets**
23+
- Click the <gui>File</gui> card
24+
25+
2. **Start Target Creation**
26+
- Click <gui>Add new File target</gui>
27+
- The target creation form opens with three tabs
28+
29+
3. **Configure General Settings** (First Tab)
30+
- **Name**: "My First File Target"
31+
- **Description (Optional)**: "Learning file output for getting started"
32+
- **Target Status**: Ensure toggle is set to **Enabled**
33+
- **Post-processing pipeline (Optional)**: Leave blank for now
34+
- Click <gui>Next Step</gui>
35+
36+
4. **Configure File Properties** (Second Tab)
37+
- **Location**: Enter a directory path where you want files saved:
38+
- **Linux/macOS**: `/opt/datastream/output` or `/home/username/datastream`
39+
- **Windows**: `C:\DataStream\Output` or `D:\Logs\DataStream`
40+
41+
:::note
42+
This path refers to a directory on your **Director host**, not your local machine. Verify the directory exists on the server where your Director is installed.
43+
:::
44+
- **File name**: `logs-{{.Year}}_{{.Month}}_{{.Day}}.json`
45+
- **Type**: JSON (default)
46+
- **Field Format (Optional)**: Leave as default
47+
- Leave other fields at their defaults
48+
- Click <gui>Next Step</gui>
49+
50+
:::caution
51+
Ensure the directory you specify in **Location** exists and **DataStream** has write permissions. Create the directory beforehand if it doesn't exist.
52+
:::
53+
54+
5. **Skip Execution Settings** (Third Tab)
55+
- Leave all defaults (scheduling and debugging settings)
56+
- Click <gui>Add target</gui>
57+
58+
## Understanding File Naming
59+
60+
The file name template `logs-{{.Year}}_{{.Month}}_{{.Day}}.json` uses dynamic fields:
61+
62+
- `{{.Year}}` - Current year (e.g., 2026)
63+
- `{{.Month}}` - Current month (e.g., 01, 02, 12)
64+
- `{{.Day}}` - Current day (e.g., 01, 15, 31)
65+
66+
This pattern automatically rotates files daily, making them easier to manage and archive.
67+
68+
## Verify Your Target
69+
70+
Your target should now appear in the File targets table with:
71+
72+
- **Status**: Enabled
73+
- **Location**: Your specified directory path
74+
- **Type**: JSON
75+
76+
## File Format Options
77+
78+
The File target supports four output formats:
79+
80+
- **JSON** (default) - One record per file, widely supported
81+
- **MultiJSON** - Multiple records per file, efficient for high-volume data
82+
- **Avro** - Binary format with embedded schema, supports compression
83+
- **Parquet** - Columnar format, optimized for analytics and compression
84+
85+
:::note
86+
Compression options (zstd, snappy, gzip) are available only for Avro and Parquet formats.
87+
:::
88+
89+
For detailed configuration of all target types, see <Topic id="targets-overview">Target Configuration Guide</Topic>.
90+
91+
## What's Next?
92+
93+
Your target is ready to receive processed data. Now we need to install processing logic that will transform your raw data into a useful format.
94+
95+
**Next:** <Topic id="getting-started-install-content-from-content-hub">Install Content from Content Hub</Topic> to add pre-built data processing pipelines.

0 commit comments

Comments
 (0)