Skip to content

Commit 8f6cd48

Browse files
committed
Update automating_json_log_loading_with_vector.md
1 parent dff3811 commit 8f6cd48

File tree

1 file changed

+89
-3
lines changed

1 file changed

+89
-3
lines changed

docs/en/tutorials/load/automating_json_log_loading_with_vector.md

Lines changed: 89 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,92 @@ In this tutorial, we'll simulate generating logs locally, collect them using [Ve
88

99
Before you start, ensure you have the following prerequisites in place:
1010

11-
- **Amazon S3 Bucket**: An S3 bucket where logs collected by Vector will be stored. [Learn how to create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). In this tutorial, we use `s3://databend-doc/` as the location for staging the collected logs.
12-
- **AWS Credentials**: AWS Access Key ID and Secret Access Key with sufficient permissions for accessing the S3 bucket. [Manage your AWS credentials](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys).
13-
- **Docker**: Ensure that [Docker](https://www.docker.com/) is installed on your local machine, as it will be used to set up Vector.
11+
- **Amazon S3 Bucket**: An S3 bucket where logs collected by Vector will be stored. [Learn how to create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html).
12+
- **AWS Credentials**: AWS Access Key ID and Secret Access Key with sufficient permissions for accessing your S3 bucket. [Manage your AWS credentials](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys).
13+
- **AWS CLI**: Ensure that the [AWS CLI](https://aws.amazon.com/cli/) is installed and configured with the necessary permissions to access your S3 bucket.
14+
- **Docker**: Ensure that [Docker](https://www.docker.com/) is installed on your local machine, as it will be used to set up Vector.
15+
16+
## Step 1: Create Target Folder in S3 Bucket
17+
18+
To store the logs collected by Vector, create a folder named logs in your S3 bucket. In this tutorial, we use `s3://databend-doc/logs/` as the target location.
19+
20+
This command creates an empty folder named `logs` in the `databend-doc` bucket:
21+
22+
```bash
23+
aws s3api put-object --bucket databend-doc --key logs/
24+
```
25+
26+
## Step 2: Create Local Log File
27+
28+
Simulate log generation by creating a local log file. In this tutorial, we use `/Users/eric/Documents/logs/app.log` as the file path.
29+
30+
Add the following JSON lines to the file to represent sample log events:
31+
32+
```json title='app.log'
33+
{"user_id": 1, "event": "login", "timestamp": "2024-12-08T10:00:00Z"}
34+
{"user_id": 2, "event": "purchase", "timestamp": "2024-12-08T10:05:00Z"}
35+
```
36+
37+
## Step 3: Configure & Run Vector
38+
39+
1. Create a Vector configuration file named `vector.yaml` on your local machine. In this tutorial, we create it at `/Users/eric/Documents/vector.yaml` with the following content:
40+
41+
```yaml title='vector.yaml'
42+
sources:
43+
logs:
44+
type: file
45+
include:
46+
- "/logs/app.log"
47+
read_from: beginning
48+
49+
transforms:
50+
extract_message:
51+
type: remap
52+
inputs:
53+
- "logs"
54+
source: |
55+
. = parse_json(.message) ?? {}
56+
57+
sinks:
58+
s3:
59+
type: aws_s3
60+
inputs:
61+
- "extract_message"
62+
bucket: databend-doc
63+
region: us-east-2
64+
key_prefix: "logs/"
65+
content_type: "text/plain"
66+
encoding:
67+
codec: "native_json"
68+
auth:
69+
access_key_id: "<your-access-key-id>"
70+
secret_access_key: "<your-secret-access-key>"
71+
```
72+
73+
2. Start Vector using Docker, mapping the configuration file and local logs directory:
74+
75+
```bash
76+
docker run \
77+
-d \
78+
-v /Users/eric/Documents/vector.yaml:/etc/vector/vector.yaml:ro \
79+
-v /Users/eric/Documents/logs:/logs \
80+
-p 8686:8686 \
81+
--name vector \
82+
timberio/vector:nightly-alpine
83+
```
84+
85+
3. Wait for a moment, then check if any logs have been synced to the `logs` folder on S3:
86+
87+
```bash
88+
aws s3 ls s3://databend-doc/logs/
89+
```
90+
91+
If the log file has been successfully synced to S3, you should see output similar to this:
92+
93+
```bash
94+
95+
```
96+
97+
98+
99+

0 commit comments

Comments
 (0)