Skip to content

Commit 8ba2fb9

Browse files
ssorichessoriche
committed
feat(vector-agent): Add log ingester
Initial commit of Vector Agent that collects logs from running pods and forwards them to Loki. The installation uses the Vector project (https://github.com/vectordotdev/vector), manifests generation uses the `Makefile`, with application using kustomize. As the upstream project using helm for configuration, most configuration options are in the values.yaml file. Changes to the settings, requires rerunning the `make manifests` to create the new manifests. Commit the results to the metacpan-k8s repository.
1 parent 82d6ed1 commit 8ba2fb9

File tree

8 files changed

+572
-0
lines changed

8 files changed

+572
-0
lines changed

‎platform/vector-agent/Makefile‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
.DEFAULT_GOAL := help
2+
3+
# Default Vector version
4+
DEFAULT_VERSION := 0.37.0
5+
VECTOR_VERSION ?= $(DEFAULT_VERSION)
6+
VECTOR_AGENT_OUTPUT_YAML := vendor/vector-agent.yaml
7+
AGENT_VALUES_FILE := $(CURDIR)/values.yaml
8+
9+
.PHONY: all clean manifests check-tools
10+
11+
all: check-tools clean manifests
12+
13+
.PHONY: init
14+
init:
15+
helm repo add vector https://helm.vector.dev
16+
helm repo update
17+
18+
.PHONY: manifests
19+
manifests:
20+
@echo "Removing previous $(VECTOR_AGENT_OUTPUT_YAML)..."
21+
rm -f $(VECTOR_AGENT_OUTPUT_YAML)
22+
@echo "Pulling Vector YAML (Version: $(VECTOR_VERSION)) from Helm chart..."
23+
helm repo update
24+
helm template vector --values $(AGENT_VALUES_FILE) --namespace vector-agent vector/vector --version $(VECTOR_VERSION) > $(VECTOR_AGENT_OUTPUT_YAML)
25+
26+
.PHONY: upgrade
27+
upgrade:
28+
@echo "Removing previous $(VECTOR_AGENT_OUTPUT_YAML)..."
29+
rm -f $(VECTOR_AGENT_OUTPUT_YAML)
30+
@echo "Pulling Vector YAML (Version: $(VECTOR_VERSION)) from Helm chart..."
31+
helm repo update
32+
helm upgrade --values $(AGENT_VALUES_FILE) --namespace vector-agent vector/vector > $(VECTOR_AGENT_OUTPUT_YAML)
33+
34+
.PHONY: check-tools
35+
check-tools:
36+
@command -v helm >/dev/null 2>&1 || { echo >&2 "helm is not installed. Aborting."; exit 1; }
37+
38+
# Help target to display usage information
39+
.PHONY: help
40+
help:
41+
@echo "Makefile for pulling Vector YAML"
42+
@echo ""
43+
@echo "Targets:"
44+
@echo " init : Add Loki helm chart repository"
45+
@echo " manifests : Pull Vector YAML from Helm chart with specified values.yaml and remove the old Vector YAML"
46+
@echo " upgrade : Pull Vector YAML from Helm chart with specified values.yaml and upgrade the old Vector YAML"
47+
@echo " check-tools : ensure required tools are installed"
48+
@echo " help : Display this help message"

‎platform/vector-agent/README.md‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# Vector Agent
2+
3+
[Vector](https://github.com/vectordotdev/vector) is a high-performance
4+
observability data pipeline that allows you to collect, transform, and route
5+
all your logs, metrics, and traces. It is designed to be highly efficient and
6+
reliable, making it a great choice for managing observability data at scale.
7+
8+
## Makefile Overview
9+
10+
This Makefile manages the Loki Helm chart, which simplifies the deployment and
11+
management of Loki in a Kubernetes environment. The Makefile provides targets
12+
to automate common tasks such as pulling the latest Loki YAML configuration,
13+
upgrading Loki, and ensure required tools are present.
14+
15+
## Prerequisites
16+
17+
Before using the Makefile, ensure you have the following tools installed:
18+
19+
- `helm`
20+
21+
### Makefile Targets
22+
23+
- **all**: This is the default target that runs `check-tools`, `clean`, and
24+
`manifests` targets in sequence.
25+
- **init**: Adds the Vector Helm repository and updates the Helm repository
26+
index.
27+
- **manifests**: Removes the previous Vector Agent YAML file and pulls the
28+
latest Vector Agent YAML configuration from the Helm chart using the specified
29+
`values.yaml` file.
30+
- **upgrade**: Removes the previous Vector Agent YAML file and upgrades the Loki
31+
deployment using the latest configuration from the Helm chart.
32+
- **check-tools**: Ensures that the required tools (e.g., Helm) are available
33+
on the system.
34+
- **help**: Displays usage information for the Makefile targets.
35+
36+
### Usage
37+
38+
1. **Initialize Helm Repository**
39+
40+
Before using the Makefile, you need to initialize the Helm repository:
41+
42+
```sh
43+
make init
44+
```
45+
46+
2. **Generate Vector Agent Manifests**
47+
48+
To generate the Vector Agent YAML configuration file, run:
49+
50+
```sh
51+
make manifests
52+
```
53+
54+
This will remove the previous `vector-agent.yaml` file and pull the latest
55+
configuration from the Helm chart.
56+
57+
3. **Upgrade Vector Agent**
58+
59+
To upgrade the Vector Agent deployment with the latest configuration, run:
60+
61+
```sh
62+
make upgrade
63+
```
64+
65+
This will remove the previous `vector-agent.yaml` file and upgrade the Loki deployment.
66+
67+
4. **Check Required Tools**
68+
69+
Ensure the installation of required tools by running:
70+
71+
```sh
72+
make check-tools
73+
```
74+
75+
This will check that Helm is available on your system.
76+
77+
5. **Display Help**
78+
79+
To display the help message with information about the Makefile targets, run:
80+
81+
```sh
82+
make help
83+
```
84+
85+
### Default Variables
86+
87+
- **DEFAULT_VERSION**: The default version of Vector Agent to use (e.g.,
88+
`0.37.0`).
89+
- **VECTOR_VERSION**: The version of Vector Agent to use, which can be
90+
overridden by setting this variable.
91+
- **VECTOR_AGENT_OUTPUT_YAML**: The output file for the Vector Agent YAML
92+
configuration (default: `vendor/vector-agent.yaml`).
93+
- **VALUES_FILE**: The path to the `values.yaml` file used for Helm
94+
configuration.
95+
96+
### Example
97+
98+
To generate the Vector Agent manifests using a specific version and values
99+
file, you can run:
100+
101+
```sh
102+
make VECTOR_VERSION=0.22.1 VALUES_FILE=/path/to/your/values.yaml manifests
103+
```
104+
105+
This will pull the specified version of Vector Agent and use the provided
106+
`values.yaml` file for configuration.

‎platform/vector-agent/do/application.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
apiVersion: argoproj.io/v1alpha1
2+
kind: Application
3+
metadata:
4+
name: vector-agent
5+
namespace: argocd
6+
spec:
7+
project: default
8+
source:
9+
repoURL: https://github.com/metacpan/metacpan-k8s
10+
targetRevision: main
11+
path: platform/vector-agent/do
12+
destination:
13+
server: https://kubernetes.default.svc
14+
namespace: vector-agent
15+
syncPolicy:
16+
automated:
17+
prune: true
18+
selfHeal: false

‎platform/vector-agent/do/configmap-vector-agent.yaml‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
---
2+
apiVersion: v1
3+
kind: ConfigMap
4+
metadata:
5+
name: vector-agent
6+
labels:
7+
app.kubernetes.io/name: vector
8+
app.kubernetes.io/instance: vector
9+
app.kubernetes.io/component: Agent
10+
data:
11+
agent.yaml: |
12+
data_dir: /vector-data-dir
13+
api:
14+
enabled: false
15+
sources:
16+
kubernetes_logs:
17+
type: kubernetes_logs
18+
max_line_bytes: 1048576
19+
host_metrics:
20+
filesystem:
21+
devices:
22+
excludes:
23+
- binfmt_misc
24+
filesystems:
25+
excludes:
26+
- binfmt_misc
27+
mountPoints:
28+
excludes:
29+
- "*/proc/sys/fs/binfmt_misc"
30+
type: host_metrics
31+
internal_metrics:
32+
type: internal_metrics
33+
sinks:
34+
prom_exporter:
35+
type: prometheus_exporter
36+
inputs:
37+
- host_metrics
38+
- internal_metrics
39+
address: 0.0.0.0:9090
40+
loki:
41+
type: loki
42+
inputs:
43+
- kubernetes_logs
44+
endpoint: http://loki-write.loki.svc.cluster.local:3100
45+
out_of_order_action: accept
46+
encoding:
47+
codec: json
48+
healthcheck:
49+
enabled: false
50+
labels:
51+
source_type: "{{ .source_type }}"
52+
namespace: "{{ kubernetes.pod_namespace }}"
53+
container: "{{ kubernetes.container_name }}"
54+
pod: "{{ kubernetes.pod_name }}"
55+
container_name: "{{ kubernetes.container_name }}"
56+
node: "{{ kubernetes.pod_node_name }}"
57+
stream: "{{ .stream }}"
58+
forwarder: vector-agent

‎platform/vector-agent/do/kustomization.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
4+
resources:
5+
- ./application.yaml
6+
- ../vendor/
7+
8+
patches:
9+
- path: ./configmap-vector-agent.yaml

‎platform/vector-agent/values.yaml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Default values for Vector
2+
# See Vector helm documentation to learn more:
3+
# https://vector.dev/docs/setup/installation/package-managers/helm/
4+
5+
# nameOverride -- Override the name of resources.
6+
nameOverride: "agent"
7+
8+
# role -- [Role](https://vector.dev/docs/setup/deployment/roles/) for this Vector instance, valid options are:
9+
# "Agent", "Aggregator", and "Stateless-Aggregator".
10+
11+
# Each role is created with the following workloads:
12+
# Agent = DaemonSet
13+
# Aggregator = StatefulSet
14+
# Stateless-Aggregator = Deployment
15+
role: "Agent"
16+
17+
# rollWorkload -- Add a checksum of the generated ConfigMap to workload annotations.
18+
rollWorkload: true
19+
20+
rbac:
21+
# rbac.create -- If true, create and use RBAC resources. Only valid for the "Agent" role.
22+
create: true
23+
24+
serviceAccount:
25+
# serviceAccount.create -- If true, create a ServiceAccount for Vector.
26+
create: true
27+
# serviceAccount.annotations -- Annotations to add to Vector's ServiceAccount.
28+
annotations: {}
29+
# serviceAccount.name -- The name of the ServiceAccount to use. If not set and serviceAccount.create is true, a name
30+
# is generated using the fullname template.
31+
name:
32+
# serviceAccount.automountToken -- Automount API credentials for Vector's ServiceAccount.
33+
automountToken: true
34+
35+
# resources -- Set Vector resource requests and limits.
36+
resources:
37+
# https://vector.dev/docs/setup/installation/platforms/kubernetes/#agent-resource-limits
38+
requests:
39+
memory: "64Mi"
40+
cpu: "500m"
41+
limits:
42+
memory: "1024Mi"
43+
cpu: "6000m"

‎platform/vector-agent/vendor/kustomization.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
4+
resources:
5+
- ./vector-agent.yaml

0 commit comments

Comments
 (0)