Skip to content

Commit d501c7f

Browse files
bene2k1bene2k1
committed
feat(k8s): add promtail tutorial
1 parent 80d930d commit d501c7f

File tree

1 file changed

+143
-0
lines changed
  • tutorials/integrating-kubernetes-container-logs-with-scaleway-cockpit

1 file changed

+143
-0
lines changed

‎tutorials/integrating-kubernetes-container-logs-with-scaleway-cockpit/index.mdx‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
---
2+
meta:
3+
title: Integrating Kubernetes container logs with Scaleway Cockpit
4+
description: This page explains how to integrate Kubernetes container logs with Scaleway Cockpit using Promtail
5+
content:
6+
h1: Integrating Kubernetes Container Logs with Scaleway Cockpit
7+
paragraph: This page explains how to integrate Kubernetes container logs with Scaleway Cockpit using Promtail
8+
categories:
9+
- iot-hub
10+
tags: kubernetes kapsule kosmos cockpit promtail logs
11+
dates:
12+
validation: 2025-01-13
13+
posted: 2025-01-13
14+
---
15+
16+
You can now send **data plane** logs from your [Kapsule or Kosmos](https://www.scaleway.com/en/kubernetes) clusters to [Cockpit](https://www.scaleway.com/en/cockpit/), providing centralized, real-time access to application and system logs. Reduce complexity and manual work thanks to this integration, powered by a **Promtail** deployment via [Easy Deploy](https://www.scaleway.com/en/docs/containers/kubernetes/how-to/enable-easy-deploy/).
17+
18+
With this feature:
19+
20+
- **Enhance observability**: View logs from all your Kubernetes containers in one place.
21+
- **Simplify troubleshooting**: Quickly drill down into specific pods or containers without needing to configure a separate logging stack.
22+
23+
<Message type="important">
24+
This feature does incur costs based on the volume of logs ingested. Refer to [Cockpit Usage and Pricing](/observability/cockpit/reference-content/understanding-cockpit-usage/) for more details and best practices to avoid unexpected bills.
25+
</Message>
26+
27+
<Macro id="requirements" />
28+
29+
- A running [Kapsule](/containers/kubernetes/how-to/create-cluster/) or [Kosmos](/containers/kubernetes/how-to/create-kosmos-cluster/) cluster.
30+
- Sufficient permissions to deploy Easy Deploy applications on your cluster
31+
32+
## Architecture and limitations
33+
34+
### Control Plane vs. Data Plane
35+
36+
- **Control Plane**: Fully managed by Scaleway. Users can already [monitor control plane components](/containers/kubernetes/how-to/monitor-cluster/) (e.g., `kube-apiserver`, `CCM`, `CSI`) via Cockpit.
37+
- **Data Plane**: Runs in your Scaleway Project (customer-managed instances, `kubelet`, `containerd`, customer Pods, etc.). You have **full access** to the data plane, including the ability to SSH into nodes.
38+
39+
Because the data plane is entirely under your control, **logs from any components running on these nodes are considered your own data**. Consequently, shipping these logs to Cockpit is billed based on data ingestion.
40+
41+
## How it works
42+
43+
The system leverages **Promtail** (a lightweight log collector) running on your Kapsule or Kosmos cluster. Promtail forwards logs to your Cockpit instance’s Loki endpoint:
44+
45+
1. **Promtail** can collect logs from:
46+
- **Container stdout/stderr** (pods)
47+
- **systemd journal** (e.g., `kubelet.service`)
48+
2. **Log data** is transmitted to **Cockpit** (Loki).
49+
3. **Cockpit** stores and indexes these logs.
50+
51+
## Step-by-Step: Enabling container logs in Cockpit
52+
53+
You can use Scaleway’s **Easy Deploy** to add a Promtail deployment to your cluster:
54+
55+
1. **Open the Scaleway Console** and go to your **Kubernetes** cluster.
56+
2. Navigate to the **Easy Deploy** tab.
57+
3. Select **Promtail for Cockpit** from the library.
58+
4. **Deploy** the application. Promtail will install on your cluster with default settings that:
59+
- Collect container logs for **all namespaces** (by default).
60+
- Collect systemd journal logs (e.g., `kubelet.service`).
61+
- Forward logs securely to **Cockpit**.
62+
63+
<Message type="note">
64+
You may edit the default configuration of the deployment to specify the sources of logs to ingest (under `config.snippets.scrapeConfigs` in the yaml file): `cockpit_promtail_scrape_config_pods: “… list of namespaces…” cockpit_promtail_scrape_config_journal: “… list of system components…”` |
65+
</Message>
66+
67+
### Example Promtail configuration
68+
69+
Below is a simplified snippet of the configuration that Easy Deploy generates by default:
70+
71+
```
72+
config:
73+
clients:
74+
- bearer_token: "{{{ cockpit_bearer_token }}}" # no need to modify
75+
url: "{{{ cockpit_loki_push_url }}}" # no need to modify
76+
77+
snippets:
78+
scrapeConfigs: |
79+
{{{- cockpit_promtail_scrape_config_pods }}} #default all pods are logged
80+
{{{- cockpit_promtail_scrape_config_journal }}} #default all system components are logged
81+
extraVolumeMounts:
82+
- mountPath: /var/log/journal
83+
name: journal
84+
readOnly: true
85+
extraVolumes:
86+
- hostPath:
87+
path: /var/log/journal
88+
name: journal
89+
```
90+
91+
<Message type="note">
92+
The placeholders like `{{{ cockpit_bearer_token }}}` and `{{{ cockpit_loki_push_url }}}` are automatically replaced by the Easy Deploy system with your actual values. |
93+
</Message>
94+
95+
## Observing logs in Cockpit
96+
97+
Once Promtail is running:
98+
99+
1. Go to **Cockpit** → **Kubernetes Cluster Pod Logs** in the Scaleway Console (or open your own Grafana connected to Cockpit).
100+
2. **Filter** by:
101+
- `Datasource` which is automatically created upon deployment, and visible in the Cockpit console
102+
- `Cluster Name` ( e.g. `my-kapsule-cluster`)
103+
- `namespace`, `pod`, or `container` labels to isolate specific workloads
104+
- **Time range** to limit how far back in history you want to query
105+
3. **Analyze** logs in real-time or historical mode to troubleshoot issues, watch for errors, or track performance.
106+
107+
## Usage & pricing
108+
109+
Sending logs to Cockpit is billed based on the **total volume of logs ingested**. You are charged *€0.15 per million log lines*\* (samples) or partial million.
110+
111+
For more details on controlling your costs, see our [Cockpit Usage and Pricing guide](#) (link to your existing doc or the excerpt below). Key points include:
112+
113+
- **Logging rate**: The more logs you produce (e.g. high-traffic workloads or verbose logging), the higher the bill.
114+
- **Filtering**: Limit logs to critical namespaces or system components only.
115+
116+
<Message type="tip">
117+
Always monitor the logs ingestion rate in the dedicated dashboards provided in Cockpit, just below the log tables, to avoid surprises. |
118+
</Message>
119+
120+
## Security considerations
121+
122+
- **Authentication**: The Promtail client uses a Cockpit Bearer Token to authenticate. Keep this token secret; do not store it in publicly accessible repos.
123+
- **Encryption**: Communication between Promtail and Cockpit (HTTPS) encrypts logs in transit.
124+
- **Access Control**: Ensure only trusted team members can deploy Easy Deploy applications or modify cluster-level configurations.
125+
126+
## Troubleshooting
127+
128+
- **No logs appearing** in Cockpit:
129+
1. Verify that the Promtail pod is running (`kubectl get pods -n <promtail-namespace>`).
130+
2. Inspect Promtail logs for errors.
131+
132+
133+
- **High log ingestion cost**:
134+
1. Review your **deployment configuration** to filter out verbose logs or unneeded namespaces.
135+
2. Check **log ingestion rate** in the dedicated dashboards for unusual spikes.
136+
137+
## Further resources
138+
139+
- [Observability Overview](https://www.scaleway.com/en/docs/observability/)
140+
- [Push Logs to Cockpit (How-To)](https://www.scaleway.com/en/docs/observability/cockpit/how-to/send-metrics-logs-to-cockpit/)
141+
- [Promtail Documentation](https://grafana.com/docs/loki/latest/clients/promtail/)
142+
- [Scaleway Kapsule Documentation](https://www.scaleway.com/en/docs/containers/kubernetes/kapsule/quickstart/)
143+
- [Scaleway Kosmos Documentation](https://www.scaleway.com/en/docs/containers/kubernetes/kosmos/quickstart/)

0 commit comments

Comments
 (0)