Skip to content

feat(cockpit): retention updates #4778

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Apr 7, 2025
Merged

feat(cockpit): retention updates #4778

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
f5edf84
docs(add): add info
nerda-codes Mar 26, 2025
616d5b1
docs(cpt): retention pricing updates
nerda-codes Apr 2, 2025
56db5c7
docs(cpt): add more info
nerda-codes Apr 3, 2025
8006f1d
docs(cpt): add more information
nerda-codes Apr 4, 2025
6a2e272
Update pages/cockpit/how-to/change-data-retention.mdx
nerda-codes Apr 4, 2025
d831a34
docs(cpt): fix
nerda-codes Apr 7, 2025
df2c7b9
docs(cpt): feedback sophie
nerda-codes Apr 7, 2025
8238311
docs(cpt): fix links
nerda-codes Apr 7, 2025
35748bc
docs(cpt): fix concept
nerda-codes Apr 7, 2025
6f9a9b6
docs(cpt): add difference between scw and custom data
nerda-codes Apr 7, 2025
2534c30
docs(cpt): fix data source typo
nerda-codes Apr 7, 2025
a6fe131
docs(fix): typo
nerda-codes Apr 7, 2025
b6c886d
fix(cpt): rm traces
nerda-codes Apr 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion changelog/may2024/2024-05-24-cockpit-added-cockpit-available-in-all-region.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@ category: observability
product: cockpit
---

Cockpit is now available in all three Scaleway regions: Paris, Amsterdam, and Warsaw. From the Scaleway console, you can now decide in which region to enable the [alert manager](/cockpit/concepts/#alert-manager) and your [preconfigured alerts](/cockpit/concepts/#preconfigured-alerts). You can also choose the regions in which to create your [data types](/cockpit/concepts/#data-types), [data sources](/cockpit/concepts/#data-sources), and [tokens](/cockpit/concepts/#tokens).
Cockpit is now available in all three Scaleway regions: Paris, Amsterdam, and Warsaw. From the Scaleway console, you can now decide in which region to enable the [alert manager](/cockpit/concepts/#alert-manager) and your [preconfigured alerts](/cockpit/concepts/#preconfigured-alerts). You can also choose the regions in which to create your [data types](/cockpit/concepts/#data-types), [data sources](/cockpit/concepts/#data-sources), and [tokens](/cockpit/concepts/#cockpit-tokens).

17 changes: 12 additions & 5 deletions macros/cockpit/plan-deprecation.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,18 @@
macro: cockpit-plan-deprecation
---

<Message type="important">
<Message type="note">

**Cockpit pricing plans were deprecated on January 1st 2025** <br /><br />
**Pricing update for Cockpit custom retention** <br /><br />

The [retention](/cockpit/concepts/#retention) period previously set for your Scaleway and custom logs, metrics and traces remains the same. <br /><br />
You can change the retention period for your metrics, logs, and traces for free during the retention period edition beta (starting January 1st 2025). <br /><br />
Find out [how to change data retention period](/cockpit/how-to/change-data-retention/) in the dedicated documentation.
On January 1st, 2025, Cockpit pricing plans were deprecated and replaced by the custom [retention](/cockpit/concepts/#retention) feature, available for free during beta.
On May 1st, 2025, this feature reaches general availability and **becomes billable**. <br /><br />

**Logs and traces**: free retention for 7 days, then charged €0.002/GB/day

**Metrics**: free retention for 31 days, then charged €0.0002/10 million samples/day

Ingestion of custom data remains billable at [the current pricing](/cockpit/faq/#how-am-i-billed-for-using-cockpit-with-custom-data).

Refer to the [dedicated documentation](/cockpit/how-to/change-data-retention/) to adjust retention settings.
</Message>
81 changes: 29 additions & 52 deletions pages/cockpit/concepts.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,11 @@ content:
paragraph: Explore Scaleway Observability concepts including metrics, logs, and alerts management through Cockpit. Learn about agents, alerting rules, Grafana integration, and data types for comprehensive monitoring.
tags: observability alert-manager contact-points endpoint grafana loki logql prometheus promql tokens
dates:
validation: 2024-12-26
validation: 2025-04-03
categories:
- observability
---

## Active series

Active series refer to [time series](#time-series) for which the latest [samples](#samples) received by your Cockpit are less than 10 minutes old.

## Agent

An agent is a software component that runs on your systems to gather [data types](#data-types) from the host system or applications running on it. The agent then forwards this data to Cockpit for analysis and visualization.
Expand All @@ -28,16 +24,12 @@ An agent is a software component that runs on your systems to gather [data types

Alerting detects complex conditions defined by a rule and keeps you aware of issues in your environments. When a condition defined by a rule is met, the rule tracks it as an alert and responds by triggering one or more actions.

<Message type="important">
The **Grafana alert manager** on the Grafana interface is inactive. We strongly recommend that you select the **Scaleway Alerting** alert manager if you want to manage your alerts using Grafana.
</Message>

## Alert manager

Scaleway's regionalized alert manager allows you to manage and respond to alerts according to the regions you have enabled it in. It handles alerts sent when the alerting rules we run are firing. The alert manager triggers alerts (e.g. emails or texts) if a criteria you have configured on your applications' metrics and logs is activated.

<Message type="important">
The **Grafana alert manager** on the Grafana interface is inactive. We strongly recommend that you select the **Scaleway Alerting** alert manager if you want to manage your alerts using Grafana.
Scaleway does not support the **Grafana alert manager** on the Grafana interface. You must select the **Scaleway Alerting** alert manager if you want to manage your alerts using Grafana.
</Message>

## Alerting rules
Expand All @@ -52,10 +44,23 @@ Alerting rules allow you to define criteria that determine whether an alert is t

A Cockpit is an instance of the Observability product that stores metrics, logs, and traces and provides a dedicated dashboarding system on Grafana to visualize them. A Scaleway Project can have only one Cockpit, which is automatically activated when you are using [Scaleway resources that are integrated into Cockpit](/cockpit/reference-content/cockpit-limitations/#product-integration-into-cockpit).

## Cockpit tokens

Tokens are regionalized secret keys that allow you to authenticate against the endpoints of your Cockpit's [data sources](#data-sources) (metrics, logs, alerts). You can generate new tokens and select token permissions such as:

- **Push**: allows you to send your metrics, logs, and traces to your Cockpit.
- **Query**: allows you to fetch your metrics, logs, and traces from your Cockpit.
- **Rules**: allow you to configure alerting and recording rules.
- **Alerts**: allow you to set up the alert manager.

## Contact points

Contact points define who is notified when an alert fires, according to the region in which you have added them. Contact points include emails, Slack, on-call systems, and texts. When an alert fires, all contact points are notified.

## Custom data

Custom data is any data (metrics, logs or traces) that you may push to Cockpit. It can be data collected from your applications hosted at Scaleway or elsewhere.

## Data sources

Data sources are regionalized backends that allow you to store and fetch your metrics, logs, and traces. By default, **Scaleway data sources** are enabled if you are using [Scaleway resources integrated with Cockpit](/cockpit/reference-content/cockpit-limitations/#product-integration-into-cockpit) on your Project.
Expand Down Expand Up @@ -102,14 +107,6 @@ A Grafana user is any individual who can log in to [Grafana](https://grafana.com

Loki is the log aggregation system used by [Grafana](https://grafana.com/docs/grafana/latest/introduction/) to store and query your logs.

## Loki Remote Write

`Loki Remote Write` is the protocol used to push your logs to your Cockpit's logs' endpoint.

## LogQL

LogQL is [Grafana Loki’s language](https://grafana.com/docs/loki/latest/logql/) for querying logs. LogQL uses labels and operators for filtering.

## Logs

Logs are a data type that provides a record of all events and errors taking place during the lifecycle of your resources. They represent an excellent source of visibility if you want to know when a problem occurred, or which events correlate with it.
Expand Down Expand Up @@ -138,22 +135,10 @@ You can push metrics with any Prometheus-compatible agent such as [Prometheus](h

A preconfigured dashboard is a set of one or more panels that Scaleway sets up and updates for you to visualize the metrics and logs associated with your Scaleway products.

## Prometheus Remote Write

`Prometheus Remote Write` is the protocol used to push your metrics to your Cockpit's metrics' endpoint.

## PromQL

PromQL, short for Prometheus Querying Language, is the main way to query metrics within Prometheus. It is designed for building queries for graphs and alerts.

## Receivers

Receivers are hubs consisting of contact points. You can associate one or several alerts with one or more receivers. This allows you to diversify your alerts.

## Recording rules

Recording rules allow you to precompute frequently needed or computationally expensive expressions and save their results as a new set of time series.

## Region

A region is the geographical area in which your Cockpit data is stored. They are duplicated on all Availability Zones of the selected regions (Paris, Amsterdam, or Warsaw).
Expand All @@ -162,45 +147,37 @@ You can decide in which region to enable the [alert manager](#alert-manager) and

## Retention

Retention or data retention refers to the duration for which data, such as metrics, logs, or traces, is stored before being automatically deleted. Retention allows you to manage the lifecycle of your Scaleway and custom data by selecting storage periods that align with your needs.
Retention or data retention refers to the duration for which the data (metrics, logs, and traces) pushed to Cockpit (by Scaleway products or yourself) is stored before being automatically deleted. Retention allows you to manage the lifecycle of your Scaleway and custom data by selecting storage periods that align with your needs.

The minimum and maximum retention periods for each data source type are as follows:

| Custom metrics | Custom logs/traces | Scaleway metrics | Scaleway logs/traces |
|------------------------------------------------|---------------------------------------------|------------------------------------------------|---------------------------------------------|
| Minimum retention period: 1 day | Minimum retention period: 1 day | Minimum retention period: 31 days | Minimum retention period: 1 day |
| Maximum retention period: 365 days (12 months) | Maximum retention period: 31 days (1 month) | Maximum retention period: 365 days (12 months) | Maximum retention period: 31 days (1 month) |
| Default retention period: 31 days | Default retention period: 7 days | Default retention period: 31 days | Default retention period: 7 days |
| Retention period | Custom metrics | Custom logs/traces | Scaleway metrics | Scaleway logs/traces |
|-------------------|-------------------|--------------------|-------------------|----------------------|
| Minimum retention | 1 day | 1 days | 31 days | 1 day |
| Maximum retention | 365 days (1 year) | 31 days | 365 days (1 year) | 31 days |
| Default retention | 31 days | 7 days | 31 days | 7 days |

**Starting May 1, 2025, the maximum retention period for custom and Scaleway metrics, logs and traces will be of 1825 days (5 years)**. The information on the table above will be updated accordingly.

## Samples

A sample is a unique measuring point on a time series.

## Scaleway data

Scaleway data is the observability data (metrics and/or logs) that is natively collected by all Scaleway products, integrated with Cockpit.
Refer to the [dedicated documentation page](/cockpit/reference-content/cockpit-limitations/#product-integration-into-cockpit) to find out which resources are integrated with Cockpit.

## Tempo

Tempo is [Grafana's open source](https://grafana.com/docs/tempo/latest/) tracing tool that allows you to search for traces, generate metrics from them, and link your tracing data with logs and metrics. Tempo can be used with open-source tracing protocols such as [Jaeger](https://www.jaegertracing.io/docs/1.50/), [Zipkin](https://zipkin.io/), and [OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/).

<Message type="important">
During the beta of the traces feature, only the [OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) HTTP push path is supported. Scaleway is working on implementing the [Open Telemetry gRPC](https://opentelemetry.io/docs/specs/semconv/rpc/grpc/), [Zipkin](https://zipkin.io/) and [Jaeger](https://www.jaegertracing.io/docs/1.50/) protocols.
</Message>
Scaleway only supports the OpenTelemetry HTTP push path.

## Time series

A time series is a sequence of numerical data points in successive order, that tracks the value of a parameter over time. Example of parameter: the disk usage of a machine hosting a database, expressed in percentage.

## Tokens

Tokens are regionalized secret keys that allow you to authenticate against your Cockpit's endpoints (metrics, logs, alerts). You can generate new tokens and select token permissions such as:

- **Push**: allows you to send your metrics, logs, and traces to your Cockpit.
- **Query**: allows you to fetch your metrics, logs, and traces from your Cockpit.
- **Rules**: allow you to configure alerting and recording rules.
- **Alerts**: allow you to set up the alert manager.

## TraceQL

[TraceQL](https://grafana.com/docs/tempo/latest/traceql/) is Grafana Tempo's query language designed for searching and extracting traces.

## Traces

Traces are detailed records of your **requests' behavior**, as they move through distributed systems such as microservices and containers.
Expand Down
62 changes: 49 additions & 13 deletions pages/cockpit/faq.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ meta:
content:
h1: Cockpit FAQ
dates:
validation: 2024-11-25
validation: 2025-03-27
category: observability
productIcon: CockpitProductIcon
---
Expand All @@ -23,27 +23,37 @@ Cockpit integrates seamlessly with Scaleway’s ecosystem. It provides pre-built

Refer to the [dedicated documentation page](/cockpit/reference-content/cockpit-limitations/#product-integration-into-cockpit) for the whole list of Scaleway resources integrated with Cockpit.

## How am I billed for using Cockpit with my Scaleway data?
## What is the difference between Scaleway and custom data?

[Scaleway data](#scaleway-data) is collected automatically from Scaleway products, while [custom data](#custom-data) is manually pushed from your own sources.

### Scaleway data

Scaleway data is the observability data (metrics and/or logs) that is natively collected by all [Scaleway products integrated with Cockpit](/cockpit/reference-content/cockpit-limitations/#product-integration-into-cockpit).

Using Cockpit with your Scaleway data is free of charge.
### Custom data

During the beta phase of the retention period edition feature, the retention period you set for your data - whether from [Scaleway resources integrated with Cockpit](/cockpit/reference-content/cockpit-limitations/#product-integration-into-cockpit) or custom data - will be free of charge.
Custom data refers to any metrics, logs, or traces that you manually push to Cockpit. This can include data from applications hosted at Scaleway or any other platform.

Additional costs apply for sending custom data to Cockpit. This means that you will be billed for sending **data from Scaleway resources that are not integrated with Cockpit**, or data from any other external resources.
## How am I billed for using Cockpit with my Scaleway data?

Scaleway data is collected and available in Cockpit for free. Retention is also free as long as it stays within the default period: 31 days for metrics and 7 days for logs and traces.

You can [adjust the retention period](/cockpit/how-to/change-data-retention) from 1 day (for logs and traces) or 31 days (for metrics) to 1 year. Refer to the [dedicated documentation](/cockpit/concepts/#retention) for more information on available retention periods. However, starting May 1, 2025, data stored beyond the default period will incur charges based on daily storage volume.

## How am I billed for using Cockpit with custom data?

Custom data is any data that is external to Scaleway, or **data from Scaleway resources not integrated with Cockpit**. Refer to the [dedicated documentation page](/cockpit/reference-content/cockpit-limitations/#product-integration-into-cockpit) to find out which resources are integrated with Cockpit.
Sending custom data to Cockpit such as data from non-integrated Scaleway resources or external sources incurs additional costs.

During the beta of the retention period edition feature, the retention period you set for your custom data is free of charge.
Retention of custom data is free within the default period: 31 days for custom metrics and 7 days for custom logs and traces. You can [adjust the retention period](/cockpit/how-to/change-data-retention) from 1 day (for logs and traces) and 31 days (for metrics) to 1 year. However, starting May 1, 2025, data stored beyond the default period will be charged based on daily storage volume.

You will not be charged for **querying** custom data.
Querying custom data is always free.

Data ingestion is billed as follows:
Ingestion for custom data is billed as follows:

- Custom metrics are billed €0.15 per million samples of metrics ingested, per month.
- Custom logs are billed €0.35 per GB ingested, per month.
- Custom traces are billed €0.35 per GB ingested, per month.
- Custom metrics are billed €0.15 per million samples of metrics ingested
- Custom logs are billed €0.35 per GB ingested
- Custom traces are billed €0.35 per GB ingested

Scaleway applies volume discounts to bill custom metrics. This means that the monthly pricing for custom metrics will be calculated in levels. **Scaleway applies six levels of volume discounts.**

Expand All @@ -59,14 +69,40 @@ Scaleway applies volume discounts to bill custom metrics. This means that the mo

Here is an example of how you would be billed for **sending 52 billion custom metrics samples per month**:

Here is an example of how you would be billed for sending 52 billion custom metrics samples per month:
- 10 billion samples at €0.15 per million samples: €1.500
- 15 billion samples at €0.13 per million samples: €1.950
- 25 billion samples at €0.12 per million samples: €3.000
- 2 billion samples at €0.11 per million samples: €220

**Total:** €6.650 per month with volume discounts instead of €7.800 without volume discounts.

## How am I billed for increasing data retention period?

All ingested data, whether from Scaleway or custom sources, is retained for free within the default periods:
- **Metrics:** 31 days
- **Logs and traces:** 7 days

**Starting May 1, 2025, data retained beyond these periods will be charged** based on daily storage volume:

- **Metrics:** €0.0002 per 10 million samples/day
- **Logs and traces:** €0.002 per GB/day

If you delete your data source or reduce its retention period below the default value, data will be deleted and you will no longer be charged for extended retention.

<Concept>
## See custom retention pricing examples
### For 10 days of retention
If you ingest an average of **2GB of logs daily** and increase retention to **10 days**. Retention is free for the first **7 days**, so you are only charged for the extra **3 days**:

**Monthly estimated cost:**
`retention_cost` = 2 GB x (10 - 7) x 0.002€ x 30 days = 0.36€/month

### For 90 days of retention
If you extend retention to **90 days**, you are charged for the extra **83 days** beyond the free period:
**Monthly estimated cost:**
`retention_cost` = 2 GB x (90 - 7) x 0.002€ x 30 days = 9.96€/month
</Concept>

## Why are Cockpit pricing plans being deprecated?

From January 1st 2025, Cockpit is transitioning away from fixed pricing plans to offer you more flexibility and granularity for managing data [retention](/cockpit/concepts/#retention).
Expand Down
Loading