Skip to content

DOCS-1062 - Update "Disabled Metrics Sources" article #5677

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Aug 13, 2025
Merged

DOCS-1062 - Update "Disabled Metrics Sources" article #5677

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a315322
Update article
jpipkin1 Aug 11, 2025
f520417
Merge branch 'main' into docs-1062-cardinality
jpipkin1 Aug 13, 2025
5246ca8
Minor tweak to first seen rules article
jpipkin1 Aug 13, 2025
59ee47d
Update audit log example
jpipkin1 Aug 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/cse/rules/write-first-seen-rule.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,5 +154,5 @@ If you are unsure whether to use a per-entity or a global baseline, consider you

## Additional resources

Blog: [From weeks to minutes: How Sumo Logic’s historic baselining supercharges UEBA](https://www.sumologic.com/blog/sumo-logic-historic-baselining)
* Blog: [From weeks to minutes: How Sumo Logic’s historic baselining supercharges UEBA](https://www.sumologic.com/blog/sumo-logic-historic-baselining)
* Glossary: [User entity behavior analytics (UEBA)](https://www.sumologic.com/glossary/ueba)
22 changes: 11 additions & 11 deletions docs/metrics/manage-metric-volume/disabled-metrics-sources.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,30 +24,30 @@ The storage is based on the metrics retention period, which you can control usi

### Warning is issued when you approach the global limits

When you approach one of these limits, Sumo Logic generates a Health Event and writes a message with level “warning” to the Audit Event Index.
When you approach one of these limits, Sumo Logic generates a health event and writes a message with level “warning” to the [system event index](/docs/manage/security/audit-indexes/system-event-index/). Enter a query to `_index=sumologic_system_events` to see events in the system event index.

The Health Event and audit message are generated when your metric ingestion reaches these levels:
The health event and audit message are generated when your metric ingestion reaches these levels:

* 35M unique timeseries per week, for metrics with long term retention
* 70M unique timeseries per week, for metrics with short term retention

The Health Event is named `MetricsHighCardinalityDetected`. 
The health event is named [`HighCardinalityMetricsDetected`](https://service.sumologic.com/audit/docs/#operation/getHighCardinalityMetricsDetected). 

The message written to the Audit Event Log is:
Following is an example of the message written to the system event index:

```json
{"status":"UnHealthy","details":{"retention":"long","trackerId":"MetricsHighCardinalityDetected","error":"Detected high cardinality of metrics time series","description":"Approaching the limit for total number of unique time series allowed. In case of exceeding the limit some of your metrics sources would be temporary disabled."},"eventType":"Health-Change","severityLevel":"Warning","accountId":"0000000000000475","eventId":"0687c55e-0b77-44a4-9a6f-6d6d5e588244","eventName":"MetricsHighCardinalityDetected","eventTime":"2020-06-18T14:45:48.252Z","eventFormatVersion":"1.0 beta","subsystem":"Metrics","resourceIdentity":{"id":"0000000000000475","name":"stagData","type":"Organisation"}}
{"status":"UnHealthy","details":{"retention":"long","trackerId":"HighCardinalityMetricsDetected","error":"Detected high cardinality in metrics time series","description":"Approaching the limit for total number of unique time series allowed. In case of exceeding the limit some of your metrics sources would be temporary disabled."},"eventType":"Health-Change","severityLevel":"Warning","accountId":"0000000000000475","eventId":"0687c55e-0b77-44a4-9a6f-6d6d5e588244","eventName":"HighCardinalityMetricsDetected","eventTime":"2020-06-18T14:45:48.252Z","eventFormatVersion":"1.0 beta","subsystem":"Metrics","resourceIdentity":{"id":"0000000000000475","name":"stagData","type":"Organisation"}}
```

### Sources are disabled when you reach the global limits

When you reach the global limits, Sumo Logic starts disabling your metric sources, starting with the one that is ingesting metrics with the highest cardinality, and continues disabled metric sources in that order, until your metric ingestion is reduced to a volume that is lower than the limit.

For each source it disabled, Sumo Logic generates a Health Event and writes a message with level “error” to the Audit Event Index.
For each source it disabled, Sumo Logic generates a health event and writes a message with level “error” to the [system event index](/docs/manage/security/audit-indexes/system-event-index/). Enter a query to `_index=sumologic_system_events` to see events in the system event index.

The Health Event is named `SourceDisabled`. 
The health event is named [`SourceDisabled`](https://service.sumologic.com/audit/docs/#operation/getSourceDisabled). 

The message written to the Audit Event Log is:
Following is an example of the message written to the system event index:

```json
{"status":"UnHealthy","details":{"trackerId":"SourceDisabled","error":"Metrics source temporarily disabled","description":"This metrics source has sent too many unique time series and has been temporarily disabled. The data sent while this source is disabled cannot be recovered."},"eventType":"Health-Change","severityLevel":"Error","accountId":"0000000000000475","eventId":"4b1e4710-bef6-4ebe-926b-57e6b4743e9a","eventName":"SourceDisabled ","eventTime":"2020-06-18T15:00:20.776Z","eventFormatVersion":"1.0 beta","subsystem":"Metrics","resourceIdentity":{"collectorId":"000000000627859B","collectorName":"stag-cass-metrics-aa-2","id":"000000000644FB28","name":"HostMetrics","type":"Source"}}
Expand Down Expand Up @@ -109,11 +109,11 @@ You can delete the metric transformation rule, but you can’t disable or modify
If you have a use case that requires the dropped dimension, contact Sumo Logic support.
:::

When a dimension is dropped, Sumo Logic generates a Health Event and writes a message with level “error” to the Audit Event Index.
When a dimension is dropped, Sumo Logic generates a health event and writes a message with level “error” to the [system event index](/docs/manage/security/audit-indexes/system-event-index/). Enter a query to `_index=sumologic_system_events` to see events in the system event index.

The Health Event is named `HighCardinalityDimensionDropped`.
The health event is named [`HighCardinalityDimensionDropped`](https://service.sumologic.com/audit/docs/#operation/getHighCardinalityDimensionDropped).

The message written to the Audit Event Index is:
Following is an example of the message written to the system event index:

```json
{"status":"UnHealthy","details":{"dimension":"monitoridentifier","trackerId":"HighCardinalityDimensionDropped","error":"Dropped highly cardinal metrics dimension","description":"This metrics source has sent metrics with too many unique values of one dimension. Therefore said dimension will be dropped from metrics coming from this source."},"eventType":"Health-Change","severityLevel":"Error","accountId":"0000000000000131","eventId":"7354fe41-bd6e-46e2-802b-bc6b42a97406","eventName":"HighCardinalityDimensionDropped","eventTime":"2020-06-18T15:49:57.803Z","eventFormatVersion":"1.0 beta","subsystem":"Metrics","resourceIdentity":{"collectorId":"00000000064C90BE","collectorName":"nite-alert-1","id":"000000000689D385","name":"carbon2udp","type":"Source"}}
Expand Down