Skip to content

[Observability] Add log data sources page #1791

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 18 commits into from
Jul 31, 2025
Merged

[Observability] Add log data sources page #1791

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
fe5c44d
add logs sources page
mdbirnstiehl Jun 18, 2025
d5ddfde
update toc
mdbirnstiehl Jun 18, 2025
fa650ed
Merge branch 'main' into log-data-sources
mdbirnstiehl Jul 1, 2025
32d4bf0
update API steps
mdbirnstiehl Jul 1, 2025
500c8ee
fix link
mdbirnstiehl Jul 1, 2025
ae41917
fix link
mdbirnstiehl Jul 1, 2025
525433a
actually fix links
mdbirnstiehl Jul 1, 2025
95942bc
Merge branch 'main' into log-data-sources
mdbirnstiehl Jul 30, 2025
e5bf3b2
update API steps
mdbirnstiehl Jul 30, 2025
cbc452c
fix link
mdbirnstiehl Jul 30, 2025
5a6f2b3
update steps
mdbirnstiehl Jul 30, 2025
0783b95
fix typo
mdbirnstiehl Jul 30, 2025
2894465
fix wording
mdbirnstiehl Jul 30, 2025
eb660eb
update note
mdbirnstiehl Jul 30, 2025
c6325e9
fix applies to
mdbirnstiehl Jul 30, 2025
788a4b2
update import instructions
mdbirnstiehl Jul 31, 2025
e4b6692
Update solutions/observability/logs/log-data-sources.md
mdbirnstiehl Jul 31, 2025
0457764
Merge branch 'main' into log-data-sources
mdbirnstiehl Jul 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions solutions/observability/logs/log-data-sources.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
---
applies_to:
stack: all
serverless: all
products:
- id: observability
---

# Configure log data sources

The `observability:logSources` {{kib}} advanced setting defines which index patterns your deployment or project uses to store and query log data.

Configure this setting at **Stack Management** → **Advanced Settings** or by searching for `Advanced Settings` in the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).


::::{note}
Adding indices to the `observability:logSources` setting that don't contain log data may cause degraded functionality. Changes to this setting can also impact the sources queried by log threshold rules.
::::

## Configure log data sources using the `saved_objects` API

To configure log data sources using an API, use the `saved_objects` API. To do this,

1. From **Stack Management** → **Saved Objects**, [export](/explore-analyze/find-and-organize/saved-objects.md) the log data views, which are stored as an `infrastructure-monitoring-log-view` saved object type, to use as a template.
1. Modify the relevant data view fields in the exported JSON.
1. Import the saved object using the [import saved objects API]({{kib-apis}}/operation/operation-importsavedobjectsdefault).
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@weltenwort I pulled these instructions from a comment you made on an older issue from last year. I was hoping you could take a look and make sure this is still applicable or if there are changes that need to be made?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @mdbirnstiehl, thanks for the ping. This approach makes sense for existing deployments that don't use advanced setting mentioned above for historic reasons. Any newer deployments default to using the advanced setting and a manipulation of the log view saved object might therefore not have any effect. AFAIK in serverless that saved object type doesn't exist at all.
So if we want to include this approach I feel we should add lots of caveat warnings around it. Does that help?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @weltenwort, that does make sense. There's a discussion in the issue for this PR, where I think what we actually want to document is using the Advanced Settings saved object. I'm not sure if the process to use Advanced Settings saved object would be similar to this or if the process would differ.

Copy link
Member

@weltenwort weltenwort Jul 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The process would be similar, but the downside is that one couldn't just export/import this one setting selectively. It would always apply to all settings.

For users that use configuration management for Kibana deployments, there is the option to define overrides to individual settings in the yaml file like this:

uiSettings:
  overrides:
    'observability:logSources': ["logs-somewhere-else"]

But I'm not sure if that is an officially supported setting or not 🤔

1 change: 1 addition & 0 deletions solutions/toc.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -406,6 +406,7 @@ toc:
- file: observability/logs/categorize-log-entries.md
- file: observability/logs/inspect-log-anomalies.md
- file: observability/logs/run-pattern-analysis-on-log-data.md
- file: observability/logs/log-data-sources.md
- file: observability/logs/add-service-name-to-logs.md
- file: observability/logs/logs-index-template-reference.md
- file: observability/logs/streams/streams.md
Expand Down
Loading