Skip to content

Conversation

dnhatn
Copy link
Member

@dnhatn dnhatn commented Oct 31, 2024

This PR adds telemetry for logsdb. However, this change only tracks the count of indices using logsdb and those that use synthetic source. Additional stats, such as shard, indexing, and search stats, will be added in a follow-up, as they require reaching out to data nodes.

Copy link
Contributor

Documentation preview:

@dnhatn dnhatn changed the title Logsdb usage Add logsdb telemetry Oct 31, 2024
@dnhatn dnhatn marked this pull request as ready for review October 31, 2024 04:32
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-storage-engine (Team:StorageEngine)

@elasticsearchmachine
Copy link
Collaborator

Hi @dnhatn, I've created a changelog YAML for you.

@dnhatn dnhatn added the auto-backport Automatically create backport pull requests when merged label Oct 31, 2024
Copy link
Member

@martijnvg martijnvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

I think we also need to update the mapping the telemetry repo. Similar to this elastic/telemetry#3415

}

@Override
protected void masterOperation(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if this is the best place for this, but can we maybe in a follow up also track the number of indices without index mode that use synthetic source, the number of tsdb indices and how many of those tsdb indices use synthetic source? If you think this is a good idea. Maybe we should in this change rename the indices_count and indices_with_synthetic_source fields in the response to be something like logsdb_index_count, logsdb_with_synthetic_source_index_count?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I considered this as well, but these fields don't seem to fit within logsdb. I think we'll need to add two separate usages: one for time_series and another for source_mode. I’ll add them in follow-up PRs.

{
	"logsdb": {
		"enabled": true,
		"available": true,
		"indices_count": 1,
		"indices_with_synthetic_source": 1
	}
}

ActionListener<XPackUsageFeatureResponse> listener
) {
int numIndices = 0;
int numSyntheticSources = 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe rename to numLogsdbIndices and numLogsdbIndicesWithSynthticSource?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed 5dc58e6

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@dnhatn
Copy link
Member Author

dnhatn commented Nov 1, 2024

Thanks Martijn!

@dnhatn dnhatn merged commit 6d4e11d into elastic:main Nov 1, 2024
16 checks passed
@dnhatn dnhatn deleted the logsdb-usage branch November 1, 2024 15:25
@elasticsearchmachine
Copy link
Collaborator

💚 Backport successful

Status Branch Result
8.x

dnhatn added a commit to dnhatn/elasticsearch that referenced this pull request Nov 1, 2024
This PR adds telemetry for logsdb. However, this change only tracks the 
count of indices using logsdb and those that use synthetic source. 
Additional stats, such as shard, indexing, and search stats, will be
added in a follow-up, as they require reaching out to data nodes.
elasticsearchmachine pushed a commit that referenced this pull request Nov 1, 2024
This PR adds telemetry for logsdb. However, this change only tracks the 
count of indices using logsdb and those that use synthetic source. 
Additional stats, such as shard, indexing, and search stats, will be
added in a follow-up, as they require reaching out to data nodes.
jfreden pushed a commit to jfreden/elasticsearch that referenced this pull request Nov 4, 2024
This PR adds telemetry for logsdb. However, this change only tracks the 
count of indices using logsdb and those that use synthetic source. 
Additional stats, such as shard, indexing, and search stats, will be
added in a follow-up, as they require reaching out to data nodes.
dnhatn added a commit that referenced this pull request Nov 5, 2024
Follow-up on #115994 to add telemetry for the total number of documents 
and size in bytes of logsdb indices.

Relates #115994
dnhatn added a commit to dnhatn/elasticsearch that referenced this pull request Nov 5, 2024
Follow-up on elastic#115994 to add telemetry for the total number of documents 
and size in bytes of logsdb indices.

Relates elastic#115994
dnhatn added a commit to dnhatn/elasticsearch that referenced this pull request Nov 5, 2024
Follow-up on elastic#115994 to add telemetry for the total number of documents
and size in bytes of logsdb indices.

Relates elastic#115994
dnhatn added a commit that referenced this pull request Nov 7, 2024
Follow-up on #115994 to add telemetry for the total number of documents
and size in bytes of logsdb indices.

Relates #115994
jozala pushed a commit that referenced this pull request Nov 13, 2024
Follow-up on #115994 to add telemetry for the total number of documents 
and size in bytes of logsdb indices.

Relates #115994
alexey-ivanov-es pushed a commit to alexey-ivanov-es/elasticsearch that referenced this pull request Nov 28, 2024
Follow-up on elastic#115994 to add telemetry for the total number of documents 
and size in bytes of logsdb indices.

Relates elastic#115994
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-backport Automatically create backport pull requests when merged >enhancement :StorageEngine/Logs You know, for Logs Team:StorageEngine v8.17.0 v9.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants