Datastream stats return 0 as maximum_timestamp

[nchaulet]

Elasticsearch Version

9.3.0

Installed Plugins

No response

Java Version

bundled

OS Version

Problem Description

Description

In Fleet we use the _data_stream/stats and retrieve the maximum_timestamp field to display activity to the user, it seems in the latest daily SNAPSHOT that field is always 0

Steps to Reproduce

POST logs-datastreams.test_logs-testwithoutfinalpipeline/_doc
{
          "@timestamp": "2025-10-27T14:51:00.564Z",
          "data_stream": {
            "dataset": "datastreams.test_logs",
            "namespace": "testwithoutfinalpipeline",
            "type": "logs"
          },
          "message": "test"
        }


GET _data_stream/logs-datastreams.test_logs-testwithoutfinalpipeline/_stats

Logs (if relevant)

No response

[0xShubhamSolanki]

Hi @nchaulet,
Reproduced on 9.3.0 snapshot: Fresh @timestamp doc shows maximum_timestamp: 0 in _stats until _refresh. Ties to committed-only searcher missing uncommitted docs; aligns with failing Kibana test (kibana#240001) and frozen index opts (PR #117953).

I'd like to fix this! Plan:

1.Switch DataStreamsStatsTransportAction to realtime searcher for accurate max without refresh.
2.Add tests for unrefreshed cases + Fleet integration.
3.Draft PR in next 2-3 days.

I haved forked the repo and started working on it . Can you assign this to me and lmk if there's a preferred code path or reviewer?

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[nielsbauman]

Hi @nchaulet, thanks for reporting this! In your screenshot, I see

"_shards" : {
  "total" : 2,
  "successful" : 1,
  "failed" : 0
}

We've seen this exact issue (maximum_timestamp of 0) in a test failure (#118217). I thought we addressed it in #129813 by waiting for all shards to be green, but looking at the failure history of that test, we're still seeing cases occasionally where the maximum_timestamp is 0 even though all shards came back successfully.

I would recommend that you start with ensuring all shards are green before you call the stats API.

In the meantime, I'll forward this issue to the Storage Engine team, as they own the shard engine code.

@0xShubhamSolanki I'll leave it up to the Storage Engine team to make any recommendations regarding a possible fix.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[nchaulet]

We've seen this exact issue (maximum_timestamp of 0) in a test failure (#118217). I thought we addressed it in #129813 by waiting for all shards to be green, but looking at the failure history of that test, we're still seeing cases occasionally where the maximum_timestamp is 0 even though all shards came back successfully.
I would recommend that you start with ensuring all shards are green before you call the stats API.

Just tested with only one shard and it seems we have the same issue

[PeteGillinElastic]

I believe that this issue is specific to logsdb. At least in my testing, this behaves as expected:

PUT _index_template/my-index-template
{
  "index_patterns": ["ds*"],
  "data_stream": { }
}

POST ds1/_doc
{
  "@timestamp": "2025-10-27T14:51:00.564Z",
  "data_stream": {
    "dataset": "datastreams.test_logs",
    "namespace": "testwithoutfinalpipeline",
    "type": "logs"
  },
  "message": "test"
}

GET _data_stream/ds1/_stats

while this gives "maximum_timestamp" : 0:

PUT _index_template/my-logsdb-index-template
{
  "index_patterns": ["lds*"],
  "data_stream": { },
  "template": {
     "settings": {
        "index.mode": "logsdb"
     }
  }
}

POST lds1/_doc
{
  "@timestamp": "2025-10-27T14:51:00.564Z",
  "data_stream": {
    "dataset": "datastreams.test_logs",
    "namespace": "testwithoutfinalpipeline",
    "type": "logs"
  },
  "message": "test"
}

GET _data_stream/lds1/_stats

[PeteGillinElastic]

(Neither waiting nor doing POST lds1/_refresh helped.)