Skip to content

Add ml.get_transform_stats issues in response #4065

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 26, 2025
Merged

Add ml.get_transform_stats issues in response #4065

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e08cfc5
Add ml.get_transform_stats issues in response
pquentin Mar 24, 2025
dd179e4
Add missing DateTime fields
pquentin Mar 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion output/openapi/elasticsearch-openapi.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 10 additions & 1 deletion output/openapi/elasticsearch-serverless-openapi.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

47 changes: 40 additions & 7 deletions output/schema/schema-serverless.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

47 changes: 40 additions & 7 deletions output/schema/schema.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 4 additions & 1 deletion output/typescript/types.ts
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion specification/transform/get_transform/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ import {
SyncContainer
} from '@transform/_types/Transform'
import { Id, Metadata, VersionString } from '@_types/common'
import { Duration, EpochTime, UnitMillis } from '@_types/Time'
import { DateTime, Duration, EpochTime, UnitMillis } from '@_types/Time'

export class TransformSummary {
/**
Expand All @@ -37,6 +37,7 @@ export class TransformSummary {
authorization?: TransformAuthorization
/** The time the transform was created. */
create_time?: EpochTime<UnitMillis>
create_time_string?: DateTime
/** Free text description of the transform. */
description?: string
/** The destination for the transform. */
Expand Down
4 changes: 3 additions & 1 deletion specification/transform/get_transform_stats/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ export class TransformHealthIssue {
count: integer
/** The timestamp this issue occurred for for the first time */
first_occurrence?: EpochTime<UnitMillis>
Copy link
Contributor

@l-trotta l-trotta Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

around the spec I saw various instance where we also map the human readable field, so both first_occurrence and first_occurrence_string

Copy link
Member Author

@pquentin pquentin Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. We can indeed get this when passing ?human, here are samples outputs for get_transform

{
  "count": 1,
  "transforms": [
    {
      "id": "test",
      "authorization": {
        "roles": [
          "superuser"
        ]
      },
      "version": "10.0.0",
      "create_time_string": "2025-03-26T11:25:45.642Z",
      "create_time": 1742988345642,
      "source": {
        "index": [
          "kibana_sample_data_ecommerce"
        ],
        "query": {
          "match_all": {}
        }
      },
      "dest": {
        "index": "test"
      },
      "pivot": {
        "group_by": {
          "category": {
            "terms": {
              "field": "category.keyword"
            }
          }
        },
        "aggregations": {
          "customer_id.cardinality": {
            "cardinality": {
              "field": "customer_id"
            }
          }
        }
      },
      "settings": {}
    }
  ]
}

and get_transform_stats:

{
  "count": 1,
  "transforms": [
    {
      "id": "test",
      "state": "stopped",
      "stats": {
        "pages_processed": 2,
        "documents_processed": 7409,
        "documents_indexed": 6,
        "documents_deleted": 0,
        "trigger_count": 1,
        "index_time_in_ms": 2,
        "index_total": 1,
        "index_failures": 0,
        "search_time_in_ms": 3,
        "search_total": 2,
        "search_failures": 0,
        "processing_time_in_ms": 0,
        "processing_total": 2,
        "delete_time_in_ms": 0,
        "exponential_avg_checkpoint_duration_ms": 19,
        "exponential_avg_documents_indexed": 6,
        "exponential_avg_documents_processed": 7409
      },
      "checkpointing": {
        "last": {
          "checkpoint": 1,
          "timestamp": "2025-03-26T11:25:46.849Z",
          "timestamp_millis": 1742988346849
        },
        "changes_last_detected_at_string": "2025-03-26T11:25:46.849Z",
        "changes_last_detected_at": 1742988346849
      },
      "health": {
        "status": "green"
      }
    }
  ]
}

I fixed the ones I saw here.

first_occurence_string?: DateTime
}

export class TransformProgress {
Expand Down Expand Up @@ -100,9 +101,10 @@ export class CheckpointStats {

export class Checkpointing {
changes_last_detected_at?: long
changes_last_detected_at_date_time?: DateTime
changes_last_detected_at_string?: DateTime
last: CheckpointStats
next?: CheckpointStats
operations_behind?: long
last_search_time?: long
last_search_time_string?: DateTime
}
Comment on lines 102 to 110
Copy link
Member Author

@pquentin pquentin Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Defined in https://github.com/elastic/elasticsearch/blob/6503c1b94bf077a9fc152019932a79c4e4f42d40/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/transform/transforms/TransformCheckpointingInfo.java#L209-L230