Add rest-api-spec converter (#5295) (#6102)

* Add initial rest-api-spec converter

* Wrap JSON data with endpoint name

* Add headers

* Extract visibility

* Add debug logging for types

* Improve type handling

* Extract builting mappings

* Skip empty headers

* Add more mappings

* Add required field to bodies

* Support CommonCatQueryParameters

* Fix availability/stability

* Improve urls generation

* Serialize description only if required?

* Support more complex types

* Add spec and docs for new logs streams endpoints (#5258)

* Add spec and docs for new logs streams endpoints

* Add @codegen name to Acked responses

* Linting fix

* Update specification/streams/status/examples/200_response/GetStreamsStatusResponseExample1.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Correct feature flag

* Add doc-ids

* Fix doc-ids

* Update specification/_doc_ids/table.csv

Co-authored-by: Quentin Pradet <quentin.pradet@elastic.co>

* Update specification/_doc_ids/table.csv

Co-authored-by: Quentin Pradet <quentin.pradet@elastic.co>

* Add default timeouts

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Quentin Pradet <quentin.pradet@elastic.co>

* Update specification output

* Add common cat API parameters for unit rendering (#5298)

Describes the `?bytes=` and `?time=` parameters which are accepted by
all the `GET _cat/...` APIs.

* Edit the log streams endpoint docs (#5305)

* Add speficaiton for a new rescore type (#5308)

Based on PR elastic/elasticsearch#74274

* Update specification output

* Update rest-api-spec (#5294)

Co-authored-by: pquentin <42327+pquentin@users.noreply.github.com>

* Update rest-api-spec (#5311)

Co-authored-by: pquentin <42327+pquentin@users.noreply.github.com>

* Commit Cargo.lock changes

* Show unexpected union

* Remove '| string' in query parameters

* Always specify body.required

* Fix type for uint

* Return string for string | string[]

* Skip empty descriptions

* Remove redundant code comments

* Specify default_value for expand_wildcards

* Stop hardcoding expand_wildcards

* Improve union handling

* Hardcode "serialize": "bulk" endpoints

* Handle deprecated URLs parts

* Improve diff script

* Tweak converter

* Handle parameter visibility

* Handle required parameters

* Fix required parameters

* Fix experimental indices sample APIs

* Allow string[] for routing parameters

* Fix type of if_version in ingest.put_pipeline

* Fix type of search_type in rank_eval

* Fix type of CategoryId

* Mark find_structure column_names as string | string[]

* Allow empty URLs

* Handle number|string special cases

* Handle Fleet IndexName | IndexCase case

* Fail on unknown cases

* Improve diff

* Remvope file committed by mistake

* Add optional body to three ML APIs

* Fix ccs_minimize_roundtrips default value

* Fix description for parameter that does not accept comma-separated values

* Stop trying to document private API

* Remove parameter in find_structure API

* Fix find_stucture format enum

* Remove cruft

* Remove more cruft

* Update Rust edition to 2024 and refactor conditional statements

* Clarify main.rs

Print at the correct place and use crate prefixes

* Remove diplay() call

* Add TODOs

* Remove wrong doc_id from knn_search

* Add body to project.tags

* Run make spec-format-fix

* wip

* Switch to HttpMethod enum

* Use http::Method and implement TryFrom

* Use enums for stability and visibility

* Switch parameter types to enum

* Simplify BUILTING_MAPPINGS code

* Add top-level Cargo.lock change

* Add WASM integration for rest-api-spec generation

* Fix leftovers from previous commit

* Restrict WASM tracing to INFO

* Add _common.json generation

* Rename is_literal to get_literal

* Fix indices.get_settings expand_wildcards default

---------

Co-authored-by: Luke Whiting <luke.whiting@elastic.co>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
Co-authored-by: Mayya Sharipova <mayya.sharipova@elastic.co>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: pquentin <42327+pquentin@users.noreply.github.com>
Co-authored-by: Laurent Saint-Félix <laurent.saintfelix@elastic.co>
(cherry picked from commit 7b7e18a)

# Conflicts:
#	compiler-rs/compiler-wasm-lib/pkg/compiler_wasm_lib_bg.wasm
#	specification/inference/put_groq/PutGroqRequest.ts
#	specification/inference/put_nvidia/PutNvidiaRequest.ts
#	specification/streams/logs_disable/StreamsLogsDisableRequest.ts
#	specification/streams/logs_enable/StreamsLogsEnableRequest.ts

Author: pquentin

Makefile

@@ -51,6 +51,10 @@ clean-dep:	## Clean npm dependencies
 transform-expand-generics: ## Create a new schema with all generics expanded
 	@npm run transform-expand-generics --prefix compiler
 
+transform-to-rest-api-spec: ## Generate the REST API spec from the compiled schema
+	@rm -rf specification/_json_spec/*.json
+	@npm run transform-to-rest-api-spec --prefix compiler -- --schema output/schema/schema.json --output specification/_json_spec
+
 transform-to-openapi: ## Generate the OpenAPI definition from the compiled schema
 	@npm run transform-to-openapi -- --schema output/schema/schema.json --flavor stack --output output/openapi/elasticsearch-openapi.json
 	@npm run transform-to-openapi -- --schema output/schema/schema.json --flavor serverless --output output/openapi/elasticsearch-serverless-openapi.json

compiler-rs/Cargo.lock

@@ -4,6 +4,7 @@ members = [
     "clients_schema",
     "openapi_to_clients_schema",
     "clients_schema_to_openapi",
+    "clients_schema_to_rest_api_spec",
     "compiler-wasm-lib",
 ]
 
@@ -16,6 +17,7 @@ console_error_panic_hook = "0.1"
 convert_case = "0.6"
 derive_more = "1.0.0-beta.6"
 either_n = "0.2"
+http = "1"
 icu_segmenter = "1"
 indexmap = "2"
 itertools = "0.14"

compiler-rs/Cargo.toml

@@ -0,0 +1 @@
+rest-api-output

compiler-rs/clients_schema_to_rest_api_spec/.gitignore

@@ -0,0 +1,18 @@
+[package]
+name = "clients_schema_to_rest_api_spec"
+version = "0.1.0"
+edition = "2024"
+publish = false
+
+[dependencies]
+clients_schema = { path = "../clients_schema" }
+
+argh = { workspace = true }
+serde = { workspace = true, features = ["derive"] }
+serde_json = { workspace = true }
+anyhow = { workspace = true }
+indexmap = { workspace = true }
+
+tracing = { workspace = true }
+tracing-subscriber = { workspace = true }
+http = { workspace = true }