Generate Elasticsearch rest-api-spec from the Elasticsearch specification

[pquentin]

To this day, https://github.com/elastic/elasticsearch/tree/main/rest-api-spec is maintained by hand by Elasticsearch developers. It currently serves two purposes:

• Generate queries in YAML tests
• Generate the Ruby 8.x / PHP 8.x / PHP 9.x clients.

However, we don't want to maintain two specifications. For the most part, the Elasticsearch specification is a superset of the rest-api-spec. We have been working hard for the past year to remove differences:

So we should now strive to generate rest-api-spec for the others.

Details

• Some APIs are not in the Elasticsearch specification, either because they're still experimental and behind a feature flag, or just internal.
• They are many differences in default values. They are only used for docs, but it would still be valuable to not lose this information.
• Some information is implicit from rest-api-spec. For example, async_search.submit has "body": {} but we would generate the more explicit "body": {"required": false}
• Mapping complex types to string/list/object isn't that easy
• Some availability/visibility information is out-of-date in the rest-api-spec
• Documentation URLs are out of date

The above is just an initial list, I only looked at a small percentage of the 560+ APIs we have.

[pquentin]

Visibility/Stability PRs:

• rest-api-spec: update connector visibility/stability elasticsearch#133814
• Mark user profile APIs as public elasticsearch#133819
• Mark migrate_reindex APIs as public and stable elasticsearch#133912
• rest-api-spec: mark search application as beta elasticsearch#133914
• rest-api-spec: mark search_mvt as stable elasticsearch#133915
• rest-api-spec: fix query_rules.test stability elasticsearch#133981
• Mark cloud-only APIs as private #5235