[8.x] Add a basic deprecation warning that the JSON format is changing in v9 (#116330)

[thecoop]

Backport #116330 to 8.17

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[thecoop]

I'm not sure if this needs to be a critical (as things are definitely changing), or a warning, as the user can't do anything about it until they upgrade.

Do we need to include a link to more detailed docs (which don't really exist, apart from the breaking changelog?)

[DaveCTurner]

the user can't do anything about it until they upgrade.

That seems bad to me. Certainly this can't be critical because critical means "must be addressed before you upgrade". But could we have a way for users to opt-in to the new format ahead of time? Should we fall back to the old format in v9 if the user requests RestApiVersion.V_8?

[thecoop]

This is the behaviour as agreed in the BwC discussion...

[DaveCTurner]

Ok I see, I wasn't in that discussion 😄

Can we tell folks to opt-in to the future behaviour now by removing http.detailed_errors.enabled from their node settings?

[thecoop]

That will cause detailed error information to be output, which is probably not what they want (as they set the option in the first place)

[DaveCTurner]

Sure but we've got to do something here, we can't just expect folks to upgrade and suck it up if their clients can't handle the new format caused by the upgrade. Either we need to help them fix their clients before upgrading or else we need to preserve the v8 error format in v9.

[thecoop]

That's what the decision of the BwC committee was - there's very little use of this option, so any attempt at pro-active mitigation is disproportionate. You can re-open the discussion with BwC if you want?

[DaveCTurner]

That's what the decision of the BwC committee was

That's not my reading of the decision. There was no mention of this change affecting clients that requested v8 compatibility in v9, and our default position for that case would be that such clients see no change in behaviour. Would you ask the BwC committee to clarify whether they intended to make this change in v8-compat mode too?

[rjernst]

I agree with Dave's concern. The point of rest api compatibility is to bridge these exact gaps, changes across majors. I think we should maintain the existing format when the v8 header is used.

[thecoop]

I've added BwC API outputs to #90529, and tweaked the warning here accordingly

[DaveCTurner]

LGTM except that I'd rather we followed the usual process of merging this into main, backporting it to 8.x, and then merging the remnants of #90529 into main. As it is, it's overly hard to review the difference between this PR and #90529.

[thecoop]

Hmm, ok, yes we could do it that way

[thecoop]

See #116330

[thecoop]

@elasticmachine update branch

[thecoop]

@elasticmachine update branch

[mosche]

LGTM, just a small suggestion.