RFC 9457 error responses

[chrisguidry]

I just came across RFC 9457: Problem Details for HTTP APIs which describes a standardized way to report errors from REST APIs. The RFC describes a mime type "application/problem+json" and a standard set of JSON fields, like type, detail, and instance. While this RFC is still just a Proposed Standard, it would be more widely adopted than our ad-hoc error handling approach.

When I audited Prefect and Prefect Cloud, I found that we were kinda-sorta abiding by this convention, just mostly inherited from FastAPI. Worse, both systems differ in their error reporting approaches depending on the endpoints, what type of exceptions are raised, and which handler is actually catching errors and formatting responses. The two systems also differ from one another in some instances.

I propose that we unify these, while still preserving all of the extant error messaging fields for compatibility. Preserving compatibility should be straightforward, at the expense of some duplicated keys; the RFC is fine with additional fields (https://www.rfc-editor.org/rfc/rfc9457.html#section-3.1).

Migration approach:

• Phase 1: Update all error responses in PrefectHQ/prefect to include the new fields as appropriate, while maintaining any currently returned fields. Only return the Content-Type: application/problem+json if it was explicitly requested via an Accept: application/json, application/problem+json header. Otherwise, keep the current application/json for all responses.
• Phase 2: Update all error responses in Prefect Cloud according to the same rules.
• Phase 3: Include error responses
• Phase 4: Update the PrefectHQ/prefect HTTP client to Accept: application/json, application/problem+json
• Phase 5: update all error logging to users to use the additional fields when present to offer more information where appropriate

I see no need to ever drop the older fields from our responses, but perhaps with a future major version break we could do that. We could also conditionally decide to return the stricter subset to clients that are asking for application/problem+json in their Accept headers.