MultiVerb, content-type negotiation, semantics and client behavior

[gdeest]

I am getting confused by the semantics of endpoint-level and response-level content-types in MultiVerb, especially regarding client behavior, and I wonder whether something needs clarification here.

Context

Let me try to lay out my thoughts in a somewhat logical manner.

1. Respond responses

The basic usage of MultiVerb is to declare content-types at the MultiVerb-level and use Respond to declare all possible responses. Respond does not accept a content-type parameter:

data Respond (s :: Nat) (description :: Symbol) (a :: Type)

which is why in the server implementation we require the a type to be renderable as all possibly requested content-types:

instance
  (AllMimeRender cs a, KnownStatus s)
  => ResponseRender cs (Respond s desc a)

This is the easy case: when the response list only contains Respond responses, we can always honor the client-requested content-type (assuming it is a member of the cs list).

2. RespondAs responses

In contrast, the RespondAs response type constitutes an escape hatch to the strict content-negotiation mechanism and hardcodes a content-type that may not even be a member of the MultiVerb-level list:

data RespondAs responseContentType (s :: Nat) (description :: Symbol) (a :: Type)

For example, the ResponseRender instance in the server implementation totally disregards the endpoint-level content-type list (see the lack of AllMimeRender cs a here) and only looks at the response-level ct parameter:

instance
  ( KnownStatus s
  , MimeRender ct a
  )
  => ResponseRender cs (RespondAs (ct :: Type) s desc a)

My intuition is that RespondAs is essentially intended to be used for error cases. E.g., success paths may support JSON but some errors just have a plain text body.

3. RespondStreaming and content negotiation

RespondStreaming is similar to RespondAs in that it specifies its own content-type. I think this might be problematic, as streaming responses are typically not error-cases, and we currently cannot ensure that these endpoints correctly participate in the content-negotiation dance.

Aside: The problem becomes a tiny bit more complex if we try to factor in framing (not supported by mainstream RespondStreaming, but I actually have working code for this): should framing depend on the selected content-type? For example NewlineFraming might work for JSON but you might want NetstringFraming for some other binary format. How should those two features interact?

4. Client-side unreachable branches

Finally, the servant-client MultiVerb implementation does not interact well with content-types that are not declared in the MultiVerb list. Consider the following API:

type UserResponses =
  '[ Respond 200 "User found" User
   , RespondAs PlainText 404 "Not found" Text
   ]

data UserResult = UserFound User | UserNotFound Text
  deriving stock (Generic)
  deriving (AsUnion UserResponses) via GenericAsUnion UserResponses UserResult

type UserApi = "user" :> Capture "id" Int
  :> MultiVerb 'GET '[JSON] UserResponses UserResult

The client builds the Accept header solely from the endpoint-level cs list:

instance ... => HasClient m (MultiVerb method cs as r) where
  clientWithRoute _ _ req =
    runMultiVerbRequest ...
      req { ...
          , requestAccept = Seq.fromList (clientAcceptList (Proxy @cs))
          }

So the client sends Accept: application/json. When the server returns a 404 with Content-Type: text/plain, the client validates the response content-type against what it originally requested:

unless (any (M.matches c) accept) $
  throwClientError $ UnsupportedContentType c response

Since text/plain is not in [application/json], the client throws UnsupportedContentType before even attempting to parse. The UserResult type promises two branches, but UserNotFound is effectively unreachable from the client and replaced by an exception. This makes some branches of our union "write-only" in a sense: the server can use them to specify how to answer, but the client does not get a similarly shaped response.

One way to fix this could be to have the client gather content-types from all response branches, not just from cs. This would automatically make all branches reachable without requiring API changes. But is this truly what we want ?

Questions

• What are the intended semantics of the MultiVerb content-type list ?
• What are the intended semantics of the content-types specified at the response level (RespondAs, RespondStreaming) ?
• How should the two interact (if at all) ?
• What should be the client behavior in the situation highlighted above ?

[tchoutri]

cc @pcapriotti @fisx