Data-Plane HTTP Failure Status Response Proxy

[ndr-brt]

This proposal aims to enable the Data Plane to proxy failed status responses and response bodies back to the consumer
whenever the provider decides to do so for specific assets when using the HttpData-PULL transfer type.

Current Situation

The HttpData-PULL transfer type is handled differently from PUSH-based transfer types. Specifically, certain features
only work with the HttpData source type (HTTP-to-HTTP PULL):

• proxyBody
• proxyPath
• proxyMethod
• proxyQueryParams

These properties, when set in the source DataAddress, allow the provider to "grant power" to the consumer by proxying
the request body, path, method, and query parameters to the source. However, this can introduce security risks if not
used carefully.

Additionally, HttpData-PULL has another notable limitation: it does not support multiple Parts on the source side.
Only the first Part is returned to the consumer, while all others are ignored. As a result, this transfer type is
unsuitable when the source refers to multiple files (e.g., a bucket folder).

Proposal Options

1. Introduce proxyResponseStatus (or similar)

A new attribute would enable the provider to allow specific response codes to be forwarded to the consumer.

• Pros: Feature becomes available for all adopters.
• Cons: The Data Plane would function more like a pass-through proxy, which it is not intended to be.

2. Remove proxyXXX Attributes and Introduce Extensible Request/Response Mapping

Instead of predefined proxyXXX attributes, make request and response mapping extensible, allowing adopters to introduce
proxy features to their Data Plane without reimplementing everything from scratch.

• Pros:
  • Provides greater extensibility and flexibility.
  • Makes the Data Plane more generic and less tied to specific use cases.
• Cons:
  • Allows adopters to shape the Data Plane as they see fit (which could also be seen as a pro).

3. Do Nothing

Adopters requiring this functionality would need to implement their own Data Plane from scratch.

• Pros: No additional development effort required from us.
• Cons:
  • Adopters must implement and maintain their own solutions. (disadvantage for adopters)
  • Less control over what is provided to the community (which may not be a disadvantage for us but could be for adopters).
  • The current Data Plane implementation retains the proxyXXX features, making it somewhat similar to a pass-through
    proxy. This could cause misunderstandings—such as the assumption behind this feature request:
    If the Data Plane can proxy the request, why can’t it proxy the response?

[jimmarino]

There is also another option, which is to deprecate and eventually remove the mappings, and for people who want to have proxy behavior, have them write a custom controller extension for the data plane that does not use the DPF. Instead, this controller would proxy directly using an HTTP client.

This would be a simple solution and could likely be optimized. Trying to shoe-horn proxy behavior into the DPF will likely lead to special-case code that is much more complex than writing a simple controller.

We also need to consider this in the context of a longer-term strategy and plan for data planes. For example, the current Kafka solution is not suitable for high-volume environments. In addition, we should look at potentially moving the provisioning process to the data plane by using Data Plane Signalling.