HTTPRequestAsync operator

HTTPRequestAsync operator proposal

Goal

A new operator that supports all HTTP 1.1 request methods (GET, POST, PUT, DELETE, etc.) and is designed to work well with HTTP(S) REST services in asynchronous manner.

Issues

• Current operators/native functions support HTTP requests in synchronous manner only.
• Streaming pipelines as well as other streams applications are required to perform HTTP/REST calls to external services (such as alerting or activating external systems) and yet, avoid back pressure.

Use cases

Functionality concepts

• The asynchronous call approach can make HTTP/REST requests possible without blocking the thread. The responses will be handled as callbacks, in asynchronous manner when they arrive.

• Operator makes an HTTP(S) request for each tuple arriving on its input port.

• The request is placed on a queue(s) and the calling thread is released

• Each queue has a working thread that sends out the HTTP request with a callback to handle the response, keeping the input tuple data.

• When the response arrives, it will call the callback with the response parameters (including the input tuple data)

• The operator will provide an output section to allow output manipulations

param
    url : "http://httpbin.org/" + action ;
    // url can be any expression (action is an attribute of an input tuple)
    body : '{"key" : "value"}' ;
    // body should be aligned with content type see next param
    contentType : json;
    // the proposed types are: json, xml, urlencoded,html, plain (text)
    connectionTimeout : 10.0;
    // set an overall timeout for HTTP requests
output
    body = Body(),        //response body as rstring
    headers = Headers(),  //response headers as map<rstring,rstring>
    message = Message(),  //response message as rstring
    status = Status();    //response HTTP code as uint16

• All output tuple's attributes will be auto completed by the same input tuple attribute names
• Automatic follow redirection will be supported