Large request bodies cannot be proxied

[pebo]

When sending a large (>100kb) body the proxy fails:

Assuming the proxy is running on localhost:

printf '==========%.0s' {1..100000} > dummy.json
curl -X POST http://localhost:8200/index1/typ1 -d @dummy.json

=> body parser error:

Error: request entity too large

However increasing the bodyparser limit causes other issues:

app.use(bodyParser.raw({
    limit: '1mb', 
    type: function() { return true; }}));

=> Error: Can't set headers after they are sent.
at ClientRequest.OutgoingMessage.setHeader (_http_outgoing.js:357:11)

Is there a workaround for this issue?

[boboTjones]

I ran up against this, too. I ran the requests through Burp proxy and the UI makes a GET request for the index fields and then returns them back to ESS in a POST body. I didn't have time to figure out why the response to the initial GET is handled fine but the subsequent POST is rejected. But I can see that the UI is sending a >100kb body to that subsequent POST request. As a monkey patch, I edited node_modules/body-parser/lib/types/raw.js on line 37 to up the body limit size to 500kb:

  var limit = typeof opts.limit !== 'number'
    ? bytes.parse(opts.limit || '500kb')
    : opts.limit

[cgballance]

This issue is in http-proxy. the on.( 'proxyReq') in index.js is performed too late. So, the lib/http-proxy/common.js should have this code in common.setupOutgoing(...) before the return:
if( options.beforeProxyRequest && typeof options.beforeProxyRequest === 'function' ) {
options.beforeProxyRequest( outgoing, options, req ) ;
}
This is a fix suggested by Rick Kilgore in a node-http-proxy.....which used http-proxy.
Then, change index.js's startProcess function, adding to the httpProxy.createProxyServer options
beforeProxyRequest: beforeProxyRequest

Finally, translate what used to be in proxy.on('proxyReq'...) into the function:
function beforeProxyRequest( outgoing, options, req ) {
var endpoint = new AWS.Endpoint(ENDPOINT);
var request = new AWS.HttpRequest(endpoint);
request.method = outgoing.method;
request.path = outgoing.path;
request.region = REGION;

if (Buffer.isBuffer(req.body)) request.body = req.body;
if (!request.headers) request.headers = {};

request.headers['presigned-expires'] = false;
request.headers['Host'] = TARGETHOST;

var signer = new AWS.Signers.V4( request, 'es' );
signer.addAuthorization( credentials, new Date() );

outgoing.headers.host = request.headers['Host'] ;
outgoing.headers['X-Amz-Date'] = request.headers['X-Amz-Date'];
outgoing.headers['Authorization'] = request.headers['Authorization'];
if (request.headers['x-amz-security-token']) outgoing.headers['x-amz-security-token'] = request.headers['x-amz-security-token'];
if (req.url.match(/\.(css|js|img|font)/)) {
        outgoing.headers['Cache-Control'] = 'public, max-age=86400';
}

}

I also had to change the way the Host name was sent, but that's in another request.
var TARGETHOST ;
anyway, i now have a fully functioning proxy.

[RemiTorracinta]

Also got this issue using kibana:

PayloadTooLargeError: request entity too large
    at readStream (/usr/local/lib/node_modules/aws-es-kibana/node_modules/raw-body/index.js:155:17)
    at getRawBody (/usr/local/lib/node_modules/aws-es-kibana/node_modules/raw-body/index.js:108:12)
    at read (/usr/local/lib/node_modules/aws-es-kibana/node_modules/body-parser/lib/read.js:77:3)
    at rawParser (/usr/local/lib/node_modules/aws-es-kibana/node_modules/body-parser/lib/types/raw.js:81:5)
    at Layer.handle [as handle_request] (/usr/local/lib/node_modules/aws-es-kibana/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/usr/local/lib/node_modules/aws-es-kibana/node_modules/express/lib/router/index.js:317:13)
    at /usr/local/lib/node_modules/aws-es-kibana/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/usr/local/lib/node_modules/aws-es-kibana/node_modules/express/lib/router/index.js:335:12)
    at next (/usr/local/lib/node_modules/aws-es-kibana/node_modules/express/lib/router/index.js:275:10)
    at compression (/usr/local/lib/node_modules/aws-es-kibana/node_modules/compression/index.js:220:5)

Same requests work when I don't use proxy

[nsaud01]

@cgballance Do you have an example of those changes or a repo where it's been implemented?

[cgballance]

Sorry, had it on a work laptop that i gave back to my client. If you can't decipher, i'll set up es and redo. Let me know