http.POST streaming the payload

[johnofleek]

I have a large payload I'd like to http.POST

int httpCode = http.POST(binaryPayload, binaryPayload_size);
Fails above a 0x4000 byte (16384) size payload

Edit:
I think that the underlying POST code limits the payload sent to IP to 128 bytes max ? Therefore I'm guessing that my issue isn't related to the payload size but is cause by something else ? From HTTPClient.cpp showing I think the code that sends 128 bytes at a time into TCP/IP

Edit2
Short story - enabling debugging didn't help so far

I did hack this bit of WiFiClientSecureBearSSL.cpp because I couldn't quickly work out a proper solution - but the issues aren't in that part anyway

[maxgerhardt]

The sendAll() function transports the data in chunks of maximum size 128 bytes to the "client". I think 16K payload limitation might come from the used firmware in the WiFi chip, the Cypress CW.. something on the Pico W (I assume that's what you're using)., that's where the payload will be written to in the most immediate path.

[earlephilhower]

It looks like the code is actually doing HTTPS and not HTTP. In that case, 16K is the TLS record size, meaning the core will buffer up 16K of writes (at 128b a pop) into that buffer, encrypt, and then transmit it using TCP. There may be some HTTP-specific things (i.e. you might need to use HTTP chunked encoding) when multiple TCP packets are sent in via a POST.

The only place I think HTTPClient can return that error is here:

arduino-pico/libraries/HTTPClient/src/HTTPClient.cpp

Lines 598 to 601 in 67a07ed

	// transfer all of it, with send-timeout
	if (size && StreamConstPtr(payload, size).sendAll(_client()) != size) {
	return returnError(HTTPC_ERROR_SEND_PAYLOAD_FAILED);
	}

The StreamConstPtr is a hack I pulled in to allow the ESP8266 HTTPClient to run (the 8266 rewrote the whole Stream class to be lower memory but incompatible with plain Arduino).

arduino-pico/libraries/HTTPClient/src/HTTPClient.cpp

Lines 85 to 101 in 67a07ed

	size_t sendAll(Client *dst) {
	uint32_t start = millis();
	size_t sent = 0;
	while ((sent < _size) && (millis() - start < 5000)) {
	size_t towrite = std::min((size_t)128, _size - sent);
	auto wrote = dst->write(_payload, towrite);
	if (wrote <= 0) {
	break;
	}
	sent += wrote;
	_payload += wrote;
	}
	return sent;
	}
	const uint8_t *_payload;
	size_t _size;
	};

So this could fail if a) 5s have passed since the start of the transmission, or b) the WiFiClientSecure::write returns 0.

It would be instructive if you could make a small MCVE that posts against some public webserver or a simple python one to repro and get to the bottom of it.