SSL - A buffer is too small to receive or write a message

[SheetLightning]

I recently used this HTTPSredirect library in one of my projects to access my Google calendar. I have it working on an ESP8266, but wanted to port it to an ESP32 because the project will need the additional GPIO pins. Unfortunately I ran into a problem with Google not returning any events from my calendar. With ESP32 verbose debugging turned on, I was able to capture the following error:

Connected to Google
[ 43293][V][ssl_client.cpp:369] send_ssl_data(): Writing HTTP request with 150 bytes...
[ 43293][V][ssl_client.cpp:374] send_ssl_data(): Handling error -27136
[ 43299][E][ssl_client.cpp:37] _handle_error(): [send_ssl_data():375]: (-27136) SSL - A buffer is too small to receive or write a message
[ 43313][V][ssl_client.cpp:324] stop_ssl_socket(): Cleaning SSL connection.
gclient->GET(url, googleHost);
Performed GET

I am using:

• Adruino IDE version 1.8.19
• ESP32 board library version 2.0.4
• ESP32 development board WROOM-32, Devkit1

The GET request is failing and not returning any data. I presume that the HTTPSredirect library somehow calls the Esspressif ssl_client module but can't see anything in the SDK except a header file. I also can't see any buffer size being set in this library as it uses a String type for the data buffer.

Can I override the size of the SSL buffer please? If so, where?
I have not been able to find anything regarding this on the Expressif Github.

The code is quite straightforward and is based on this project by Andreas Spiess.

https://www.youtube.com/watch?v=sm1-l5-z3ag

I have attached a minimal example.

The error occurs when running the gclient->GET(url, googleHost); statement in my sketch. I have attached a minimal example. See the getCalendar() function.

I should probably point out that I made a minor change to the program to allow it to insert the "ESP32" string in the User-agent line in the GET and POST requests in place of "ESP8266". I don't believe this makes any functional difference, except to indicate to the Google API that the request came from an ESP32 rather than an ESP8266.

Minimal code example:

Google-Minimal.zip

[github-actions]

Welcome to HTTPSRedirect! Please provide enough info to debug the issue.

[SheetLightning]

In case anyone is still interested in this, I think I have made some progress in identifying the issue although it has been a bit long- winded.

Since httpsRedirect is a wrapper for the underlying ESP WiFiClientSecure library, I reconstructed the minimal version of my project without using httpsRedirect and using only direct calls to WiFiClientSecure. This has actually been quite useful as it has allowed me to see what was otherwise abstracted by the library.

It turns out that the ESP32 appears to be rather unreliable when making WiFi connections. I don't know whether this is an issue with my particular ESP Devkit modules, or ESP32s in general, however I have tested with two different ESP32 dev modules with the same result. Once is a recent purchase, the other I have had lying around for some time. What I found is that the initial connection using the WiFiSecureClient.print() method fails randomly. In addition, the reported SSL error usually occurs on the first attempt to connect with the re-directed URL, but succeeds on the second or third attempt. The httpsRedirect library tries to connect with the re-directed URL only once. Because the first attempt at connection almost always fails, connection to the re-directed URL fails and ultimately the .GET() call fails. Perhaps it would be useful to build in a number of re-try attempts?

I have not observed this problem on the ESP8266, only on the ESP32.

It seems to be possible to work around the problem by having up to 3 re-try attempts for both the initial URL and the returned re-directed URL. Its not ideal because it introduces some delay and uncertainty but at least the odds are that it will succeed by the 3rd attempt.

[SheetLightning]

PS, you might also want to correct this which I found in void HTTPSRedirect::getResponseStatus(void){}, lines 341 and 342 :

  unsigned int pos = -1;
  unsigned int pos2 = -1;

The variables are declared as unsigned, so can't have a negative value. Since the compiler does not complain, that would probably set the value to 65534. I don't think it would have much effect on the result from the function however.

[BasDeBass]

@SheetLightning I'm running into the same problem. Could you share code of your workaround? I have tried implementing your workaround myself, but it seems i'm not great at diving into libraries...

[bmacdoug]

Re: "ESP appears to be rather unreliable when making WiFi connections."
I have noted that the statement WiFi.mode(WIFI_STA) is necessary in order to ensure the ESP32 starts as a WiFi station rather than a WiFi repeater. Without this statement I get unreliable and quirky wifi performance. I am using Lolin D32 ESP32 boards.

[SheetLightning]

Thank you for the responses.

BasDeBass, I have appended my test code, which can be considered an updated minimal test case.
main.cpp.txt
This attempts 3 re-tries, but even then sometimes fails. Its was written in vscode+PlatfformIO using the Arduino architecture so should work in Arduino IDE, but will need to be re-named and the extension changed to .ino. It will also need to be placed in a directory of the same name as the file as per Arduino requirements. You will have to add macros as follows:

#define SSID_NAME myssid
#define SSID_WPAKEY mywpakey
#define GSCRIPT_HOST some.host.name
#define GSCRIPT_URL /some/url

I am connecting to Google Scripting API, have also tried with static sites such as scambusters.org. Obviously there is no re-direct in this case, but the same erratic behaviour can be observed during the connection attempt.

bmacdoug, thank you for that information. That is interesting. Since I am using the ESP32 as a client connecting to an existing WiFi router only, it seems odd that it should require to be started as a WiFi AP. I really would prefer that the ESP did not appear as an AP, however I will test your suggestion and see what happens.

[SheetLightning]

Just an update. I tried adding:

WiFi.mode(WIFI_STA);
delay(200);

to the code, but unfortunately this made no difference. I am still getting random errors including the one in the title of this issue. I tried with and without and using progressively longer delay intervals up to 1000 but this had no effect. I also tried adding similar delays between the first request and the re-directed request, but still no change.

[bmacdoug]

Hi, I was suggesting that WiFi.mode(WIFI_STA) starts the WiFi process as a client not as a router. I am sorry it had no effect. It has been suggested to me as a good idea in any case.

[SheetLightning]

Fair enough. After looking into it, I think that the WiFi.mode(STA) call just sets the WiFi config to station mode but doesn't actually start an AP. Bringing up an AP would require a WiFi.softAP() call with appropriate SSID and password parameters. It does make sense to start from a known state so adding the WiFi.mode() call was certainly worth a try. It is a shame that it had no effect though.

[lucasromeiro]

I recently used this HTTPSredirect library in one of my projects to access my Google calendar. I have it working on an ESP8266, but wanted to port it to an ESP32 because the project will need the additional GPIO pins. Unfortunately I ran into a problem with Google not returning any events from my calendar. With ESP32 verbose debugging turned on, I was able to capture the following error:

Connected to Google [ 43293][V][ssl_client.cpp:369] send_ssl_data(): Writing HTTP request with 150 bytes... [ 43293][V][ssl_client.cpp:374] send_ssl_data(): Handling error -27136 [ 43299][E][ssl_client.cpp:37] _handle_error(): [send_ssl_data():375]: (-27136) SSL - A buffer is too small to receive or write a message [ 43313][V][ssl_client.cpp:324] stop_ssl_socket(): Cleaning SSL connection. gclient->GET(url, googleHost); Performed GET

I am using:

• Adruino IDE version 1.8.19
• ESP32 board library version 2.0.4
• ESP32 development board WROOM-32, Devkit1

The GET request is failing and not returning any data. I presume that the HTTPSredirect library somehow calls the Esspressif ssl_client module but can't see anything in the SDK except a header file. I also can't see any buffer size being set in this library as it uses a String type for the data buffer.

Can I override the size of the SSL buffer please? If so, where? I have not been able to find anything regarding this on the Expressif Github.

The code is quite straightforward and is based on this project by Andreas Spiess.

https://www.youtube.com/watch?v=sm1-l5-z3ag

I have attached a minimal example.

The error occurs when running the gclient->GET(url, googleHost); statement in my sketch. I have attached a minimal example. See the getCalendar() function.

I should probably point out that I made a minor change to the program to allow it to insert the "ESP32" string in the User-agent line in the GET and POST requests in place of "ESP8266". I don't believe this makes any functional difference, except to indicate to the Google API that the request came from an ESP32 rather than an ESP8266.

Minimal code example:

Google-Minimal.zip

hello @SheetLightning do you found a solution for this? i have the same problem here.

[probonopd]

I am also getting [ 3217][E][ssl_client.cpp:37] _handle_error(): [send_ssl_data():387]: (-27136) SSL - A buffer is too small to receive or write a message trying to get data from script.google.com. I was trying to use https://github.com/jbuszkie/HTTPSRedirect.git#31601fd.

Cross-reference:

• SSL - A buffer is too small to receive or write a message espressif/arduino-esp32#7069

[n1lby73]

any lead on this error, currently facing same thing with another ending with the message

[752202][E][ssl_client.cpp:37] _handle_error(): [start_ssl_client():264]: (-32512) SSL - Memory allocation failed

[SheetLightning]

Unfortunately, I couldn't solve the original problem so ended up not using this library and instead used direct calls to WiFiClientSecure. In case it may help someone, here is my connect routine. I set httpRequestRetries to 5 which seemed to ensure a reliable response:

WiFiClientSecure httpsClient;
const uint8_t httpRequestRetries = 5;

bool connectHost(const char * host, const char * url, uint8_t keepalive, String& response) {

  String request = ("GET ") + String(url) + " HTTP/1.1\r\n" +
                   "Host: " + String(host) + "\r\n" +
                   "User-Agent: " + WIFIMCU + "\r\n" +
                   "Connection: " + (keepalive ? "keep-alive" : "close") + "\r\n" +
                   "\r\n";

  String line = "";

  Serial.println(F("GET request:"));
  Serial.println(request);

  Serial.print(F("Connecting to "));
  Serial.println(host);

  // Insecure mode
  httpsClient.setInsecure();

  // Try to connect for a maximum of 'httpRequestRetries' times
  for (uint8_t i = 0; i < httpRequestRetries; i++) {
    int retval = httpsClient.connect(host, httpsPort);
    if (retval == 1) {
      Serial.print(F("Connected to "));
      Serial.println(host);

      Serial.print(F("Sending GET request to: "));
      Serial.print("https://");
      Serial.print(host);
      Serial.print(":");
      Serial.println(url);
      // Perform the actual request
      httpsClient.print(request);
      Serial.println(F("Sent GET request."));

      Serial.println(F("Reading response..."));
      response = "";
      while (httpsClient.connected()) {
        line = httpsClient.readStringUntil('\n');
        response += (line + '\n');
        if (line == "\r") {
          Serial.println(F("Headers received."));
          return true;
        }
      }
    }
    Serial.println(F("Connection failed. Retrying..."));
  }

  Serial.print(F("Could not connect to server: "));
  Serial.println(host);

  return false;
}