Cannot connect to http socket (zsock_connect) with TLS credential setup due to error 22 - Invalid Argument

[Qtran1511]

Hi, I'm facing a problem after TLS credential setup, the socket I created cannot be connected, it throws the error number 22 - Invalid Argument which I have no idea where it comes from. My purpose with this code is to send a POST request to http host login.microsoftonline.com to get OAuth2.0 access token. Without TLS credential setup, it works perfectly fine but of course I would get redirect URL response due to missing credential to talk with port 443. But when I try setting TLS, it has error at zsock_connect function. This is my sample code, WiFi is already setup and connected.

#include <stddef.h>
#include <errno.h>
#include <stdint.h>
#include <stdbool.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>

#include <zephyr/kernel.h>
#include <zephyr/logging/log.h>
#include <zephyr/net/socket.h>
#include <zephyr/net/http/client.h>
#include <zephyr/net/tls_credentials.h>

#include "ca_certificate.h"

#define HTTP_HOST "login.microsoftonline.com"
#define HTTP_PROTOCOL "HTTP/1.1"
#if defined(CONFIG_NET_SOCKETS_SOCKOPT_TLS)
#define HTTP_PORT "443"
#else
#define HTTP_PORT "80"
#endif

LOG_MODULE_REGISTER(esp32_get_token);

static void dump_addrinfo(const struct zsock_addrinfo *ai)
{
	LOG_INF("Address info @%p", ai);
	LOG_INF("Address info Family: %s", ai->ai_family ? "IPv4" : "IPv6");
	LOG_INF("Socket type: %d", ai->ai_socktype);
	LOG_INF("Socket address Family: %s", ai->ai_addr->sa_family ? "IPv4" : "IPv6");
	LOG_INF("Port: %" PRIu16, ((struct sockaddr_in *)ai->ai_addr)->sin_port);

	char peer_addr[INET6_ADDRSTRLEN];
	struct sockaddr *sa = ai->ai_addr;
	inet_ntop(sa->sa_family, (void *)&((struct sockaddr_in *)sa)->sin_addr,
			  peer_addr, INET6_ADDRSTRLEN);
	LOG_INF("Peer address: %s", peer_addr);
}

void get_token(void)
{
	LOG_INF("Setting up socket...");
	int ret, sock;
	static struct zsock_addrinfo hints =
		{
			.ai_flags = AI_NUMERICSERV,
			.ai_family = AF_INET,
			.ai_socktype = SOCK_STREAM,
			.ai_next = NULL,
		};
	struct zsock_addrinfo *res;
	ret = zsock_getaddrinfo(HTTP_HOST, HTTP_PORT, &hints, &res);
	if (ret != 0)
	{
		LOG_ERR("Get address info failed (%d)", ret);
		return;
	}
	LOG_INF("Got address info");
	dump_addrinfo(res);

#if defined(CONFIG_NET_SOCKETS_SOCKOPT_TLS)
	sock = zsock_socket(res->ai_family, res->ai_socktype, IPPROTO_TLS_1_2);
#else
	sock = zsock_socket(res->ai_family, res->ai_socktype, res->ai_protocol);
#endif
	if (sock < 0)
	{
		LOG_ERR("Failed to create HTTP socket (%d)", ret);
		return;
	}
	LOG_INF("Created HTTP socket");

#if defined(CONFIG_NET_SOCKETS_SOCKOPT_TLS)
	sec_tag_t sec_tag_list[] = {
		CA_CERTIFICATE_TAG,
	};
	ret = tls_credential_add(CA_CERTIFICATE_TAG, TLS_CREDENTIAL_CA_CERTIFICATE,
							 ca_cert, sizeof(ca_cert));
	if (ret < 0)
	{
		LOG_ERR("Failed to add TLS credential (%d)", -errno);
		return;
	}
	LOG_INF("TLS credential added at tag %d", sec_tag_list[0]);

	ret = zsock_setsockopt(sock, SOL_TLS, TLS_SEC_TAG_LIST, sec_tag_list, sizeof(sec_tag_list));
	if (ret < 0)
	{
		LOG_ERR("Failed to set secure option (%d)", -errno);
		return;
	}
	LOG_INF("Set %d secure options succeeded", sizeof(sec_tag_list) / sizeof(sec_tag_list[0]));

	ret = zsock_setsockopt(sock, SOL_TLS, TLS_HOSTNAME, TLS_PEER_HOST, strlen(TLS_PEER_HOST));
	if (ret < 0)
	{
		LOG_ERR("Failed to set %s hostname (%d)", TLS_PEER_HOST, -errno);
		return;
	}
	LOG_INF("Set %s hostname succeeded", TLS_PEER_HOST);

#endif

	ret = zsock_connect(sock, res->ai_addr, res->ai_addrlen);
	if (ret < 0)
	{
		LOG_ERR("Cannot connect to socket (%d)", -errno);
		return;
	}
	LOG_INF("Connected to socket");

//send http POST request ...
}

This is my ca_certificate.h file, the credential certificate I used which I took from login.microsoftonline.com, it is called DigiCert Global Root CA.

#ifndef CA_CERTIFICATE_H
#define CA_CERTIFICATE_H

#define CA_CERTIFICATE_TAG 1
#define TLS_PEER_HOST "login.microsoftonline.com"

static const unsigned char ca_cert[] = {
    // #include "DigiCertGlobalRootCA.der.inc"
    "-----BEGIN CERTIFICATE-----\
MIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh\
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3\
d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBD\
QTAeFw0wNjExMTAwMDAwMDBaFw0zMTExMTAwMDAwMDBaMGExCzAJBgNVBAYTAlVT\
MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5j\
b20xIDAeBgNVBAMTF0RpZ2lDZXJ0IEdsb2JhbCBSb290IENBMIIBIjANBgkqhkiG\
9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4jvhEXLeqKTTo1eqUKKPC3eQyaKl7hLOllsB\
CSDMAZOnTjC3U/dDxGkAV53ijSLdhwZAAIEJzs4bg7/fzTtxRuLWZscFs3YnFo97\
nh6Vfe63SKMI2tavegw5BmV/Sl0fvBf4q77uKNd0f3p4mVmFaG5cIzJLv07A6Fpt\
43C/dxC//AH2hdmoRBBYMql1GNXRor5H4idq9Joz+EkIYIvUX7Q6hL+hqkpMfT7P\
T19sdl6gSzeRntwi5m3OFBqOasv+zbMUZBfHWymeMr/y7vrTC0LUq7dBMtoM1O/4\
gdW7jVg/tRvoSSiicNoxBN33shbyTApOB6jtSj1etX+jkMOvJwIDAQABo2MwYTAO\
BgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUA95QNVbR\
TLtm8KPiGxvDl7I90VUwHwYDVR0jBBgwFoAUA95QNVbRTLtm8KPiGxvDl7I90VUw\
DQYJKoZIhvcNAQEFBQADggEBAMucN6pIExIK+t1EnE9SsPTfrgT1eXkIoyQY/Esr\
hMAtudXH/vTBH1jLuG2cenTnmCmrEbXjcKChzUyImZOMkXDiqw8cvpOp/2PV5Adg\
06O/nVsJ8dWO41P0jmP6P6fbtGbfYmbW0W5BjfIttep3Sp+dWOIrWcBAI+0tKIJF\
PnlUkiaY4IBIqDfv8NZ5YBberOgOzW6sRBc4L0na4UU+Krk2U886UAb3LujEV0ls\
YSEY1QSteDwsOoBrp+uvFRTp2InBuThs4pFsiv9kuXclVzDAGySj4dzp30d8tbQk\
CAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4=\
-----END CERTIFICATE-----"};

#endif

I have tried to include the .der file to directly get the data but it has a problem with handshake function so I decide to paste the whole block of certificate instead.

This is my prj.conf:

CONFIG_MAIN_STACK_SIZE=4096
CONFIG_INIT_STACKS=y
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=2048
CONFIG_LOG=y

CONFIG_MP_NUM_CPUS=2
CONFIG_MP_MAX_NUM_CPUS=3
CONFIG_ASSERT=y
CONFIG_SETTINGS=y
CONFIG_ASSERT_ON_ERRORS=y
CONFIG_ESP_HEAP_MEM_POOL_REGION_1_SIZE=20000
CONFIG_HEAP_MEM_POOL_SIZE=40000
CONFIG_MINIMAL_LIBC_MALLOC=y
CONFIG_MINIMAL_LIBC=y
CONFIG_MINIMAL_LIBC_MALLOC_ARENA_SIZE=5000

CONFIG_WIFI=y
CONFIG_WIFI_ESP32=y
CONFIG_NETWORKING=y
CONFIG_NET_MGMT=y
CONFIG_NET_MGMT_EVENT=y
CONFIG_NET_MGMT_EVENT_INFO=y
CONFIG_NET_IPV4=y
CONFIG_NET_IPV6=y
CONFIG_NET_TCP=y
CONFIG_NET_SOCKETS=y
CONFIG_NET_SOCKETS_POSIX_NAMES=y
CONFIG_NET_DHCPV4=y
CONFIG_NET_TX_STACK_SIZE=2048
CONFIG_NET_RX_STACK_SIZE=2048
CONFIG_NET_L2_ETHERNET=y
CONFIG_HTTP_CLIENT=y

# CONFIG_NET_HTTP_LOG_LEVEL_DBG=y
# CONFIG_NET_SOCKETS_LOG_LEVEL_DBG=y
# CONFIG_NET_CONTEXT_LOG_LEVEL_DBG=y
# CONFIG_NET_TCP_LOG_LEVEL_DBG=y

CONFIG_MBEDTLS=y
CONFIG_MBEDTLS_BUILTIN=y
CONFIG_MBEDTLS_ENABLE_HEAP=y
CONFIG_MBEDTLS_HEAP_SIZE=15000
CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN=2048
CONFIG_MBEDTLS_LOG_LEVEL_DBG=y

CONFIG_NET_SOCKETS_SOCKOPT_TLS=y
CONFIG_NET_SOCKETS_TLS_MAX_CONTEXTS=6

I have digged very deep in the source code of zephyr subsystem to find the source of error, turn on debug log, place serveral printk() to track the progress. I dig to TCP module, no problem or log error at all, it can send and receive data perfectly. But when I track the sockets.c module in zephyrproject/zephyr/subsys/net/lib/sockets, I place a printk("%s %d\n", __func__, ret) inside VTABLE_CALL(fn, sock, ...) macro function like this:

#define VTABLE_CALL(fn, sock, ...)                                                                 \
	do {                                                                                       \
		const struct socket_op_vtable *vtable;                                             \
		struct k_mutex *lock;                                                              \
		void *obj;                                                                         \
		int ret;                                                                           \
                                                                                                   \
		obj = get_sock_vtable(sock, &vtable, &lock);                                       \
		if (obj == NULL) {                                                                 \
			errno = EBADF;                                                             \
			return -1;                                                                 \
		}                                                                                  \
                                                                                                   \
		if (vtable->fn == NULL) {                                                          \
			errno = EOPNOTSUPP;                                                        \
			return -1;                                                                 \
		}                                                                                  \
                                                                                                   \
		(void)k_mutex_lock(lock, K_FOREVER);                                               \
                                                                                                   \
		ret = vtable->fn(obj, __VA_ARGS__);                                                \
                printk("%s %d\n", __func__, ret);                                                  \
		k_mutex_unlock(lock);                                                              \
                                                                                                   \
		return ret;                                                                        \
                                                                                                   \
	} while (0)

I saw this while loop run twice in the function z_impl_zsock_connect, the first attempt works perfectly with the ret = 0 from sock_connect_vmeth as well as zsock_connect_ctx and have no problem at all in the deeper nest of subsystem code, at this point, I'm 99% sure the socket is connected and ready to send the http client request. But then the second attempt pops up with ret = -1 without any deeper execution, and errno = 22 from nowhere, no log error, none of my printk() appears.

Can anyone explain this situation. I'm really confused, I even tried swap to google.com host and the corresponding TLS certificate but still the same error. I'm extremely appriciate if anyone can help.

[scath999]

For what it's worth I'm in the same boat.

Oddly, I had TLS working in March 2023, but when I returned to this code in April it had stopped working.

Would appreciate some input from the @zephyr team. The layers of abstraction make issues like this difficult to trace, and lack of meaningful documentation in key areas (like TLS) is stunning.

[Qtran1511]

Hi,

Luckily I have it sorted out a couple weeks ago, the solution was so straightforward that make myself feel so dumb. I thought when calling zsock_connect, it would execute zsock_connect_ctx directly so I kept digging it and found no way out. It turns out that zsock_connect_ctx works totally fine, the issue is that the socket module will find the appropriate function according to fdtable, if my socket (or file descriptor) is just a regular TCP socket without TLS setup, zsock_connect_ctx is called directly. However, if TLS is used, it will pass the socket to sockets_tls.c module and execute ztls_connect_ctx function instead in which zsock_connect is called again. My issue was from incorrect certificate format, it must be .der file instead of pasting the whole certificate content. I mentioned before that I had handshake problem when using .der file, of course it's because of incorrect certificate so I needed to use the right one, and use different certificates for different webpages or servers.

I'm not sure what your problem is but my hint is that you shoud avoid (if possible) using different version of zephyr, I used to struggle with my code functionality because I installed 2 zephyr version 2 weeks apart since I had my company computer upgraded. Although it's the same big version, they are still on the way developing it and some line of code swapped between functions made my code stop working properly, so I had to check the commits and changed it back locally. Hopefully it's helpful for you.

[scath999]

Hi,

I'm glad your issue has been resolved. Ours has not, though I'm (technically) on vacation atm.

My biggest issue is that little meaningful documentation exists for understanding and solving these issues. The documentation that does exist is largely unhelpful.

Worse are the error codes. In your case, you were getting an EINVAL because you weren't using a .der file - but that file isn't an argument for connect(), which is what was reporting the error.

That's bad.

Respectfully, we shouldn't feel "dumb" because other engineers aren't properly documenting. We all know - because we're all taught on day one - that documentation is the fundamental activity of all science and engineering. Failure on the part of our colleagues to observe this isn't reason to feel bad about ourselves.

[GilDev]

Hi,
Also getting the same issues. Somehow example.com works fine but both google.com and gildev.dev won’t. Certificates for the latter two are different (and longer than for example.com), maybe this could be a reason?

[MateoSegura]

Also having issues with this, with no debug output to help debug the issue. Err -22 on connect() using TLS

[GilDev]

Looks like Oberon backend doesn’t support >3kb certificates. example.com uses a 2048 bit certificate, google.com a 4096 one. CC33XX backend doesn’t have the issue. That is what was causing the issue for me on an nRF9160DK.

[MateoSegura]

Okay so I figured out what the issue was on my end. It's rather simple, but not obvious anywhere in the docs.
I'm guessing you're using EmbedTLS, so what you most likely are running into is an issue within the library.

The problem is that a large number of issues will simply get marked in Zephyr as -22 (Invalid argument). So what you need to do is to enable debug output from the library itself, which would give you some sort of error code.

Here's what you need to do:

1. In your prj.conf add this line

CONFIG_MBEDTLS_DEBUG=y

2. In you main.c include the following file

#include <mbedtls/debug.h>

3. In you main() add the following line

mbedtls_debug_set_threshold(DEBUG_LEVEL); 

Note that the debug level will vary from person to person, according to the beautiful comment in the function documentation:

/**
 * \brief   Set the threshold error level to handle globally all debug output.
 *          Debug messages that have a level over the threshold value are
 *          discarded.
 *          (Default value: 0 = No debug )
 *
 * \param threshold     threshold level of messages to filter on. Messages at a
 *                      higher level will be discarded.
 *                          - Debug levels
 *                              - 0 No debug
 *                              - 1 Error
 *                              - 2 State change
 *                              - 3 Informational
 *                              - 4 Verbose
 */
void mbedtls_debug_set_threshold(int threshold);

4. This should hopefully output an error code to the console, either in HEX or decimal. You can then go to the link below and figure out what your error is:
   https://gist.github.com/erikcorry/b25bdcacf3e0086f8a2afb688420678e

For me personally, I was running into issues where I didn't have the right Kconfig settings enabled for the algorithms my certificates and keys were using, as well as the bulk encryption used/accepted by my remote POSIX end.

Hope it helps :)