Problems connecting to etcd version 3.4.19 and later via grpc with SSL/TLS (C++)

[aletit]

Hi, I would need some assistance on understanding the issue that happens when I upgrade etcd to v 3.4.19 and newer.
Same code works fine with etcd < 3.4.19.

I have a C++ app connecting to etcd via grpc with SSL/TLS.
I use either CentOS 7 or 8, and get the same failures.

The question is what has changed in etcd 3.4.18 -> 3.4.19 that needs some changes (and which changes in that case if you have any suggestions) on my side?
I suspect this change is relevant, but I have hard times understanding the implications of it:
Upgrade #14179 to v0.0.0-20220411220226-7b82a4e95df4 to address CVE-2022-27191 .

I would appreciate any advices and help, thank you!

The code of creating a Grpc channel is as following:

    grpc::SslCredentialsOptions sslOptions;
    sslOptions.pem_root_certs = load_from_file("cacert");
    sslOptions.pem_private_key = load_from_file("key");
    sslOptions.pem_cert_chain = load_from_file("cert");

    auto credentials = grpc::SslCredentials(sslOptions);
    grpc::ChannelArguments args;
    args.SetInt(GRPC_ARG_KEEPALIVE_TIME_MS, 5);
    args.SetInt(GRPC_ARG_KEEPALIVE_TIMEOUT_MS, 5);
    return grpc::CreateCustomChannel("localhost:17123", credentials, args);

And I get the following output:
`
I0913 14:53:51.518087297 5699 subchannel.cc:711] subchannel 0x2694ca0 {address=ipv6:%5B::1%5D:17123, args={grpc.client_channel_factory=0x2646520, grpc.default_authority=localhost:17123, grpc.http2_scheme=https, grpc.internal.channel_credentials=0x266a260, grpc.internal.event_engine=0x2693c30, grpc.internal.security_connector=0x266f2c0, grpc.internal.subchannel_pool=0x266a400, grpc.keepalive_time_ms=1000, grpc.keepalive_timeout_ms=1000, grpc.primary_user_agent=grpc-c++/1.58.0, grpc.resource_quota=0x265b770, grpc.server_uri=dns:///localhost:17123}}: connect failed (UNKNOWN:Failed to connect to remote host: Connection refused {file:"/data/grpc/1.58.0/src/core/lib/iomgr/tcp_client_posix.cc", file_line:248, created_time:"2023-09-13T14:53:51.5152105+02:00", errno:111, os_error:"Connection refused", syscall:"connect", target_address:"ipv6:%5B::1%5D:17123"}), backing off for 994 ms

D0913 14:53:51.518766647 5709 tcp_posix.cc:2004] cannot set inq fd=14 errno=92

I0913 14:53:51.550658795 5709 subchannel.cc:711] subchannel 0x269ab90 {address=ipv4:127.0.0.1:17123, args={grpc.client_channel_factory=0x2646520, grpc.default_authority=localhost:17123, grpc.http2_scheme=https, grpc.internal.channel_credentials=0x266a260, grpc.internal.event_engine=0x2699bc0, grpc.internal.security_connector=0x2695750, grpc.internal.subchannel_pool=0x266a400, grpc.keepalive_time_ms=1000, grpc.keepalive_timeout_ms=1000, grpc.primary_user_agent=grpc-c++/1.58.0, grpc.resource_quota=0x265b770, grpc.server_uri=dns:///localhost:17123}}: connect failed (UNKNOWN:Failed parsing HTTP/2 {created_time:"2023-09-13T14:53:51.550215861+02:00", file_line:2544, file:"/data/grpc/1.58.0/src/core/ext/transport/chttp2/transport/chttp2_transport.cc", children:[UNKNOWN:Expected SETTINGS frame as the first frame, got frame type 80 {file:"/data/grpc/1.58.0/src/core/ext/transport/chttp2/transport/parsing.cc", file_line:402, created_time:"2023-09-13T14:53:51.550149979+02:00"}, UNKNOWN:Trying to connect an http1.x server {grpc_status:13, http_status:400, created_time:"2023-09-13T14:53:51.550197684+02:00", file_line:2507, file:"/data/grpc/1.58.0/src/core/ext/transport/chttp2/transport/chttp2_transport.cc"}]}), backing off for 966 ms

I0913 14:53:52.514451399 5704 subchannel.cc:666] subchannel 0x2694ca0 {address=ipv6:%5B::1%5D:17123, args={grpc.client_channel_factory=0x2646520, grpc.default_authority=localhost:17123, grpc.http2_scheme=https, grpc.internal.channel_credentials=0x266a260, grpc.internal.event_engine=0x2693c30, grpc.internal.security_connector=0x266f2c0, grpc.internal.subchannel_pool=0x266a400, grpc.keepalive_time_ms=1000, grpc.keepalive_timeout_ms=1000, grpc.primary_user_agent=grpc-c++/1.58.0, grpc.resource_quota=0x265b770, grpc.server_uri=dns:///localhost:17123}}: backoff delay elapsed, reporting IDLE

I0913 14:53:52.515384201 5699 subchannel.cc:711] subchannel 0x2694ca0 {address=ipv6:%5B::1%5D:17123, args={grpc.client_channel_factory=0x2646520, grpc.default_authority=localhost:17123, grpc.http2_scheme=https, grpc.internal.channel_credentials=0x266a260, grpc.internal.event_engine=0x2693c30, grpc.internal.security_connector=0x266f2c0, grpc.internal.subchannel_pool=0x266a400, grpc.keepalive_time_ms=1000, grpc.keepalive_timeout_ms=1000, grpc.primary_user_agent=grpc-c++/1.58.0, grpc.resource_quota=0x265b770, grpc.server_uri=dns:///localhost:17123}}: connect failed (UNKNOWN:Failed to connect to remote host: Connection refused {target_address:"ipv6:%5B::1%5D:17123", file:"/data/grpc/1.58.0/src/core/lib/iomgr/tcp_client_posix.cc", file_line:248, created_time:"2023-09-13T14:53:52.515120189+02:00", errno:111, os_error:"Connection refused", syscall:"connect"}), backing off for 1506 ms
`

[jmhbnz]

Hey @aletit - Thanks for your question. I'm tagging @sighingnow who maintains the etcd c++ client (https://github.com/etcd-cpp-apiv3/etcd-cpp-apiv3) who may have some ideas as I'm not experienced with c++.

[sighingnow]

Hi @jmhbnz Thanks for the ping. I would take a look on this version.

[sighingnow]

Hi @aletit may I know more details about your setup? e.g., how you install the grpc dependency. I failed to reproduce the issue on Ubuntu 20.04 with user-agent: grpc-c++/1.51.0-dev grpc-c/28.0.0 (linux; chttp2).

[sighingnow]

Cannot reproduce using the centos:latest docker image with the following setup script: https://github.com/etcd-cpp-apiv3/etcd-cpp-apiv3/blob/master/.github/workflows/centos-latest.yml (changed grpc version to v1.58.0).

[aletit]

Hi @sighingnow, thank you for such a fast turnaround! I was obviously unclear in my suspects. I was not suspecting grpc to cause this, but more wondering if an older version of openssl may cause such a problem.
On my Centos 7 system I have OpenSSL 1.0.2k-fips installed.

[sighingnow]

I still cannot reproduce using the centos:8 environments and openssl-1.0.2k (I failed to find the source tarball for 1.0.2k-fips):

• Using the centos:8 docker image
• Install dependencies:

# switch to centos stream
dnf -y --disablerepo '*' --enablerepo=extras swap centos-linux-repos centos-stream-repos
dnf -y update

# install required dependencies
yum -y group install "Development Tools"
yum -y install boost-devel \
                cmake \
                git \
                wget

• Build & install openssl-1.0.2k: https://ftp.openssl.org/source/old/1.0.2/openssl-1.0.2k.tar.gz

make build_ssl build_crypto build_engines install CFLAG="-O -fPIC"

• Install grpc-1.58.0 from source git clone https://github.com/grpc/grpc.git --depth 1 --branch v1.58.0:

mkdir cmake-build
cd cmake-build
cmake .. -DBUILD_SHARED_LIBS=ON \
  -DgRPC_INSTALL=ON \
  -DgRPC_BUILD_TESTS=OFF \
  -DgRPC_BUILD_CSHARP_EXT=OFF \
  -DgRPC_BUILD_GRPC_CSHARP_PLUGIN=OFF \
  -DgRPC_BUILD_GRPC_NODE_PLUGIN=OFF \
  -DgRPC_BUILD_GRPC_OBJECTIVE_C_PLUGIN=OFF \
  -DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
  -DgRPC_BUILD_GRPC_PYTHON_PLUGIN=OFF \
  -DgRPC_BUILD_GRPC_RUBY_PLUGIN=OFF \
  -DgRPC_BACKWARDS_COMPATIBILITY_MODE=ON \
  -DgRPC_ZLIB_PROVIDER=package \
  -DgRPC_SSL_PROVIDER=package -DOPENSSL_ROOT_DIR=/usr/local/ssl
make -j`nproc`
make install

• Install cppresksdk from source git clone https://github.com/microsoft/cpprestsdk.git --depth=1:

mkdir cmake-build
cd cmake-build
cmake .. -DCMAKE_BUILD_TYPE=Debug \
          -DBUILD_TESTS=OFF \
          -DBUILD_SAMPLES=OFF \
          -DCPPREST_EXCLUDE_WEBSOCKETS=ON -DOPENSSL_ROOT_DIR=/usr/local/ssl
make -j`nproc`
make install

• Build etcd-cpp-apiv3 and the SecurityChannelTest test case:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib:/usr/local/lib/x86_64-linux-gnu

mkdir build
cd build
cmake .. -DBUILD_ETCD_TESTS=ON \
         -DOPENSSL_ROOT_DIR=/usr/local/ssl
make SecurityChannelTest

• Setup keys and secrets:

# change to the source directory of etcd-cpp-apiv3
./security-config/reset-ca.sh
./security-config/setup-ca.sh

• Launch etcd-3.4.19:

./build/etcd-v3.4.19-linux-amd64/etcd \
  --cert-file security-config/certs/etcd0.example.com.crt \
  --key-file security-config/private/etcd0.example.com.key \
  --client-cert-auth \
  --trusted-ca-file security-config/certs/ca.crt \
  --advertise-client-urls=https://127.0.0.1:2379 \
  --listen-client-urls=https://127.0.0.1:2379

• Run the test case:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib:/usr/local/lib/x86_64-linux-gnu

./build/bin/SecurityChannelTest