[custom connectors] SSL certificate verify failed

[robinboettcher]

Hey,

I am currently trying to create my own custom API connector.
I deployed airbyte on a local machine (Ubuntu 22.04) in Docker.

When I try to test the connector, I get the following error message:
HTTPSConnectionPool(host='...', port=44301): Max retries exceeded with url: /sap/bc/rest/... (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))"

I assume from the error message that the connector is missing the required SSL certificate for the API.
Does anyone know in which Docker container (e.g. proxy?) I have to install such a certificate?
And especially where? (e.g. /etc/ssl/certs)

Many thanks in advance!

[marcosmarxm]

You probably need to add the SSL certificate inside the custom connector Dockerfile.

[canob]

@robinboettcher , did you find the way to connect to a service that has a self signed certificate with a custom connector?

[canob]

No, I didn't find a way to make it work.

[quantum-byte]

Hey @canob , @kenske and @robinboettcher

What worked for my use case (SSL Inspection blocking Postgres->Postgres Airbyte Data transfer) was to add the relevant self signed Root Certificate to the airbyte-server docker image (both its linux cert store and java cert store) and using that image:

example Dockerfile content:

ARG AIRBYTE_VERSION
FROM airbyte/server:${AIRBYTE_VERSION}

ADD "Path/to/Root/Cert/your-cert-name.crt" "/etc/pki/ca-trust/source/anchors/your-cert-name.crt"
RUN update-ca-trust \
	&& keytool -import -noprompt -alias self-signed-root -keystore $JAVA_HOME/lib/security/cacerts -file /etc/pki/ca-trust/source/anchors/your-cert-name.crt

Though i am still using a pretty old version (AIRBYTE_VERSION=0.50.40) due to the major deployment model changes to the self deployed version. Not sure if this still works for newer versions

[aaronsteers]

@quantum-byte - Thanks for sharing this workaround! To confirm, I'm a bit surprised that modifying the airbyte-server Docker image was able to resolve the issue related to behaviors when running the connector image itself (presumably a different image). Can you confirm I'm understanding correctly? And did you come to an understanding of why this works if so? (For instance, is the server injecting its CA certificates into the image of the connector?)

Much appreciated,
AJ

[quantum-byte]

@aaronsteers

To confirm, I'm a bit surprised that modifying the airbyte-server Docker image was able to resolve the issue related to behaviors when running the connector image itself (presumably a different image). Can you confirm I'm understanding correctly? And did you come to an understanding of why this works if so?

Good question. I remembered it wrong / mixed something up here. I just had another look. I am pretty sure my workaround does not apply here. It solves two other issues with custom certificates / SSL Inspection and Airbyte (Setting up Webhooks and using the connector update functionality in the UI. Both are done by airbyte-server in the background)

(For instance, is the server injecting its CA certificates into the image of the connector?)

My guess right now is that the airbyte-server does no such CA certificate injection. It does not really make sense. Airbyte does not allow you to setup your own custom cert in a straight forward manner. Why then would they implement such a feature to inject/share such custom certificates.

Some thoughts regarding adding SSL Certs to an actual custom connector container image:

Try to figure out what base os is used by the relevant container. Distributions do SSL Certs differently. For Ubuntu (which i think is used here) you should be able to do something like

ADD "Path/to/Root/Cert/your-cert-name.crt" "/usr/local/share/ca-certificates/your-cert-name.crt"
RUN update-ca-certificates

This way at least the OS already has the correct cert. Some things work out of the box after that. Others do not. It all depends how they are implemented and what technologies they use.

Then you need to figure out what technology is used for the connector implementation. If i go by the above Exception HTTPSConnectionPool my guess would be that python and its requests library are used.

You can set the following ENV variable (for example through the Dockerfile) to make them pick up the custom cert added to the os cert bundle.

ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

/etc/ssl/certs/ca-certificates.crt in this case is the place where ubuntu places its cert bundle file built by update-ca-certificates.

If the technology used would be java, it would be quite different. Java uses a separate Cert store from the OS and you would need to figure out where the java cert store file is stored and then add your cert there (similiar to how its being done in #33498 (reply in thread))

But it gets complicated because there are a lots of different technologies/tools/libraries which each require a specific way to make them work with a custom cert.

And keep in mind, that you might need to add more then one cert file. Besides the "root" SSL cert you might need to add each "intermediate" SSL cert which is part of the chain root-cert=>intermediate-cert=>server-you-want-to-connect-to-cert

[quantum-byte]

Quick addition: To verify you have the correct certificates and the os certificate bundle works you can use openssl and run it on your local machine with the modified custom connector container image:

Here as a linux onliner based on the ubuntu latest image (which does not come with the openssl and ca-certificates packages installed by default)

docker run -it --rm ubuntu:latest bash -c "apt update && apt install openssl ca-certificates -y && echo -n "Q" | openssl s_client -connect google.com:443"

You need to replace ubuntu:latest with your custom container image and google.com:443 with the domain:port you want to connect to with the custom cert.

If everything worked you should see a

Verify return code: 0 (ok)
---
DONE