SSL: CERTIFICATE_VERIFY_FAILED - Downloading MY_ATP.pdb

[sxm525]

I am getting "SSL: CERTIFICATE_VERIFY_FAILED" during db startup, could you please help.

Find below my full log,

[user@BRL72J3 ~]$ podman logs -t f2f6d724008ad02027d72c8c393a8221cafe9b5c3a1bbd08e0d6880f5a3f70fd
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable PODMAN_IGNORE_CGROUPSV1_WARNING to hide this warning.
2024-05-22T15:31:39.345767000-04:00 Archive: /u01/POD1.zip
2024-05-22T15:31:39.347480000-04:00 creating: /u01/app/oracle/oradata/
2024-05-22T15:31:39.348648000-04:00 creating: /u01/app/oracle/oradata/POD1/
2024-05-22T15:31:39.349110000-04:00 creating: /u01/app/oracle/oradata/POD1/17204CBA6183008CE063C4D75E6492EC/
2024-05-22T15:31:39.349152000-04:00 creating: /u01/app/oracle/oradata/POD1/17204CBA6183008CE063C4D75E6492EC/datafile/
2024-05-22T15:31:39.349178000-04:00 creating: /u01/app/oracle/oradata/POD1/175E8D561B6D4CDEE0636402000A1C40/
2024-05-22T15:31:39.349644000-04:00 creating: /u01/app/oracle/oradata/POD1/175E8D561B6D4CDEE0636402000A1C40/datafile/
2024-05-22T15:31:39.442155000-04:00 inflating: /u01/app/oracle/oradata/POD1/redo01.log
2024-05-22T15:31:39.532134000-04:00 inflating: /u01/app/oracle/oradata/POD1/redo02.log
2024-05-22T15:31:39.532407000-04:00 creating: /u01/app/oracle/oradata/POD1/datafile/
2024-05-22T15:31:39.585161000-04:00 inflating: /u01/app/oracle/oradata/POD1/datafile/o1_mf_temp_m33n3tv9_.tmp
2024-05-22T15:32:06.735885000-04:00 inflating: /u01/app/oracle/oradata/POD1/datafile/o1_mf_system_m33lvr7m_.dbf
2024-05-22T15:32:06.814266000-04:00 inflating: /u01/app/oracle/oradata/POD1/datafile/o1_mf_undotbs1_m33n3t3g_.dbf
2024-05-22T15:32:15.172292000-04:00 inflating: /u01/app/oracle/oradata/POD1/datafile/o1_mf_sysaux_m33lz7xh_.dbf
2024-05-22T15:32:15.601036000-04:00 inflating: /u01/app/oracle/oradata/POD1/datafile/o1_mf_data_m33m2s61_.dbf
2024-05-22T15:32:15.618548000-04:00 creating: /u01/app/oracle/oradata/POD1/175E8F7D3A3D4D65E0636402000AA841/
2024-05-22T15:32:15.618700000-04:00 creating: /u01/app/oracle/oradata/POD1/175E8F7D3A3D4D65E0636402000AA841/datafile/
2024-05-22T15:32:15.660253000-04:00 creating: /u01/app/oracle/oradata/POD1/onlinelog/
2024-05-22T15:32:15.673523000-04:00 creating: /u01/app/oracle/oradata/POD1/171F1841E82CF4B3E063C4D75E643770/
2024-05-22T15:32:15.673625000-04:00 creating: /u01/app/oracle/oradata/POD1/171F1841E82CF4B3E063C4D75E643770/datafile/
2024-05-22T15:32:15.696223000-04:00 creating: /u01/app/oracle/oradata/POD1/175E81D3E11F4989E0636402000AD075/
2024-05-22T15:32:15.696298000-04:00 creating: /u01/app/oracle/oradata/POD1/175E81D3E11F4989E0636402000AD075/datafile/
2024-05-22T15:32:15.696324000-04:00 creating: /u01/app/oracle/oradata/POD1/controlfile/
2024-05-22T15:32:15.696351000-04:00 inflating: /u01/app/oracle/oradata/POD1/controlfile/o1_mf_m33m65kk_.ctl
2024-05-22T15:32:16.320877000-04:00 TIME ELAPSED Unzipping /u01/POD1.zip: 0 minutes and 37 seconds elapsed
2024-05-22T15:32:18.065494000-04:00 User input JSON not found
2024-05-22T15:32:18.128315000-04:00 MY ADB WORKLOAD_TYPE is ATP
2024-05-22T15:32:18.128971000-04:00 MY ADB CUSTOM NAME is MYATP
2024-05-22T15:32:18.151187000-04:00 BUILDER: Configuring TCPS
2024-05-22T15:32:18.152329000-04:00 BUILDER: Cleanup /u01/app/oracle/wallets/tls_wallet
2024-05-22T15:32:18.164291000-04:00 BUILDER: Creating auto login wallet for server
2024-05-22T15:32:18.456193000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:18.456334000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:18.456874000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:18.456945000-04:00
2024-05-22T15:32:18.489135000-04:00 Enter password:
2024-05-22T15:32:18.489506000-04:00 Enter password again:
2024-05-22T15:32:18.733760000-04:00 Operation is successfully completed.
2024-05-22T15:32:18.749360000-04:00 BUILDER: Creating a self-signed certificate using orapki utility; VALIDITY: 10 years
2024-05-22T15:32:18.909708000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:18.911516000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:18.911602000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:18.911624000-04:00
2024-05-22T15:32:19.240145000-04:00 Cannot modify auto-login (sso) wallet
2024-05-22T15:32:19.241408000-04:00 Enter wallet password:
2024-05-22T15:32:21.144281000-04:00 Operation is successfully completed.
2024-05-22T15:32:21.151481000-04:00 BUILDER: exporting server's cert
2024-05-22T15:32:21.347089000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:21.347278000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:21.347709000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:21.347776000-04:00
2024-05-22T15:32:22.043356000-04:00 Operation is successfully completed.
2024-05-22T15:32:22.059921000-04:00 BUILDER: exporting server's cert
2024-05-22T15:32:22.318691000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:22.319136000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:22.319746000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:22.319829000-04:00
2024-05-22T15:32:23.128622000-04:00 Operation is successfully completed.
2024-05-22T15:32:23.145920000-04:00 BUILDER: exporting encrypted private key
2024-05-22T15:32:23.437932000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:23.438636000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:23.438723000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:23.438760000-04:00
2024-05-22T15:32:23.453315000-04:00 Private key password:
2024-05-22T15:32:23.454948000-04:00 Enter password:
2024-05-22T15:32:23.469735000-04:00 Enter password again:
2024-05-22T15:32:41.651260000-04:00 Enter wallet password:
2024-05-22T15:32:41.979993000-04:00 Operation is successfully completed.
2024-05-22T15:32:41.994723000-04:00 BUILDER: exporting private and certificates together in PEM
2024-05-22T15:32:41.999597000-04:00 BUILDER: generating keystore.jks and truststore.jks
2024-05-22T15:32:42.152267000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:42.152415000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:42.152441000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:42.152480000-04:00
2024-05-22T15:33:00.351039000-04:00 Enter wallet password:
2024-05-22T15:33:00.687456000-04:00 Enter Key store password:
2024-05-22T15:33:00.688343000-04:00 Enter Trust store password:
2024-05-22T15:33:00.785159000-04:00 Operation is successfully completed.
2024-05-22T15:33:00.800159000-04:00 BUILDER: generating sqlnet.ora for client
2024-05-22T15:33:00.800328000-04:00 BUILDER: Generating tnsnames.ora based on the new CN
2024-05-22T15:33:00.803804000-04:00 BUILDER: Overriding service names based on user input
2024-05-22T15:33:00.805435000-04:00 BUILDER: generating ojdbc.properties
2024-05-22T15:33:00.805852000-04:00 BUILDER: zipping wallet for ORDS
2024-05-22T15:33:00.814854000-04:00 updating: README (stored 0%)
2024-05-22T15:33:00.814996000-04:00 updating: adb_container.cert (deflated 24%)
2024-05-22T15:33:00.815033000-04:00 updating: cwallet.sso (stored 0%)
2024-05-22T15:33:00.815062000-04:00 updating: cwallet.sso.lck (stored 0%)
2024-05-22T15:33:00.815108000-04:00 updating: ewallet.p12 (stored 0%)
2024-05-22T15:33:00.815139000-04:00 updating: ewallet.p12.lck (stored 0%)
2024-05-22T15:33:00.815194000-04:00 updating: ewallet.pem (deflated 27%)
2024-05-22T15:33:00.815272000-04:00 updating: keystore.jks (stored 0%)
2024-05-22T15:33:00.815342000-04:00 updating: ojdbc.properties (deflated 49%)
2024-05-22T15:33:00.815451000-04:00 updating: sqlnet.ora (deflated 16%)
2024-05-22T15:33:00.815494000-04:00 updating: tnsnames.ora (deflated 87%)
2024-05-22T15:33:00.815539000-04:00 updating: truststore.jks (deflated 5%)
2024-05-22T15:33:00.817348000-04:00 TIME ELAPSED Wallet Generation: 0 minutes and 42 seconds elapsed
2024-05-22T15:33:01.258150000-04:00 User has requested to download '.pdb' archive file from Object Storage bucket
2024-05-22T15:33:01.258531000-04:00 Downloading MY_ATP.pdb..
2024-05-22T15:33:01.267569000-04:00 Traceback (most recent call last):
2024-05-22T15:33:01.267744000-04:00 File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
2024-05-22T15:33:01.267786000-04:00 chunked=chunked)
2024-05-22T15:33:01.267823000-04:00 File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 344, in _make_request
2024-05-22T15:33:01.267860000-04:00 self._validate_conn(conn)
2024-05-22T15:33:01.267895000-04:00 File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 844, in validate_conn
2024-05-22T15:33:01.267931000-04:00 conn.connect()
2024-05-22T15:33:01.267972000-04:00 File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 358, in connect
2024-05-22T15:33:01.268007000-04:00 ssl_context=context)
2024-05-22T15:33:01.268043000-04:00 File "/usr/lib/python3.6/site-packages/urllib3/util/ssl.py", line 354, in ssl_wrap_socket
2024-05-22T15:33:01.268078000-04:00 return context.wrap_socket(sock, server_hostname=server_hostname)
2024-05-22T15:33:01.268153000-04:00 File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket
2024-05-22T15:33:01.268195000-04:00 _context=self, _session=session)
2024-05-22T15:33:01.268226000-04:00 File "/usr/lib64/python3.6/ssl.py", line 810, in init
2024-05-22T15:33:01.268259000-04:00 self.do_handshake()
2024-05-22T15:33:01.268360000-04:00 File "/usr/lib64/python3.6/ssl.py", line 1070, in do_handshake
2024-05-22T15:33:01.268424000-04:00 self._sslobj.do_handshake()
2024-05-22T15:33:01.268462000-04:00 File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
2024-05-22T15:33:01.268491000-04:00 self._sslobj.do_handshake()
2024-05-22T15:33:01.268526000-04:00 ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
2024-05-22T15:33:01.268564000-04:00
2024-05-22T15:33:01.268594000-04:00 During handling of the above exception, another exception occurred:
2024-05-22T15:33:01.268623000-04:00
2024-05-22T15:33:01.268650000-04:00 Traceback (most recent call last):
2024-05-22T15:33:01.268677000-04:00 File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2024-05-22T15:33:01.268709000-04:00 timeout=timeout
2024-05-22T15:33:01.268737000-04:00 File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
2024-05-22T15:33:01.268762000-04:00 _stacktrace=sys.exc_info()[2])
2024-05-22T15:33:01.268798000-04:00 File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
2024-05-22T15:33:01.268826000-04:00 raise MaxRetryError(_pool, url, error or ResponseError(cause))
2024-05-22T15:33:01.268855000-04:00 urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='objectstorage.us-phoenix-1.oraclecloud.com', port=443): Max retries exceeded with url: /n/dwcsdev/b/adb-free-23c/o/ADBS-24.4.4.2-23ai/MY_ATP.pdb (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))
2024-05-22T15:33:01.269425000-04:00
2024-05-22T15:33:01.269500000-04:00 During handling of the above exception, another exception occurred:
2024-05-22T15:33:01.269540000-04:00
2024-05-22T15:33:01.269567000-04:00 Traceback (most recent call last):
2024-05-22T15:33:01.269588000-04:00 File "/u01/scripts/download_my_container_pdb.py", line 102, in
2024-05-22T15:33:01.269607000-04:00 downloader.download()
2024-05-22T15:33:01.269634000-04:00 File "/u01/scripts/download_my_container_pdb.py", line 61, in download
2024-05-22T15:33:01.269655000-04:00 r = requests.get(download_url, stream=True)
2024-05-22T15:33:01.269676000-04:00 File "/usr/lib/python3.6/site-packages/requests/api.py", line 75, in get
2024-05-22T15:33:01.269696000-04:00 return request('get', url, params=params, **kwargs)
2024-05-22T15:33:01.269718000-04:00 File "/usr/lib/python3.6/site-packages/requests/api.py", line 60, in request
2024-05-22T15:33:01.269738000-04:00 return session.request(method=method, url=url, **kwargs)
2024-05-22T15:33:01.269760000-04:00 File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 535, in request
2024-05-22T15:33:01.269781000-04:00 resp = self.send(prep, **send_kwargs)
2024-05-22T15:33:01.269799000-04:00 File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 648, in send
2024-05-22T15:33:01.269820000-04:00 r = adapter.send(request, **kwargs)
2024-05-22T15:33:01.269840000-04:00 File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
2024-05-22T15:33:01.269861000-04:00 raise SSLError(e, request=request)
2024-05-22T15:33:01.269884000-04:00 requests.exceptions.SSLError: HTTPSConnectionPool(host='objectstorage.us-phoenix-1.oraclecloud.com', port=443): Max retries exceeded with url: /n/dwcsdev/b/adb-free-23c/o/ADBS-24.4.4.2-23ai/MY_ATP.pdb (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))

[aosingh]

@sxm525

Does this always happen during container start ? I don't expect to see this failure.

[sxm525]

Yes, I tried multiple times and it failed with the same error.

[aosingh]

• Which Operating system ?

• What is the podman version ?

podman version

• Are you running podman in root or rootless mode ?

• What is the output of following curl command on the host machine ?

curl https://objectstorage.us-phoenix-1.oraclecloud.com:443

[sxm525]

OS - Windows 10
podman version - 5.0.3
Podman running in rootless mode.

Find below the curl output,
curl https://objectstorage.us-phoenix-1.oraclecloud.com:443
{"code":"NotFound","message":"Not Found"}

[aosingh]

Could you SSH to the podman virtual machine and then try the same curl request ?

podman machine ssh
curl https://objectstorage.us-phoenix-1.oraclecloud.com:443

What is the container start command used ?

[sxm525]

Find below my curl output from podman machine,

[user@BR~]$ curl https://objectstorage.us-phoenix-1.oraclecloud.com:443
curl: (60) SSL certificate problem: self-signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I used below syntax,
podman run -d
-p 1521:1522
-p 1522:1522
-p 8443:8443
-p 27017:27017
-e WORKLOAD_TYPE=ATP
-e WALLET_PASSWORD=***
-e ADMIN_PASSWORD=***
--cap-add SYS_ADMIN
--device /dev/fuse
--name adb-free
ghcr.io/oracle/adb-free:latest-23ai

[aosingh]

The issues seems to be that on the podman virtual machine which is running on the Windows host, there is no TLS CA bundle to verify the certificate's validity. Could you connect to any host using HTTPS on the podman VM ?

I don't have a Windows machine myself but have been trying to find someone to reproduce this. This does not happen on Linux or MacOS.

Another option could be download the root CA for objectstorage.us-phoenix-1.oraclecloud.com and add it to the podman VM's truststore

sudo cp <oci-root-ca.pem> /etc/pki/ca-trust/source/anchors
sudo update-ca-trust

[sxm525]

I copied the PEM file and tried but same issue. Build failed during download "MY_ATP.pdb" file.
But I am able to download this file from windows machine & podman machine without any issue. Find below my output.

This is failing only build runtime,

ERROR,
#############################################
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='objectstorage.us-phoenix-1.oraclecloud.com', port=443): Max retries exceeded with url: /n/dwcsdev/b/adb-free-23c/o/ADBS-24.4.4.2-23ai/MY_ATP.pdb (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))
######################################

Find below my curl out for file MY_ATP.pdb

Windows machine,
##############
C:\Users>curl --output MY_ATP.pdb https://objectstorage.us-phoenix-1.oraclecloud.com/n/dwcsdev/b/adb-free-23c/o/ADBS-24.4.4.2-23ai/MY_ATP.pdb
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 363M 100 363M 0 0 3405k 0 0:01:49 0:01:49 --:--:-- 2838k

Podman machine,
############
[user@BRL72J3 ~]$ curl --output MY_ATP.pdb https://objectstorage.us-phoenix-1.oraclecloud.com/n/dwcsdev/b/adb-free-23c/o/ADBS-24.4.4.2-23ai/MY_ATP.pdb
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
24 363M 24 87.6M 0 0 2674k 0 0:02:19 0:00:33 0:01:46 2449k

[aosingh]

Could we try updating the VM OS packages ?

podman machine ssh 'sudo rpm-ostree upgrade --check'
podman machine stop && podman machine start

and then retry starting the container ?

[henrikedsparr]

Is there a fix for this. I'm getting the same results even though i added all certificates to the VM.
I can fetch the image from the VM but not the container during startup.
I'm running podman 5.0.3 and RPM 4.19.1.1
I'm not able to update the OS packages since rpm-ostree is not present, just regular rpm.

[aosingh]

@henrikedsparr @sxm525

After some reading and consulting with users using Windows, It is recommended to use Window Subsystem for Linux (WSL2)

Please refer the link : https://medium.com/@sociable_flamingo_goose_694/setup-wsl-for-local-docker-development-on-windows-f0767e0a72d4 to setup WSL2 and install docker engine.

This will start an Linux VM on your Windows host using WSL in which the adb-container can run using docker.

[henrikedsparr]

@aosingh Thanks for the information. Yesterday i managed to solve it by building my own container image and adding the certificates to /etc/pki/ca-trust/source/anchors and running update-ca-trust.
So if you are using Zscaler you need to add certificates to the container this way for it to work.

[alexesca]

can this be closed?

[sxm525]

Yes, please close it. Thanks [Graphical user interface Description automatically generated] Sudhan Madhavan The Sherwin-Williams Company Cell:216-906-0489 ***@***.******@***.***> From: Alexander Escamilla ***@***.***> Sent: Friday, September 20, 2024 1:09 PM To: oracle/adb-free ***@***.***> Cc: Sudhan Madhavan ***@***.***>; Mention ***@***.***> Subject: [EXTERNAL] Re: [oracle/adb-free] SSL: CERTIFICATE_VERIFY_FAILED - Downloading MY_ATP.pdb (Issue #22) [Caution] External email. Be sure you trust or verify the sender before entering usernames or passwords when prompted by a link. can this be closed? — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/oracle/adb-free/issues/22*issuecomment-2364159189__;Iw!!Cg_6rE7FVGHU6vd7!4qgoi8HBvX10sUKC0pd7vTU9qKuVKrhKHNfZXj9kcW9c7B3CLlZNd3NWLGl6OqmMSDiySGMjb2owPFSWH3VUWY7UtaHgHA$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ASYZJAURHAGEU2Q565IITX3ZXRJBHAVCNFSM6AAAAABIEJEC4GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRUGE2TSMJYHE__;!!Cg_6rE7FVGHU6vd7!4qgoi8HBvX10sUKC0pd7vTU9qKuVKrhKHNfZXj9kcW9c7B3CLlZNd3NWLGl6OqmMSDiySGMjb2owPFSWH3VUWY47VZW-Rw$>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>