SNOW-1857047: Timeout issues on lambda (#2337)

Author: sfc-gh-fpawlowski

.github/workflows/build_test.yml

@@ -7,20 +7,46 @@ https://docs.snowflake.com/
 Source code is also available at: https://github.com/snowflakedb/snowflake-connector-python
 
 # Release Notes
-- v3.14.1(TBD)
+- v3.16(TBD)
+  - Bumped numpy dependency from <2.1.0 to <=2.2.4
+  - Added Windows support for Python 3.13.
+
+- v3.15.1(May 20, 2025)
+  - Added basic arrow support for Interval types.
+  - Fix `write_pandas` special characters usage in the location name.
+  - Fix usage of `use_virtual_url` when building the location for gcs storage client.
+  - Bind cryptography to <=44.0.3 to avoid issues with 45.0.0.
+
+- v3.15.0(Apr 29,2025)
+  - Bumped up min boto and botocore version to 1.24.
+  - OCSP: terminate certificates chain traversal if a trusted certificate already reached.
+  - Added new authentication methods support for programmatic access tokens (PATs), OAuth 2.0 Authorization Code Flow, OAuth 2.0 Client Credentials Flow, and OAuth Token caching.
+    - For OAuth 2.0 Authorization Code Flow:
+      - Added the `oauth_client_id`, `oauth_client_secret`, `oauth_authorization_url`, `oauth_token_request_url`, `oauth_redirect_uri`, `oauth_scope`, `oauth_disable_pkce`, `oauth_enable_refresh_tokens` and `oauth_enable_single_use_refresh_tokens` parameters.
+      - Added the `OAUTH_AUTHORIZATION_CODE` value for the parameter authenticator.
+    - For OAuth 2.0 Client Credentials Flow:
+      - Added the `oauth_client_id`, `oauth_client_secret`, `oauth_token_request_url`, and `oauth_scope` parameters.
+      - Added the `OAUTH_CLIENT_CREDENTIALS` value for the parameter authenticator.
+    - For OAuth Token caching: Passing a username to driver configuration is required, and the `client_store_temporary_credential property` is to be set to `true`.
+
+- v3.14.1(April 21, 2025)
   - Added support for Python 3.13.
     - NOTE: Windows 64 support is still experimental and should not yet be used for production environments.
   - Dropped support for Python 3.8.
-  - Basic decimal floating-point type support.
-  - Added handling of PAT provided in `password` field.
-  - Improved error message for client-side query cancellations due to timeouts.
+  - Added basic decimal floating-point type support.
+  - Added experimental authentication methods.
   - Added support of GCS regional endpoints.
-  - Added `gcs_use_virtual_endpoints` connection property that forces the usage of the virtual GCS usage. Thanks to this it should be possible to set up private DNS entry for the GCS endpoint. See more: https://cloud.google.com/storage/docs/request-endpoints#xml-api
-  - Fixed a bug that caused driver to fail silently on `TO_DATE` arrow to python conversion when invalid date was followed by the correct one.
+  - Added support of GCS virtual urls. See more: https://cloud.google.com/storage/docs/request-endpoints#xml-api
+  - Added `client_fetch_threads` experimental parameter to better utilize threads for fetching query results.
   - Added `check_arrow_conversion_error_on_every_column` connection property that can be set to `False` to restore previous behaviour in which driver will ignore errors until it occurs in the last column. This flag's purpose is to unblock workflows that may be impacted by the bugfix and will be removed in later releases.
-  - Lower log levels from info to debug for some of the messages to make the output easier to follow.
-  - Allow the connector to inherit a UUID4 generated upstream, provided in statement parameters (field: `requestId`), rather than automatically generate a UUID4 to use for the HTTP Request ID.
+  - Lowered log levels from info to debug for some of the messages to make the output easier to follow.
+  - Allowed the connector to inherit a UUID4 generated upstream, provided in statement parameters (field: `requestId`), rather than automatically generate a UUID4 to use for the HTTP Request ID.
   - Improved logging in urllib3, boto3, botocore - assured data masking even after migration to the external owned library in the future.
+  - Improved error message for client-side query cancellations due to timeouts.
+  - Improved security and robustness for the temporary credentials cache storage.
+  - Fixed a bug that caused driver to fail silently on `TO_DATE` arrow to python conversion when invalid date was followed by the correct one.
+  - Fixed expired S3 credentials update and increment retry when expired credentials are found.
+  - Deprecated `insecure_mode` connection property and replaced it with `disable_ocsp_checks` with the same behavior as the former property.
 
 - v3.14.0(March 03, 2025)
   - Bumped pyOpenSSL dependency upper boundary from <25.0.0 to <26.0.0.
@@ -32,7 +58,6 @@ Source code is also available at: https://github.com/snowflakedb/snowflake-conne
   - Fixed a bug where file permission check happened on Windows.
   - Added support for File types.
   - Added `unsafe_file_write` connection parameter that restores the previous behaviour of saving files downloaded with GET with 644 permissions.
-  - Deprecated `insecure_mode` connection property and replaced it with `disable_ocsp_checks` with the same behavior as the former property.
 
 - v3.13.2(January 29, 2025)
   - Changed not to use scoped temporary objects.

.github/workflows/parameters/private/parameters_aws_auth_tests.json.gpg

@@ -35,29 +35,46 @@ timestamps {
         string(name: 'parent_job', value: env.JOB_NAME),
         string(name: 'parent_build_number', value: env.BUILD_NUMBER)
       ]
-      stage('Test') {
-          try {
-          def commit_hash = "main" // default which we want to override
-          def bptp_tag = "bptp-built"
-          def response = authenticatedGithubCall("https://api.github.com/repos/snowflakedb/snowflake/git/ref/tags/${bptp_tag}")
-          commit_hash = response.object.sha
-          // Append the bptp-built commit sha to params
-          params += [string(name: 'svn_revision', value: commit_hash)]
-          } catch(Exception e) {
-          println("Exception computing commit hash from: ${response}")
+      parallel(
+      'Test': {
+        stage('Test') {
+            try {
+            def commit_hash = "main" // default which we want to override
+            def bptp_tag = "bptp-stable"
+            def response = authenticatedGithubCall("https://api.github.com/repos/snowflakedb/snowflake/git/ref/tags/${bptp_tag}")
+            commit_hash = response.object.sha
+            // Append the bptp-stable commit sha to params
+            params += [string(name: 'svn_revision', value: commit_hash)]
+            } catch(Exception e) {
+            println("Exception computing commit hash from: ${response}")
+            }
+          parallel (
+            'Test Python 39': { build job: 'RT-PyConnector39-PC',parameters: params},
+            'Test Python 310': { build job: 'RT-PyConnector310-PC',parameters: params},
+            'Test Python 311': { build job: 'RT-PyConnector311-PC',parameters: params},
+            'Test Python 312': { build job: 'RT-PyConnector312-PC',parameters: params},
+            'Test Python 313': { build job: 'RT-PyConnector313-PC',parameters: params},
+            'Test Python 39 OldDriver': { build job: 'RT-PyConnector39-OldDriver-PC',parameters: params},
+            'Test Python 39 FIPS': { build job: 'RT-FIPS-PyConnector39',parameters: params},
+            )
+          }
+        },
+      'Test Authentication': {
+        stage('Test Authentication') {
+          withCredentials([
+            string(credentialsId: 'a791118f-a1ea-46cd-b876-56da1b9bc71c', variable: 'NEXUS_PASSWORD'),
+            string(credentialsId: 'sfctest0-parameters-secret', variable: 'PARAMETERS_SECRET')
+          ]) {
+            sh '''\
+            |#!/bin/bash -e
+            |$WORKSPACE/ci/test_authentication.sh
+            '''.stripMargin()
           }
-        parallel (
-          'Test Python 39': { build job: 'RT-PyConnector39-PC',parameters: params},
-          'Test Python 310': { build job: 'RT-PyConnector310-PC',parameters: params},
-          'Test Python 311': { build job: 'RT-PyConnector311-PC',parameters: params},
-          'Test Python 312': { build job: 'RT-PyConnector312-PC',parameters: params},
-          'Test Python 313': { build job: 'RT-PyConnector313-PC',parameters: params},
-          'Test Python 39 OldDriver': { build job: 'RT-PyConnector39-OldDriver-PC',parameters: params},
-          'Test Python 39 FIPS': { build job: 'RT-FIPS-PyConnector39',parameters: params},
-          )
         }
       }
-    }
+    )
+  }
+}
 
 
 pipeline {

.github/workflows/parameters/private/rsa_keys/rsa_key.p8.gpg

@@ -24,6 +24,7 @@ exclude tox.ini
 exclude mypy.ini
 exclude .clang-format
 exclude .wiremock/*
+exclude cmd/prober/testing_matrix.json
 
 prune ci
 prune benchmark

.github/workflows/parameters/private/rsa_keys/rsa_key_invalid.p8.gpg

@@ -0,0 +1,24 @@
+#!/bin/bash -e
+
+set -o pipefail
+
+
+export WORKSPACE=${WORKSPACE:-/mnt/workspace}
+export SOURCE_ROOT=${SOURCE_ROOT:-/mnt/host}
+
+MVNW_EXE=$SOURCE_ROOT/mvnw
+AUTH_PARAMETER_FILE=./.github/workflows/parameters/private/parameters_aws_auth_tests.json
+eval $(jq -r '.authtestparams | to_entries | map("export \(.key)=\(.value|tostring)")|.[]' $AUTH_PARAMETER_FILE)
+
+export SNOWFLAKE_AUTH_TEST_PRIVATE_KEY_PATH=./.github/workflows/parameters/private/rsa_keys/rsa_key.p8
+export SNOWFLAKE_AUTH_TEST_INVALID_PRIVATE_KEY_PATH=./.github/workflows/parameters/private/rsa_keys/rsa_key_invalid.p8
+
+export SF_OCSP_TEST_MODE=true
+export SF_ENABLE_EXPERIMENTAL_AUTHENTICATION=true
+export RUN_AUTH_TESTS=true
+export AUTHENTICATION_TESTS_ENV="docker"
+export PYTHONPATH=$SOURCE_ROOT
+
+python3 -m pip install --break-system-packages -e .
+
+python3 -m pytest test/auth/*

DESCRIPTION.md

@@ -3,8 +3,7 @@ FROM $BASE_IMAGE
 
 RUN yum install -y java-11-openjdk
 
-# TODO: When there are prebuilt wheels accessible for our dependencies (i.e. numpy)
-# for Python 3.13 this rust cargo install command can be removed.
+# Our dependencies rely on the Rust toolchain being available in the build-time environment (https://github.com/pyca/cryptography/issues/5771)
 RUN yum -y install rust cargo
 
 # This is to solve permission issue, read https://denibertovic.com/posts/handling-permissions-with-docker-volumes/

Jenkinsfile

@@ -2,24 +2,13 @@ FROM public.ecr.aws/lambda/python:3.13-x86_64
 
 WORKDIR /home/user/snowflake-connector-python
 
-# TODO: When there are prebuilt wheels accessible for our dependencies (i.e. numpy)
-# for Python 3.13 all dnf ... commands installing building kits can be removed.
-
-# Install necessary packages and compilers - we need to build numpy for newer version
-# Update dnf and install development tools
 RUN dnf -y update && \
-    dnf -y install \
-    gcc \
-    gcc-c++ \
-    make \
-    python3-devel \
-    openblas-devel \
-    lapack-devel && \
     dnf clean all
+
+# Our dependencies rely on the Rust toolchain being available in the build-time environment (https://github.com/pyca/cryptography/issues/5771)
 RUN dnf -y install rust cargo
 RUN dnf -y upgrade
 
-
 RUN chmod 777 /home/user/snowflake-connector-python
 ENV PATH="${PATH}:/opt/python/cp313-cp313/bin/"
 ENV PYTHONPATH="${PYTHONPATH}:/home/user/snowflake-connector-python/ci/docker/connector_test_lambda/"