CDRIVER-4801 support named KMS providers (#1509)

* copy in new unified tests

* copy in new legacy spec test

* add new KMS providers to test runner

add partial support for JSON schema 1.18

* implement `encrypt` and `decrypt` operations in unified test runner

* export env vars in run-tests.sh

* update prose test 11 for named KMS providers

* add map for TLS options

Required to configure TLS options on named KMS providers

* check out libmongocrypt with changes of MONGOCRYPT-605

* update docs to reflect spec terminology

KMS provider is specified with string `<KMS provider type>` or `<KMS provider type>:<KMS provider name>`

Author: kevinAlbs

.evergreen/scripts/compile-libmongocrypt.sh

@@ -9,20 +9,21 @@ compile_libmongocrypt() {
   # libmongocrypt's kms-message in `src/kms-message`. Run
   # `.evergreen/scripts/kms-divergence-check.sh` to ensure that there is no
   # divergence in the copied files.
-  git clone -q --depth=1 https://github.com/mongodb/libmongocrypt --branch 1.8.0 || return
 
-  # Remove this workaround once fcf2b5b5 is included in the minimum libmongocrypt commit.
+  # TODO: once 1.9.0 is released (containing MONGOCRYPT-605) replace the following with:
+  # git clone -q --depth=1 https://github.com/mongodb/libmongocrypt --branch 1.9.0 || return
   {
-    git -C libmongocrypt fetch -q origin master || return
-    git -C libmongocrypt cherry-pick --no-commit fcf2b5b5 || return
-    git -C libmongocrypt diff HEAD
+    git clone -q https://github.com/mongodb/libmongocrypt || return
+    # Check out commit containing MONGOCRYPT-605
+    git -C libmongocrypt checkout c87cc3489c9a68875ff7fab541154841469991fb
   }
 
   declare -a crypt_cmake_flags=(
     "-DMONGOCRYPT_MONGOC_DIR=${mongoc_dir}"
     "-DBUILD_TESTING=OFF"
     "-DENABLE_ONLINE_TESTS=OFF"
     "-DENABLE_MONGOC=OFF"
+    "-DBUILD_VERSION=1.9.0-pre"
   )
 
   DEBUG="0" \

.evergreen/scripts/run-tests.sh

@@ -95,6 +95,8 @@ if [[ -n "${CLIENT_SIDE_ENCRYPTION}" ]]; then
     export MONGOC_TEST_AWS_TEMP_SESSION_TOKEN="${CSFLE_AWS_TEMP_SESSION_TOKEN:?}"
     export MONGOC_TEST_AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}"
     export MONGOC_TEST_AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}"
+    export MONGOC_TEST_AWSNAME2_SECRET_ACCESS_KEY="${client_side_encryption_awsname2_secret_access_key}"
+    export MONGOC_TEST_AWSNAME2_ACCESS_KEY_ID="${client_side_encryption_awsname2_access_key_id}"
     export MONGOC_TEST_AZURE_TENANT_ID="${client_side_encryption_azure_tenant_id:?}"
     export MONGOC_TEST_AZURE_CLIENT_ID="${client_side_encryption_azure_client_id:?}"
     export MONGOC_TEST_AZURE_CLIENT_SECRET="${client_side_encryption_azure_client_secret:?}"

CONTRIBUTING.md

@@ -247,6 +247,8 @@ For AWS:
 
 * `MONGOC_TEST_AWS_SECRET_ACCESS_KEY=<string>`
 * `MONGOC_TEST_AWS_ACCESS_KEY_ID=<string>`
+* `MONGOC_TEST_AWSNAME2_SECRET_ACCESS_KEY=<string>`
+* `MONGOC_TEST_AWSNAME2_ACCESS_KEY_ID=<string>`
 
 An Azure:
 

src/libmongoc/doc/mongoc_auto_encryption_opts_set_kms_providers.rst

@@ -19,9 +19,13 @@ Parameters
 * ``opts``: The :symbol:`mongoc_auto_encryption_opts_t`
 * ``kms_providers``: A :symbol:`bson_t` containing configuration for an external Key Management Service (KMS).
 
-``kms_providers`` is a BSON document containing configuration for each KMS provider. Currently ``aws``, ``local``, ``azure``, ``gcp``, and ``kmip`` are supported. At least one must be specified.
+``kms_providers`` is a BSON document containing configuration for each KMS provider.
 
-The format for "aws" is as follows:
+KMS providers are specified as a string of the form ``<KMS provider type>`` or ``<KMS provider type>:<KMS provider name>``.
+The supported KMS provider types are ``aws``, ``azure``, ``gcp``, ``local``, and ``kmip``. The optional name enables configuring multiple KMS providers with the same KMS provider type (e.g. ``aws:name1`` and ``aws:name2`` can refer to different AWS accounts).
+At least one KMS provider must be specified.
+
+The format for the KMS provider type ``aws`` is as follows:
 
 .. code-block:: javascript
 
@@ -30,15 +34,15 @@ The format for "aws" is as follows:
       secretAccessKey: String
    }
 
-The format for "local" is as follows:
+The format for the KMS provider type ``local`` is as follows:
 
 .. code-block:: javascript
 
    local: {
       key: <96 byte BSON binary of subtype 0> or String /* The master key used to encrypt/decrypt data keys. May be passed as a base64 encoded string. */
    }
 
-The format for "azure" is as follows:
+The format for the KMS provider type ``azure`` is as follows:
 
 .. code-block:: javascript
 
@@ -49,7 +53,7 @@ The format for "azure" is as follows:
       identityPlatformEndpoint: Optional<String> /* Defaults to login.microsoftonline.com */
    }
 
-The format for "gcp" is as follows:
+The format for the KMS provider type ``gcp`` is as follows:
 
 .. code-block:: javascript
 
@@ -59,14 +63,27 @@ The format for "gcp" is as follows:
       endpoint: Optional<String> /* Defaults to oauth2.googleapis.com */
    }
 
-The format for "kmip" is as follows:
+The format for the KMS provider type ``kmip`` is as follows:
 
 .. code-block:: javascript
 
    kmip: {
       endpoint: String
    }
 
+KMS providers may include an optional name suffix separate with a colon. This enables configuring multiple KMS providers with the same KMS provider type. Example:
+
+.. code-block:: javascript
+
+   "aws:name1": {
+      accessKeyId: String,
+      secretAccessKey: String
+   },
+   "aws:name2": {
+      accessKeyId: String,
+      secretAccessKey: String
+   }   
+
 .. seealso::
 
   | :symbol:`mongoc_client_enable_auto_encryption()`

src/libmongoc/doc/mongoc_auto_encryption_opts_set_tls_opts.rst

@@ -17,19 +17,19 @@ Parameters
 ----------
 
 * ``opts``: The :symbol:`mongoc_auto_encryption_opts_t`
-* ``tls_opts``: A :symbol:`bson_t` mapping a Key Management Service (KMS) provider name to a BSON document with TLS options.
+* ``tls_opts``: A :symbol:`bson_t` mapping a Key Management Service (KMS) provider to a BSON document with TLS options.
 
 ``tls_opts`` is a BSON document of the following form:
 
 .. code-block:: javascript
 
-   <KMS provider name>: {
+   <KMS provider>: {
       tlsCaFile: Optional<String>
       tlsCertificateKeyFile: Optional<String>
       tlsCertificateKeyFilePassword: Optional<String>
    }
 
-The KMS providers ``aws``, ``azure``, ``gcp``, and ``kmip`` are supported as keys in the ``tls_opts`` document.
+The KMS providers ``aws``, ``azure``, ``gcp``, and ``kmip`` are supported as keys in the ``tls_opts`` document. They may include an optional name suffix separated with a colon. Example: ``aws:name2``.
 
 ``tls_opts`` maps the KMS provider name to a BSON document for TLS options.
 

src/libmongoc/doc/mongoc_client_encryption_datakey_opts_set_masterkey.rst

@@ -23,11 +23,11 @@ Parameters
 Description
 -----------
 
-Setting the masterkey is required when creating a data key with the KMS providers: ``aws``, ``azure``, ``gcp``, and ``kmip``.
+Setting the masterkey is required when creating a data key with the KMS provider types: ``aws``, ``azure``, ``gcp``, and ``kmip``.
 
-Setting the masterkey is prohibited with the KMS provider ``local``.
+Setting the masterkey is prohibited with the KMS provider type ``local``.
 
-The format of ``masterkey`` for "aws" is as follows:
+The format of ``masterkey`` for the KMS provider type ``aws`` is as follows:
 
 .. code-block:: javascript
 
@@ -37,7 +37,7 @@ The format of ``masterkey`` for "aws" is as follows:
       endpoint: Optional<String> /* An alternate host identifier to send KMS requests to. May include port number. Defaults to "kms.<region>.amazonaws.com" */
    }
 
-The format of ``masterkey`` for "azure" is as follows:
+The format of ``masterkey`` for the KMS provider type ``azure`` is as follows:
 
 .. code-block:: javascript
 
@@ -47,7 +47,7 @@ The format of ``masterkey`` for "azure" is as follows:
       keyVersion: Optional<String> /* A specific version of the named key, defaults to using the key's primary version. */
    }
 
-The format of ``masterkey`` for "gcp" is as follows:
+The format of ``masterkey`` for the KMS provider type ``gcp`` is as follows:
 
 .. code-block:: javascript
 
@@ -60,11 +60,11 @@ The format of ``masterkey`` for "gcp" is as follows:
       endpoint: Optional<String> /* Host with optional port. Defaults to "cloudkms.googleapis.com". */
    }
 
-The format of ``masterkey`` for "kmip" is as follows:
+The format of ``masterkey`` for the KMS provider type ``kmip`` is as follows:
 
 .. code-block:: javascript
 
    {
       keyId: Optional<String>,
       endpoint: Optional<String> /* Host with optional port. */
-   }
+   }

src/libmongoc/doc/mongoc_client_encryption_opts_set_kms_providers.rst

@@ -18,9 +18,13 @@ Parameters
 * ``opts``: The :symbol:`mongoc_client_encryption_opts_t`
 * ``kms_providers``: A :symbol:`bson_t` containing configuration for an external Key Management Service (KMS).
 
-``kms_providers`` is a BSON document containing configuration for each KMS provider. Currently ``aws``, ``local``, ``azure``, ``gcp``, and ``kmip`` are supported. At least one must be specified.
+``kms_providers`` is a BSON document containing configuration for each KMS provider.
 
-The format for "aws" is as follows:
+KMS providers are specified as a string of the form ``<KMS provider type>`` or ``<KMS provider type>:<KMS provider name>``.
+The supported KMS provider types are ``aws``, ``azure``, ``gcp``, ``local``, and ``kmip``. The optional name enables configuring multiple KMS providers with the same KMS provider type (e.g. ``aws:name1`` and ``aws:name2`` can refer to different AWS accounts).
+At least one KMS provider must be specified.
+
+The format for the KMS provider type ``aws`` is as follows:
 
 .. code-block:: javascript
 
@@ -29,15 +33,15 @@ The format for "aws" is as follows:
       secretAccessKey: String
    }
 
-The format for "local" is as follows:
+The format for the KMS provider type ``local`` is as follows:
 
 .. code-block:: javascript
 
    local: {
       key: <96 byte BSON binary of subtype 0> or String /* The master key used to encrypt/decrypt data keys. May be passed as a base64 encoded string. */
    }
 
-The format for "azure" is as follows:
+The format for the KMS provider type ``azure`` is as follows:
 
 .. code-block:: javascript
 
@@ -48,7 +52,7 @@ The format for "azure" is as follows:
       identityPlatformEndpoint: Optional<String> /* Defaults to login.microsoftonline.com */
    }
 
-The format for "gcp" is as follows:
+The format for the KMS provider type ``gcp`` is as follows:
 
 .. code-block:: javascript
 
@@ -58,14 +62,26 @@ The format for "gcp" is as follows:
       endpoint: Optional<String> /* Defaults to oauth2.googleapis.com */
    }
 
-The format for "kmip" is as follows:
+The format for the KMS provider type ``kmip`` is as follows:
 
 .. code-block:: javascript
 
    kmip: {
       endpoint: String
    }
 
+KMS providers may include an optional name suffix separate with a colon. This enables configuring multiple KMS providers with the same KMS provider type. Example:
+
+.. code-block:: javascript
+
+   "aws:name1": {
+      accessKeyId: String,
+      secretAccessKey: String
+   },
+   "aws:name2": {
+      accessKeyId: String,
+      secretAccessKey: String
+   }   
 
 .. seealso::
 

src/libmongoc/doc/mongoc_client_encryption_opts_set_tls_opts.rst

@@ -17,21 +17,22 @@ Parameters
 ----------
 
 * ``opts``: The :symbol:`mongoc_client_encryption_opts_t`
-* ``tls_opts``: A :symbol:`bson_t` mapping a Key Management Service (KMS) provider name to a BSON document with TLS options.
+* ``tls_opts``: A :symbol:`bson_t` mapping a Key Management Service (KMS) provider to a BSON document with TLS options.
 
 ``tls_opts`` is a BSON document of the following form:
 
 .. code-block:: javascript
 
-   <KMS provider name>: {
+   <KMS provider>: {
       tlsCaFile: Optional<String>
       tlsCertificateKeyFile: Optional<String>
       tlsCertificateKeyFilePassword: Optional<String>
    }
 
-The KMS providers ``aws``, ``azure``, ``gcp``, and ``kmip`` are supported as keys in the ``tls_opts`` document.
+KMS providers are specified as a string of the form ``<KMS provider type>`` or ``<KMS provider type>:<KMS provider name>``.
+The supported KMS provider types are ``aws``, ``azure``, ``gcp``, ``local``, and ``kmip``. The optional name enables configuring multiple KMS providers with the same KMS provider type (e.g. ``aws:name1`` and ``aws:name2`` can refer to different AWS accounts).
 
-``tls_opts`` maps the KMS provider name to a BSON document for TLS options.
+``tls_opts`` maps the KMS provider to a BSON document for TLS options.
 
 The BSON document for TLS options may contain the following keys: