DRIVERS-1541 Retry KMS decrypt requests on transient errors (#1586)

Author: adriandole

source/client-side-encryption/client-side-encryption.md

@@ -2384,6 +2384,8 @@ explicit session parameter as described in the [Drivers Sessions Specification](
 
 ## Changelog
 
+- 2024-10-09: Add retry prose test.
+
 - 2024-07-29: Document range as stable.
 
 - 2024-07-22: Make `trimFactor` and `sparsity` optional.

source/client-side-encryption/tests/README.md

@@ -3290,3 +3290,109 @@ Assert the returned payload size is greater than the size of `payload_defaults`.
 > [!NOTE]
 > Do not compare the payload contents. The payloads include random data. The `trimFactor` and `sparsity` directly affect
 > the payload size.
+
+### 24. KMS Retry Tests
+
+The following tests that certain AWS, Azure, and GCP KMS operations are retried on transient errors.
+
+This test uses a mock server with configurable failpoints to simulate network failures. To start the server:
+
+```
+python -u kms_failpoint_server.py --port 9003
+```
+
+See the [TLS tests](#10-kms-tls-tests) for running the mock server on Evergreen. See
+[the mock server implementation](https://github.com/mongodb-labs/drivers-evergreen-tools/blob/4ba50d373652b6fb39239745664637e33e2b01e6/.evergreen/csfle/kms_failpoint_server.py)
+and the
+[C driver tests](https://github.com/mongodb/mongo-c-driver/blob/d934cd5de55af65220816e4fd01ce3f9c0ef1cd4/src/libmongoc/tests/test-mongoc-client-side-encryption.c#L6295)
+for how to configure failpoints.
+
+#### Setup
+
+1. Start a `mongod` process with **server version 4.2.0 or later**.
+2. Start the failpoint KMS server with: `python -u kms_failpoint_server.py --port 9003`.
+3. Create a `MongoClient` for key vault operations.
+4. Create a `ClientEncryption` object (referred to as `client_encryption`) with `keyVaultNamespace` set to
+   `keyvault.datakeys`.
+
+The failpoint server is configured using HTTP requests. Example request to simulate a network failure:
+
+`curl -X POST https://localhost:9003/set_failpoint/network -d '{"count": 1}' --cacert drivers-evergreen-tools/.evergreen/x509gen/ca.pem`
+
+To simulate an HTTP failure, replace `network` with `http`.
+
+When the following test cases request setting `masterKey`, use the following values based on the KMS provider:
+
+For "aws":
+
+```javascript
+{
+   "region": "foo",
+   "key": "bar",
+   "endpoint": "127.0.0.1:9003",
+}
+```
+
+For "azure":
+
+```javascript
+{
+   "keyVaultEndpoint": "127.0.0.1:9003",
+   "keyName": "foo",
+}
+```
+
+For "gcp":
+
+```javascript
+{
+   "projectId": "foo",
+   "location": "bar",
+   "keyRing": "baz",
+   "keyName": "qux",
+   "endpoint": "127.0.0.1:9003"
+}
+```
+
+#### Case 1: createDataKey and encrypt with TCP retry
+
+1. Configure the mock server to simulate one network failure.
+2. Call `client_encryption.createDataKey()` with "aws" as the provider. Expect this to succeed. Store the returned key
+   ID in a variable named `keyId`.
+3. Configure the mock server to simulate another network failure.
+4. Call `clientEncryption.encrypt` with the following `EncryptOpts` to encrypt the int32 value `123` with the newly
+   created key:
+   ```typescript
+   class EncryptOpts {
+      keyId : <keyID>,
+      algorithm: "AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic",
+   }
+   ```
+   Expect this to succeed.
+
+Repeat this test with the `azure` and `gcp` masterKeys.
+
+#### Case 2: createDataKey and encrypt with HTTP retry
+
+1. Configure the mock server to simulate one HTTP failure.
+2. Call `client_encryption.createDataKey()` with "aws" as the provider. Expect this to succeed. Store the returned key
+   ID in a variable named `keyId`.
+3. Configure the mock server to simulate another HTTP failure.
+4. Call `clientEncryption.encrypt` with the following `EncryptOpts` to encrypt the int32 value `123` with the newly
+   created key:
+   ```typescript
+   class EncryptOpts {
+      keyId : <keyID>,
+      algorithm: "AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic",
+   }
+   ```
+   Expect this to succeed.
+
+Repeat this test with the `azure` and `gcp` masterKeys.
+
+#### Case 3: createDataKey fails after too many retries
+
+1. Configure the mock server to simulate four network failures.
+2. Call `client_encryption.createDataKey()` with "aws" as the provider. Expect this to fail.
+
+Repeat this test with the `azure` and `gcp` masterKeys.