Skip to content

docs: add documentation for running aws integ tests locally #4830

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
PavelSafronov wants to merge 4 commits into
base: main
Choose a base branch
from
+64 −0
Open

docs: add documentation for running aws integ tests locally #4830

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
576e8c6
doc: add documentation for running aws integ tests locally
PavelSafronov Dec 26, 2025
5587eba
Merge branch 'main' into local-aws
PavelSafronov Jan 7, 2026
04d1fc1
Merge branch 'main' into local-aws
PavelSafronov Jan 7, 2026
971f85d
Merge branch 'main' into local-aws
PavelSafronov Jan 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .evergreen/run-mongodb-aws-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,7 @@ source .evergreen/setup-mongodb-aws-auth-tests.sh
# load node.js environment
source $DRIVERS_TOOLS/.evergreen/init-node-and-npm-env.sh

# clear AWS_PROFILE, so it doesn't interfere with env-creds
unset AWS_PROFILE
Copy link
Contributor

@baileympearson baileympearson Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this environment variable used by any AWS auth variants in CI?

aWs there a particular reason this change was needed? iirc the AWS credential provider SDK fetches credentials from the environment first. So it's unclear how having the AWS_PROFILE set would interfere with AWS environment variable credential fetching.

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/Package/-aws-sdk-credential-providers/#:~:text=This%20credential%20provider%20will,exposed%20via%20process.env

Copy link
Contributor Author

@PavelSafronov PavelSafronov Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This variable is not used by any AWS auth variants in our CI.

The tests fetch the credentials from the environment, and this is fine in CI where we use env access/secret keys directly: we use the env creds to generate temporary/EC2/whatever credentials, then the new access/secret/token values overwrite the env vars, and the SDK picks up the newest values.

But running these locally with a profile configured I saw a different behavior: we use the profile to generate temporary/EC2/whatever credentials, then the new access/secret/token values are added to the env, in addition to the profile. When this happens, our tests fail with an auth error (you can verify this by commenting out the line unset AWS_PROFILE) :

> mongodb@7.0.0 check:aws
> nyc mocha --config test/mocha_mongodb.js test/integration/auth/mongodb_aws.test.ts test/integration/auth/mongodb_aws.prose.test.ts

[baseline-browser-mapping] The data in this module is over two months old.  To ensure accurate Baseline data, please update: `npm i baseline-browser-mapping@latest -D`


@aws-sdk/credential-provider-node - defaultProvider::fromEnv WARNING:
    Multiple credential sources detected: 
    Both AWS_PROFILE and the pair AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY static credentials are set.
    This SDK will proceed with the AWS_PROFILE value.
    
    However, a future version may change this behavior to prefer the ENV static credentials.
    Please ensure that your environment only sets either the AWS_PROFILE or the
    AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY pair.

  1) "before all" hook: testConfigBeforeHook in "{root}"

  0 passing (1s)
  1 failing

  1) "before all" hook: testConfigBeforeHook in "{root}":
     MongoServerError: Authentication failed.
      at Connection.sendCommand (src/cmap/connection.ts:168:71)
      at processTicksAndRejections (node:internal/process/task_queues:103:5)
      at async Connection.command (src/cmap/connection.ts:168:71)
      at async MongoDBAWS.auth (src/cmap/auth/mongodb_aws.ts:17:45)
      at async performInitialHandshake (src/cmap/connect.ts:7:53)
      at async connect (src/cmap/connect.ts:2:1757)

Compare this to a run where the AWS_PROFILE is unset:

> mongodb@7.0.0 check:aws
> nyc mocha --config test/mocha_mongodb.js test/integration/auth/mongodb_aws.test.ts test/integration/auth/mongodb_aws.prose.test.ts

[baseline-browser-mapping] The data in this module is over two months old.  To ensure accurate Baseline data, please update: `npm i baseline-browser-mapping@latest -D`


{
  topology: 'Single',
  version: '8.3.0-alpha0-4662-g7385039',
  node: 'v24.12.0',
  os: 'darwin',
  alpineLinux: false,
  cryptdUri: undefined,
  pid: 15263,
  auth: true,
  tls: false,
  csfle: {
    enabled: false,
    mongodbClientEncryption: {
      cryptoCallbacks: [Getter],
      MongoCryptContext: [class MongoCryptContext],
      MongoCrypt: [Function]
    },
    version: '7.0.0',
    libmongocrypt: '1.15.1'
  },
  cryptSharedVersion: null,
  cryptSharedLibPath: undefined,
  serverApi: undefined,
  atlas: false,
  aws: true,
  awsSdk: undefined,
  azure: false,
  kerberos: false,
  ldap: false,
  socks5: false,
  compressor: undefined,
  zstdVersion: '7.0.0'
}

====================================================================================================
 Flaky test: "Change Streams should properly handle a changeStream event being processed mid-close when invoked with promises" is not run at all 
====================================================================================================

  MONGODB-AWS Prose Tests
    1. Custom Credential Provider Authenticates
      ✔ authenticates with a user provided credentials provider (578ms)
    2. Custom Credential Provider Authentication Precedence
      Case 2: Custom Provider Takes Precedence Over Environment Variables
        ✔ authenticates with a user provided credentials provider (480ms)

  MONGODB-AWS
    when the AWS SDK is not present
      when attempting AWS auth
        ✔ throws an error
    when the AWS SDK is present
      ✔ should authorize when successfully authenticated (560ms)
      ✔ should allow empty string in authMechanismProperties.AWS_SESSION_TOKEN to override AWS_SESSION_TOKEN environment variable
      ✔ should store a MongoDBAWS provider instance per client (526ms)
      ConversationId
        ✔ should use conversationId returned by saslStart in saslContinue
      with missing aws token
        ✔ should not throw an exception when aws token is missing (622ms)
      EC2 with missing credentials
        - should respect the default timeout of 10000ms
          ↬ requires an AWS EC2 environment
    when using AssumeRoleWithWebIdentity
      when no AWS region settings are set
        - uses the default region
        - fromNodeProviderChain called once
      when only AWS_STS_REGIONAL_ENDPOINTS is set
        - uses the default region
        - fromNodeProviderChain called once
      when only AWS_REGION is set
        - uses the default region
        - fromNodeProviderChain called once
      when AWS_STS_REGIONAL_ENDPOINTS is set to regional and region is legacy
        - uses the region from the environment
        - fromNodeProviderChain called once
      when AWS_STS_REGIONAL_ENDPOINTS is set to regional and region is new
        - uses the region from the environment
        - fromNodeProviderChain called once
      when AWS_STS_REGIONAL_ENDPOINTS is set to legacy and region is legacy
        - uses the region from the environment
        - fromNodeProviderChain called once
      when AWS_STS_REGIONAL_ENDPOINTS is set to legacy and region is new
        - uses the default region
        - fromNodeProviderChain called once
    AWS KMS Credential Fetching
      when the AWS SDK is not installed
        ✔ fetching AWS KMS credentials throws an error
      when the AWS SDK is installed
        ✔ does not return any extra keys for the `aws` credential provider
        when a credential provider is not provided
          ✔ KMS credentials are successfully fetched.
        when a credential provider is provided
          ✔ KMS credentials are successfully fetched.


  12 passing (3s)
  15 pending

wrote xunit.xml

=============================== Coverage summary ===============================
Statements   : 40.05% ( 4184/10445 )
Branches     : 23.8% ( 1081/4541 )
Functions    : 26.96% ( 518/1921 )
Lines        : 40.43% ( 4096/10131 )
================================================================================


npm run check:aws
33 changes: 33 additions & 0 deletions etc/run-aws-integ-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/usr/bin/env bash
set -euxo pipefail # Exit on error, undefined variable, and fail on pipe errors

Copy link
Contributor

@baileympearson baileympearson Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally, if we add a script that we plan to use for local testing, we also ensure that it is the same script used to run tests in CI. This ensures the script is always up-to-date.

  • Is there something this script does that isn't captured by our testing readme's instructions?
  • Do you think it's worth refactoring CI to use this script?

Copy link
Contributor Author

@PavelSafronov PavelSafronov Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't feel too strongly about including this script. The way we run tests in CI makes sense and I don't think we need to change it.
This was mostly my own experimentation to figure out a way to simply run AWS tests locally. Can definitely drop it from the PR.

# NOTE: This script assumes that you've created an AWS SSO session already, as outlined in
# ./test/readme.md, section `AWS Profile`, and you have an AWS profile
Copy link
Contributor

@baileympearson baileympearson Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# ./test/readme.md, section `AWS Profile`, and you have an AWS profile
# ../test/readme.md, section `AWS Profile`, and you have an AWS profile

# named `drivers-test-secrets-role-857654397073` in your AWS config.

# Choose credential type: env-creds or session-creds
# export AWS_CREDENTIAL_TYPE="env-creds"
export AWS_CREDENTIAL_TYPE="session-creds"
export VERSION="latest"
export NODE_LTS_VERSION="24"
export AUTH="auth"
export ORCHESTRATION_FILE="auth-aws.json"
export TOPOLOGY="server"
export NODE_DRIVER="$DRIVERS_TOOLS/.."
export AWS_PROFILE="drivers-test-secrets-role-857654397073"

# Enable for verbose logging
# export MONGODB_LOG_ALL="debug"
# export MONGODB_LOG_PATH="stderr"

echo "Assuming AWS SSO role..."
aws sso login --sso-session drivers-test-secrets-session

echo "Installing dependencies..."
bash ${NODE_DRIVER}/.evergreen/install-dependencies.sh

echo "Bootstrapping orchestration..."
bash ${NODE_DRIVER}/.evergreen/run-orchestration.sh

echo "Running AWS integration tests with env-creds from $NODE_DRIVER ..."
bash ${NODE_DRIVER}/.evergreen/run-mongodb-aws-test.sh
28 changes: 28 additions & 0 deletions test/readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ about the types of tests and how to run them.
- [Deployed Lambda Tests](#deployed-lambda-tests)
- [Kerberos Tests](#kerberos-tests)
- [AWS Authentication tests](#aws-authentication-tests)
- [AWS Profile](#aws-profile)
- [Container Tests](#container-tests)
- [GCP](#gcp)
- [Azure](#azure)
Expand Down Expand Up @@ -647,6 +648,33 @@ Choose your AWS authentication credential type and export the `AWS_CREDENTIAL_TY

1. Run the `bash .evergreen/run-mongodb-aws-tests.sh`.

An example of performing the above is [`etc/run-aws-integ-tests.sh`](etc/run-aws-integ-tests.sh).

#### AWS Profile

Setup an AWS_PROFILE locally to be able to use AWS and to run AWS tests locally.

1. Get SSO sign-in info from AWS
Copy link
Contributor

@baileympearson baileympearson Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these steps just a duplicate of the docs here? https://wiki.corp.mongodb.com/spaces/DRIVERS/pages/239737385/Using+AWS+Secrets+Manager+to+Store+Testing+Secrets

If so, I'd suggest just linking these docs instead

Copy link
Contributor

@baileympearson baileympearson Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, we do link to these docs (indirectly) in the secrets section of the readme

Copy link
Contributor Author

@PavelSafronov PavelSafronov Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The instructions are indirectly linked, so it did take a bit of experimentation to figure out what was necessary or not. I think it may be helpful to be explicit about these instructions, hence why I included them.

What about adding a link to the wiki, but keeping this section as "quick setup" instructions?

1. Navigate to https://corp.mongodb.com/app/UserHome
2. Open AWS
3. Choose `Drivers` account
4. Choose `drivers-test-secrets-role`
5. Click `Access Keys`
6. Copy down `SSO start URL` and `SSO Region`
2. Sign in locally
1. Run `aws configure sso-session`
2. Pick a name, like `drivers-test-secrets-session`
3. Specify `SSO start URL` and `SSO Region` from earlier steps
3. Add a profile
1. Add the following profile to `~/.aws/config`

```ini
[profile drivers-test-secrets-role-857654397073]
sso_session = drivers-test-secrets-session
sso_account_id = 857654397073
sso_role_name = drivers-test-secrets-role
```

### Container Tests

It may become required to run tests or debug code inside a live Azure or GCP container. The best way to do this is to leverage
Expand Down