Skip to content

DRIVERS-2888 Support QE with Client.bulkWrite #1770

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 19 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions source/client-side-encryption/limits/limits-qe-doc.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"foo": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
}
37 changes: 29 additions & 8 deletions source/client-side-encryption/tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -565,10 +565,13 @@ First, perform the setup.
2. Using `client`, drop and create the collection `db.coll` configured with the included JSON schema
[limits/limits-schema.json](../limits/limits-schema.json).

3. Using `client`, drop the collection `keyvault.datakeys`. Insert the document
3. Using `client`, drop and create the collection `db.coll2` configured with the included encryptedFields
[limits/limits-encryptedFields.json](../limits/limits-encryptedFields.json).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm missing this file, can you add it to the PR?


4. Using `client`, drop the collection `keyvault.datakeys`. Insert the document
[limits/limits-key.json](../limits/limits-key.json)

4. Create a MongoClient configured with auto encryption (referred to as `client_encrypted`)
5. Create a MongoClient configured with auto encryption (referred to as `client_encrypted`)

Configure with the `local` KMS provider as follows:

Expand All @@ -580,27 +583,27 @@ First, perform the setup.

Using `client_encrypted` perform the following operations:

1. Insert `{ "_id": "over_2mib_under_16mib", "unencrypted": <the string "a" repeated 2097152 times> }`.
1. Insert `{ "_id": "over_2mib_under_16mib", "unencrypted": <the string "a" repeated 2097152 times> }` into `coll`.

Expect this to succeed since this is still under the `maxBsonObjectSize` limit.

2. Insert the document [limits/limits-doc.json](../limits/limits-doc.json) concatenated with
`{ "_id": "encryption_exceeds_2mib", "unencrypted": < the string "a" repeated (2097152 - 2000) times > }` Note:
`{ "_id": "encryption_exceeds_2mib", "unencrypted": < the string "a" repeated (2097152 - 2000) times > }` into `coll`. Note:
limits-doc.json is a 1005 byte BSON document that encrypts to a ~10,000 byte document.

Expect this to succeed since after encryption this still is below the normal maximum BSON document size. Note, before
auto encryption this document is under the 2 MiB limit. After encryption it exceeds the 2 MiB limit, but does NOT
exceed the 16 MiB limit.

3. Bulk insert the following:
3. Use MongoCollection.bulkWrite to insert the following into `coll`:

- `{ "_id": "over_2mib_1", "unencrypted": <the string "a" repeated (2097152) times> }`
- `{ "_id": "over_2mib_2", "unencrypted": <the string "a" repeated (2097152) times> }`

Expect the bulk write to succeed and split after first doc (i.e. two inserts occur). This may be verified using
[command monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.md).

4. Bulk insert the following:
4. Use MongoCollection.bulkWrite insert the following into `coll`:

- The document [limits/limits-doc.json](../limits/limits-doc.json) concatenated with
`{ "_id": "encryption_exceeds_2mib_1", "unencrypted": < the string "a" repeated (2097152 - 2000) times > }`
Expand All @@ -610,15 +613,33 @@ Using `client_encrypted` perform the following operations:
Expect the bulk write to succeed and split after first doc (i.e. two inserts occur). This may be verified using
[command logging and monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.md).

5. Insert `{ "_id": "under_16mib", "unencrypted": <the string "a" repeated 16777216 - 2000 times>`.
5. Insert `{ "_id": "under_16mib", "unencrypted": <the string "a" repeated 16777216 - 2000 times>` into `coll`.

Expect this to succeed since this is still (just) under the `maxBsonObjectSize` limit.

6. Insert the document [limits/limits-doc.json](../limits/limits-doc.json) concatenated with
`{ "_id": "encryption_exceeds_16mib", "unencrypted": < the string "a" repeated (16777216 - 2000) times > }`
`{ "_id": "encryption_exceeds_16mib", "unencrypted": < the string "a" repeated (16777216 - 2000) times > }` into `coll`.

Expect this to fail since encryption results in a document exceeding the `maxBsonObjectSize` limit.

7. Use MongoClient.bulkWrite to insert the following into `coll2`:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two tests should note that an 8.0+ server is required for bulkWrite support


- `{ "_id": "over_2mib_3", "unencrypted": <the string "a" repeated (2097152 - 1500) times> }`
- `{ "_id": "over_2mib_4", "unencrypted": <the string "a" repeated (2097152 - 1500) times> }`

Expect the bulk write to succeed and split after first doc (i.e. two inserts occur). This may be verified using
[command logging and monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.md).

8. Use MongoClient.bulkWrite to insert the following into `coll2`:

- The document [limits/limits-qe-doc.json](../limits/limits-qe-doc.json) concatenated with
`{ "_id": "encryption_exceeds_2mib_3", "foo": < the string "a" repeated (2097152 - 2000 - 1500) times > }`
- The document [limits/limits-qe-doc.json](../limits/limits-qe-doc.json) concatenated with
`{ "_id": "encryption_exceeds_2mib_4", "foo": < the string "a" repeated (2097152 - 2000 - 1500) times > }`

Expect the bulk write to succeed and split after first doc (i.e. two inserts occur). This may be verified using
[command logging and monitoring](../../command-logging-and-monitoring/command-logging-and-monitoring.md).

Optionally, if it is possible to mock the maxWriteBatchSize (i.e. the maximum number of documents in a batch) test that
setting maxWriteBatchSize=1 and inserting the two documents `{ "_id": "a" }, { "_id": "b" }` with `client_encrypted`
splits the operation into two inserts.
Expand Down
9 changes: 2 additions & 7 deletions source/crud/bulk-write.md
Original file line number Diff line number Diff line change
Expand Up @@ -634,13 +634,6 @@ write concern containing the following message:

> Cannot request unacknowledged write concern and ordered writes

## Auto-Encryption

If `MongoClient.bulkWrite` is called on a `MongoClient` configured with `AutoEncryptionOpts`, drivers MUST return an
error with the message: "bulkWrite does not currently support automatic encryption".

This is expected to be removed once [DRIVERS-2888](https://jira.mongodb.org/browse/DRIVERS-2888) is implemented.

## Command Batching

Drivers MUST accept an arbitrary number of operations as input to the `MongoClient.bulkWrite` method. Because the server
Expand Down Expand Up @@ -917,6 +910,8 @@ error in this specific situation does not seem helpful enough to require size ch

## **Changelog**

- 2025-04-23: Removed the requirement to error when QE is enabled.

- 2024-11-05: Updated the requirements regarding the size validation.

- 2024-10-07: Error if `w:0` is used with `ordered=true` or `verboseResults=true`.
Expand Down
Loading