DRIVERS-2888 Support QE with Client.bulkWrite

[mdb-ad]

Add spec test for client bulkWrite QE.

C driver implementation

libmongocrypt requirement: 1.10
Drivers changes required:

• QE unified test support (DRIVERS-3106)
• bulkWrite specific changes: don't error on QE bulkWrite. Implement batch splitting.

Please complete the following before merging:

• Update changelog.
• Test changes in at least one language driver. (C driver)
• Test these changes against all server versions and topologies (including standalone, replica set, sharded
  clusters, and serverless). (Note: C driver doesn't currently test CSFLE with sharded)

[nbbeeken]

I may be out of the loop on some of the context, are drivers changes needed to get these passing? Is there a specific libmongocrypt version needed?

There seems to be more unified test changes in the C driver than are here, are those just extra test sync(s) or are they new tests?

Node.js Test Failures

  1) CRUD unified
       client bulkWrite with queryable encryption
         client bulkWrite QE replaceOne:
     MongoBulkWriteError: Document failed validation
      at LegacyOrderedBulkOperation.handleWriteError (src/bulk/common.ts:1224:13)
      at executeCommands (src/bulk/common.ts:587:19)
      at processTicksAndRejections (node:internal/process/task_queues:105:5)
      at async BulkWriteShimOperation.execute (src/bulk/common.ts:882:12)
      at async tryOperation (src/operations/execute_operation.ts:283:14)
      at async executeOperation (src/operations/execute_operation.ts:115:12)
      at async LegacyOrderedBulkOperation.execute (src/bulk/common.ts:1211:12)
      at async BulkWriteOperation.execute (src/operations/bulk_write.ts:60:12)
      at async InsertManyOperation.execute (src/operations/insert.ts:147:19)
      at async tryOperation (src/operations/execute_operation.ts:283:14)
      at async executeOperation (src/operations/execute_operation.ts:115:12)
      at async Collection.insertMany (src/collection.ts:307:12)
      at async insertMany (test/tools/unified-spec-runner/operations.ts:411:10)
      at async executeOperationAndCheck (test/tools/unified-spec-runner/operations.ts:1039:14)
      at async runUnifiedTest (test/tools/unified-spec-runner/runner.ts:226:9)
      at async Context.<anonymous> (test/tools/unified-spec-runner/runner.ts:336:13)

  2) CRUD unified
       client bulkWrite with queryable encryption
         client bulkWrite QE with multiple replace fails:
     AssertionError: Operation clientBulkWrite succeeded but was not supposed to
      at executeOperationAndCheck (test/tools/unified-spec-runner/operations.ts:1057:12)
      at processTicksAndRejections (node:internal/process/task_queues:105:5)
      at async runUnifiedTest (test/tools/unified-spec-runner/runner.ts:226:9)
      at async Context.<anonymous> (test/tools/unified-spec-runner/runner.ts:336:13)

[kevinAlbs]

Suggest updating this PR description to give an overview of needed changes. I expect implementors will refer to this PR (and the C driver PR).

C driver implementation

Suggest linking to the C driver PR (once created) rather than the Evergreen patch. The patch appears to include unrelated spec tests.

[mdb-ad]

I may be out of the loop on some of the context, are drivers changes needed to get these passing? Is there a specific libmongocrypt version needed?

libmongocrypt 1.10 is required.

There seems to be more unified test changes in the C driver than are here, are those just extra test sync(s) or are they new tests?

Description updated: the linked C driver changes in the description are bulkWrite-related only

[isabelatkinson]

Tests pass in Rust. Draft PR: mongodb/mongo-rust-driver#1445

There are some updates needed in the bulk write spec to account for the different command-building logic. Here's a commit with proposed changes: isabelatkinson@a060234

[isabelatkinson]

These two tests should note that an 8.0+ server is required for bulkWrite support

[isabelatkinson]

I'm missing this file, can you add it to the PR?

[baileympearson]

Can we update the tests in this PR to specify the libmongocrypt version as a runOnRequirement?

https://github.com/mongodb/specifications/blob/master/source/unified-test-format/unified-test-format.md#:~:text=minLibmongocryptVersion%3A%20Optional%20string.%20The%20minimum%20libmongocrypt%20(inclusive)%20required%20to%20successfully%20run%20the%20tests.%20If%20this%20field%20is%20omitted%2C%20there%20is%20no%20lower%20bound%20on%20the%20required%20libmongocrypt%20version.%20The%20format%20of%20this%20string%20is%20defined%20in%20Version%20String.

[baileympearson]

@kevinAlbs @mdb-ad Is there any chance https://jira.mongodb.org/browse/DRIVERS-2859 can be completed before DRIVERS-2888? It makes much more sense to make the change in once place than making the changes in every driver. This is especially true because drivers supporting non-document sequences for client bulk write will be temporary until https://jira.mongodb.org/browse/DRIVERS-2859 is addressed. So making drivers implement the changes before https://jira.mongodb.org/browse/DRIVERS-2859 will cost extra work in every driver, extra work updating the spec to allow (or enforce) document sequences with bulk write after https://jira.mongodb.org/browse/DRIVERS-2859 is implemented, and then again extra work in drivers to support document sequencing for bulkWrite with QE.

From a more selfish, Node-specific perspective - our bulkWrite implementation is tightly coupled to document sequences and making this change might be non-trivial. I'd rather not refactor our bulk write implementation unless absolutely necessary.

[kevinAlbs]

drivers supporting non-document sequences for client bulk write will be temporary until https://jira.mongodb.org/browse/DRIVERS-2859 is addressed

The C driver only required a minor change to support QE with MongoClient.bulkWrite. Supporting CSFLE with MongoCollection.bulkWrite already requires passing all write documents in a non-document sequence, and the C driver reuses it.

I propose: proceed with this PR to unblock some drivers with QE, but add a blurb suggesting drivers wait for DRIVERS-2859:

Auto-Encryption

libmongocrypt does not-yet support OP_MSG document sequences (DRIVERS-2859). Drivers requiring significant changes to pass a bulkWrite command to libmongocrypt may prefer to wait until DRIVERS-2859 is implemented to support automatic encryption.