DRIVERS-3227 test text indexes with auto encryption

[kevinAlbs]

Summary

• Add automatic encryption tests of Queryable Encryption (QE) text indexes added in SPM-4158.
• Add schema version 1.25 with minLibmongocryptVersion.
• Fix ajv validation with schema version 1.24+.

To run tests:

• Upgrade libmongocrypt to a 1.15.0.
• Implement unified test runner support in DRIVERS-3106.

Tests were run in the C driver in mongodb/mongo-c-driver#2069

Background & Motivation

Tests are intended to serve as simple end-to-end tests to ensure drivers have upgraded libmongocrypt and work as expected. Tests are not intended to exhaustively test behavior of text queries.

This PR does not update EncryptOpts for explicit encryption. This is expected in DRIVERS-3213.

libmongocrypt 1.15.0

libmongocrypt 1.15.0 is released. Binaries can be obtained from the Files tab from upload tasks on the 1.15.0 tagged commit.

libmongocrypt 1.14.0 includes most of the text index support but excludes support for cleanup/compact commands (MONGOCRYPT-810 + MONGOCRYPT-811). Tests are added for compactStructuredEncryptionData and cleanupStructuredEncryptionData to ensure drivers have updated libmongocrypt to get latest changes from SPM-4158. Attempting to run these tests on libmongocrypt 1.14.0 results in a mismatch failure:

# Override to use libmongocrypt 1.14.0. Does not include updates for text compact/cleanup:
export LD_LIBRARY_PATH=$INSTALL_PATH/libmongocrypt-1.14.0/lib
$TEST_PATH/test-libmongoc --match "/client_side_encryption/unified/QE-Text-compactStructuredEncryptionData"
# Error: key 'encryptionInformation' is not present
$TEST_PATH/test-libmongoc --match "/client_side_encryption/unified/QE-Text-cleanupStructuredEncryptionData"
# Error: at path: 'cleanupTokens.encryptedText': expected type document, got binData

minLibmongocryptVersion

runOnRequirements are extended to add a minLibmongocryptVersion constraint. This is intended to help drivers skip tests when using older libmongocrypt versions (see comment).

This is implemented as an alternative form of the csfle constraint, summarized as:

# Test prohibits CSFLE:
runOnRequirements:
  - csfle: false

# Test requires CSFLE:
runOnRequirements:
  - csfle: true

# (New) Test requires CSFLE with libmongocrypt version 1.15+.
runOnRequirements:
  - csfle:
      minLibmongocryptVersion: 1.15

This required a new Unified Test Format schema version 1.25. This led to the discovery of an existing issue using ajv to validate with schema 1.24:

% ajv test -s ./source/unified-test-format/schema-1.24.json -d ./source/unified-test-format/tests/valid-pass/poc-crud.json --valid
schema ./source/unified-test-format/schema-1.24.json is invalid
error: no schema with key or ref "https://json-schema.org/draft/2019-09/schema#"

I expect this was undiscovered since there are no tests currently using the 1.24 format. Quoting #1809:

JSON Schema draft 2019-09 adds a new "deprecated" property

Explicit --spec arguments are added to ajv invocations to fix. ajv help notes draft7 (used in spec 1.23 and older) is the default.

---

Please complete the following before merging:

• Update changelog.
• Test changes in at least one language driver.
• [ ] Test these changes against all server versions and topologies (including standalone, replica set, and sharded
  clusters). C does not-yet test sharded with In-Use Encryption.

[baileympearson]

Two small comments. Passing in Node: https://parsley.mongodb.com/evergreen/mongo_node_driver_next_rhel8_custom_dependency_tests_run_custom_csfle_tests_latest_patch_acd86250216d0edf0fbc4a1893701a2df038d9bb_6883ddf0c6cd1900073e32b9_25_07_25_19_41_51/0/task?bookmarks=0,1811,1983&selectedLineRange=L1811-L1826

[baileympearson]

What would you think about making libmongocrypt version a runOnRequirement? This would be useful in Node and maybe other drivers who have bindings separate from their driver code. We actually do run our latest driver test suite against an older version of our bindings, so this would be necessary in Node (if it isn't codified into the UTR, we'll just have to skip it manually).

[kevinAlbs]

I like the idea. Added a minLibmongocryptVersion constraint. I expect this can also benefit backporting tests to release branches using an older libmongocrypt.

[baileympearson]

We probably want to adopt these changes in #1784. I added a comment to that PR about these UTR changes.

[baileympearson]

Passing in Node: https://spruce.mongodb.com/version/688baf1b0ff321000788a71c/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC

[codeowners-service-app]

Assigned bisht2050 for team dbx-leadership because jmikola is out of office.

[alcaeus]

Changes to schema extraction LGTM

[jmikola]

Noted that this pertains to https://ajv.js.org/packages/ajv-cli.html#json-schema-language-and-version

Did something in this PR require introduction of the --spec option for ajv-cli? It looks like the schema URL was changed in db69351 when deprecated options were introduced.

[kevinAlbs]

Adding schema 1.25 led to the discovery of an existing issue using ajv to validate with schema 1.24:

% ajv test -s ./source/unified-test-format/schema-1.24.json -d ./source/unified-test-format/tests/valid-pass/poc-crud.json --valid
schema ./source/unified-test-format/schema-1.24.json is invalid
error: no schema with key or ref "https://json-schema.org/draft/2019-09/schema#"

I expect this was undiscovered since there are no tests currently using the 1.24 format. Quoting #1809:

JSON Schema draft 2019-09 adds a new "deprecated" property

Adding the --spec argument appears to fix. ajv help notes draft7 (used in spec 1.23 and older) is the default.

[jmikola]

This seems fine.

runOnRequirement-csfle-type.yml remains as-is, because you're making the type of csfle more permissive, but I think you can add a new test for your restriction on the object value's structure.

[kevinAlbs]

Added invalid tests for minLibmongocryptVersion.