INTPYTHON-527 Add Queryable Encryption support #329

aclark4life · 2025-06-27T02:16:28Z

Previous attempts and additional context here:

INTPYTHON-527 Add Queryable Encryption config #318
- Branched from transaction support and includes only helpers
INTPYTHON-527 Add queryable encryption support #319
- In which this bug in libmongocrypt was discovered
- A fix for that bug in PyMongo is being discussed here.
INTPYTHON-527 Add Queryable Encryption support #323
- Before I understood that the shared lib for mongocrypt was included in the pymongocrypt wheel.

tests/encryption_/tests.py

django_mongodb_backend/models.py

django_mongodb_backend/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/base.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/features.py

django_mongodb_backend/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

tests/encryption_/routers.py

tests/encryption_/models.py

tests/backend_/test_features.py

django_mongodb_backend/schema.py

tests/encryption_/models.py

django_mongodb_backend/schema.py

django_mongodb_backend/encryption.py

tests/encryption_/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/models.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

docs/source/ref/encryption.rst

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

- MongoDB 7 has rangePreview which we do not plan to support

- Add mongocryptd to atlas CI - Doc updates - Remove custom db_type, not needed after mongodb#414 - SupportsQueryableEncryptionTests require MongoDB 8 - Update test model names prefix and suffix

django_mongodb_backend/schema.py

- Isolate test failures - admin_scripts - check_framework - migrations - test_runner - user_commands - Add support for EncryptedEmbeddedModelArrayField - Refactor - Factor _get_data_key from _get_encrypted_fields - Refactor for DRY in _get_encrypted_fields method with _field_dict helper

django_mongodb_backend/schema.py

Ruff killed the nested if statements

django_mongodb_backend/schema.py

tests/encryption_/test_fields.py

django_mongodb_backend/schema.py

- Remove extra check for auto_encryption_opts - Earlier check should suffice - Remove Encrypted* class name prefix - We know these are the QE tests - Move create data key back to _get_encrypted_fields - Remove getattr for client_encryption - Remove EmbeddedModelArrayField

timgraham · 2025-10-10T00:30:44Z

django_mongodb_backend/fields/encryption.py

+class EncryptedTimeField(EncryptedFieldMixin, models.TimeField):
+    pass
+
+
+class EncryptedTextField(EncryptedFieldMixin, models.TextField):
+    pass


out of alphabetical order

timgraham · 2025-10-10T15:15:22Z

django_mongodb_backend/schema.py

-        if not key:
-            raise ValueError(
-                f"No key found in keyvault for keyAltName={key_alt_name}. "
-                "Run with '--create-data-keys' to create missing keys."
-            )
-        return key["_id"]


We lose this helpful message:

File "/home/tim/code/django-mongodb/django_mongodb_backend/schema.py", line 541, in _get_encrypted_fields data_key = key["_id"] ~~~^^^^^^^ TypeError: 'NoneType' object is not subscriptable

I think that logic was flawed. Updated to check for existing keys or create if not found.

Then showencryptedfieldsmap isn't read-only and may have a side effect of creating keys?

What is still missing for me is the "why" of setting encrypted_fields_map in AutoEncryptionOpts. Based on past discussion in this PR, I recall it has something to do with security, but I don't think this is explained in this PR's documention or in the design doc.

Then showencryptedfieldsmap isn't read-only and may have a side effect of creating keys?

I've removed --create-data-keys so now we're just talking about showencryptedfieldsmap and yes, since it calls _get_encrypted_fields it will create the data keys if they aren't found in the key vault.

What is still missing for me is the "why" of setting encrypted_fields_map in AutoEncryptionOpts. Based on past discussion in this PR, I recall it has something to do with security, but I don't think this is explained in this PR's documention or in the design doc.

That's in the spec:

Supplying an encryptedFieldsMap provides more security than relying on an encryptedFields obtained from the server. It protects against a malicious server advertising a false encryptedFields.

I don't fully understand that explanation so I haven't gone too far in documenting it other than to say "recommended".

timgraham · 2025-10-10T15:27:46Z

tests/encryption_/test_fields.py

+    def test_array(self):
+        self.assertEqual(len(self.movie.cast), 2)
+        self.assertEqual(self.movie.cast[0].name, "Actor One")
+        self.assertEqual(self.movie.cast[1].name, "Actor Two")
+


Like the other tests, I think you meant to fetch the data from the database before checking it.

In addition to removing the arg from the showencryptedfields command, reduces complexity in schema editor with removal of the `create_data_keys` boolean. Previous logic may have been flawed in looking up existing keys.

Also put the key vault inside the test database for easy tear down, tested and working local.

aclark4life commented Jun 27, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/models.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/encryption.py Outdated Show resolved Hide resolved

timgraham reviewed Jun 27, 2025

View reviewed changes

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jun 30, 2025

View reviewed changes

django_mongodb_backend/features.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/routers.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 4, 2025

View reviewed changes

tests/encryption_/models.py Show resolved Hide resolved

timgraham reviewed Jul 7, 2025

View reviewed changes

tests/encryption_/models.py Outdated Show resolved Hide resolved

aclark4life commented Jul 7, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 9, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

aclark4life commented Jul 11, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

aclark4life and others added 6 commits October 6, 2025 13:20

Update QE guide with complete Python tutorial

36c6bfd

update version added to 5.2.2

86486eb

Add tests for EncryptedFieldMixin

59865f7

Remove confusing paragraph about crypt shared

fc87e9f

Target 5.2.3 for release and require MongoDB 8

75873e9

- MongoDB 7 has rangePreview which we do not plan to support

try Mongo 8.0.15 on CI

324c959

aclark4life force-pushed the INTPYTHON-527 branch 3 times, most recently from 54bd164 to db4aadb Compare October 6, 2025 19:57

Misc updates

13ed19a

- Add mongocryptd to atlas CI - Doc updates - Remove custom db_type, not needed after mongodb#414 - SupportsQueryableEncryptionTests require MongoDB 8 - Update test model names prefix and suffix

aclark4life force-pushed the INTPYTHON-527 branch from 8b8704f to 13ed19a Compare October 6, 2025 23:19

timgraham reviewed Oct 7, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

aclark4life force-pushed the INTPYTHON-527 branch from 0cbca21 to 59825cd Compare October 7, 2025 16:26

aclark4life force-pushed the INTPYTHON-527 branch from 59825cd to b23c4f2 Compare October 7, 2025 16:40

timgraham reviewed Oct 7, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

Kill the helper

a0cd197

Ruff killed the nested if statements

aclark4life force-pushed the INTPYTHON-527 branch from e10d8dc to 96156c8 Compare October 7, 2025 20:37

timgraham reviewed Oct 7, 2025

View reviewed changes

aclark4life force-pushed the INTPYTHON-527 branch 3 times, most recently from 7e155a9 to f7846a6 Compare October 8, 2025 23:51

Misc updates

65b96b2

- Remove extra check for auto_encryption_opts - Earlier check should suffice - Remove Encrypted* class name prefix - We know these are the QE tests - Move create data key back to _get_encrypted_fields - Remove getattr for client_encryption - Remove EmbeddedModelArrayField

aclark4life force-pushed the INTPYTHON-527 branch from f7846a6 to 65b96b2 Compare October 8, 2025 23:54

aclark4life added 2 commits October 8, 2025 20:34

Add assertEncrypted to verify field data is binary

80881fa

Fails on CI only

25e7da1

aclark4life force-pushed the INTPYTHON-527 branch from 52bb16e to 25e7da1 Compare October 9, 2025 22:37

timgraham reviewed Oct 10, 2025

View reviewed changes

aclark4life added 2 commits October 10, 2025 14:01

Remove create_data_keys until use case manifests

ff84902

In addition to removing the arg from the showencryptedfields command, reduces complexity in schema editor with removal of the `create_data_keys` boolean. Previous logic may have been flawed in looking up existing keys.

Add partial index on keyAltNames for uniqueness

23f20a4

Also put the key vault inside the test database for easy tear down, tested and working local.

INTPYTHON-527 Add Queryable Encryption support #329

Are you sure you want to change the base?

INTPYTHON-527 Add Queryable Encryption support #329

Uh oh!

Conversation

aclark4life commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

timgraham Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

timgraham Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

aclark4life Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

timgraham Oct 10, 2025

aclark4life commented Jun 27, 2025 •

edited

Loading