INTPYTHON-527 Add Queryable Encryption support #329

aclark4life · 2025-06-27T02:16:28Z

Previous attempts and additional context here:

INTPYTHON-527 Add Queryable Encryption config #318
- Branched from transaction support and includes only helpers
INTPYTHON-527 Add queryable encryption support #319
- In which this bug in libmongocrypt was discovered
- A fix for that bug in PyMongo is being discussed here.
INTPYTHON-527 Add Queryable Encryption support #323
- Before I understood that the shared lib for mongocrypt was included in the pymongocrypt wheel.

tests/encryption_/tests.py

django_mongodb_backend/models.py

django_mongodb_backend/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/base.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

django_mongodb_backend/features.py

django_mongodb_backend/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

tests/encryption_/routers.py

tests/encryption_/models.py

tests/backend_/test_features.py

django_mongodb_backend/schema.py

tests/encryption_/models.py

django_mongodb_backend/schema.py

django_mongodb_backend/encryption.py

tests/encryption_/routers.py

django_mongodb_backend/schema.py

django_mongodb_backend/models.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

docs/source/ref/encryption.rst

django_mongodb_backend/fields/encryption.py

django_mongodb_backend/management/commands/get_encrypted_fields_map.py

django_mongodb_backend/schema.py

tests/encryption_/tests.py

docs/source/topics/index.rst

docs/source/ref/django-admin.rst

docs/source/ref/models/fields.rst

docs/source/howto/queryable-encryption.rst

docs/source/ref/django-admin.rst

docs/source/topics/queryable-encryption.rst

tests/encryption_/test_base.py

tests/encryption_/test_management.py

docs/source/topics/queryable-encryption.rst

tests/encryption_/models.py

docs/source/howto/queryable-encryption.rst

tests/encryption_/test_base.py

tests/encryption_/routers.py

docs/source/howto/queryable-encryption.rst

django_mongodb_backend/schema.py

django_mongodb_backend/base.py

django_mongodb_backend/management/commands/showencryptedfieldsmap.py

django_mongodb_backend/schema.py

timgraham · 2025-08-11T18:51:31Z

django_mongodb_backend/schema.py

+                raise ImproperlyConfigured(
+                    "Encrypted fields found but "
+                    "DATABASES[[self.connection.alias}]['OPTIONS'] is missing "
+                    "auto_encryption_opts. Please set `auto_encryption_opts` "
+                    "in the connection settings."
+                )


Did you try adding a test for this exception?

I've tested it manually but haven't written a test for it yet.

django_mongodb_backend/schema.py

timgraham · 2025-08-11T19:02:09Z

docs/source/howto/queryable-encryption.rst

+        "KMS_PROVIDERS": {},
+        "KMS_CREDENTIALS": {},


For KMS_CREDENTIALS, I believe the options are documented here: https://pymongo.readthedocs.io/en/stable/api/pymongo/encryption.html#pymongo.encryption.ClientEncryption.create_data_key (but pymongo docs are deprecated?)

So a minimal example for AWS: "KMS_CREDENTIALS": {"aws": {"region": "...", "key": "..."}

- Fix rebase merge conflict edits - Remove integer field FIXME comments - Remove pos_int

Don't break the build!

Don't break the build Part II

addaleax · 2025-08-11T23:26:48Z

docs/source/howto/queryable-encryption.rst

+
+.. _server-side-queryable-encryption-settings:
+
+Server-side Queryable Encryption


Definitely not a phrase we should be using in documentation; the core point of CSFLE/QE is that the encryption itself is primarily a client-side feature, and this title sounds like some of the encryption logic actually happens on the server side. This should be something like "Server-side schema for Queryable Encryption".

😂 OK will fix

addaleax · 2025-08-11T23:27:45Z

docs/source/howto/queryable-encryption.rst

+
+.. _client-side-queryable-encryption:
+
+Client-side Queryable Encryption


docs/source/howto/queryable-encryption.rst

addaleax · 2025-08-11T23:29:10Z

docs/source/howto/queryable-encryption.rst

+        …
+        "OPTIONS": {
+            "auto_encryption_opts": AutoEncryptionOpts(
+                …


Should this include an example of how to specify the crypt_shared library path?

Yes! Will add.

Co-authored-by: Anna Henningsen <[email protected]>

- Removed all signs of client side vs. server side configuration of Queryable Encryption which can be too easily confused with client-side encryption, which is what Queryable Encryption is.

Copilot

Pull Request Overview

This PR adds support for MongoDB's Queryable Encryption feature to the Django MongoDB Backend, allowing developers to store sensitive data in encrypted fields that can still be queried using specific query types. The implementation includes encrypted field classes, schema support, management commands, and comprehensive documentation.

Adds encrypted field types that wrap Django's built-in fields with encryption capabilities
Implements schema editor support for creating collections with encrypted fields and managing data keys
Provides management command for generating encrypted fields maps and creating encryption keys

Reviewed Changes

Copilot reviewed 32 out of 33 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`django_mongodb_backend/fields/encryption.py`	Defines encrypted field classes and mixin for queryable encryption
`django_mongodb_backend/schema.py`	Adds schema editor support for creating encrypted collections and managing data keys
`django_mongodb_backend/management/commands/showencryptedfieldsmap.py`	Management command for generating encrypted fields maps
`django_mongodb_backend/features.py`	Adds feature detection for queryable encryption support
`tests/encryption_/`	Comprehensive test suite covering encrypted fields, schema operations, and management commands
`docs/source/`	Documentation for configuring and using queryable encryption
`pyproject.toml`	Adds optional encryption dependency

django_mongodb_backend/management/commands/showencryptedfieldsmap.py

django_mongodb_backend/schema.py

tests/encryption_/test_management.py

docs/source/howto/queryable-encryption.rst

…map.py Co-authored-by: Copilot <[email protected]>

Co-authored-by: Copilot <[email protected]>

aclark4life commented Jun 27, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/models.py Outdated Show resolved Hide resolved

aclark4life commented Jun 27, 2025

View reviewed changes

django_mongodb_backend/encryption.py Outdated Show resolved Hide resolved

timgraham reviewed Jun 27, 2025

View reviewed changes

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jun 30, 2025

View reviewed changes

django_mongodb_backend/features.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/routers.py Outdated Show resolved Hide resolved

timgraham reviewed Jul 1, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 4, 2025

View reviewed changes

tests/encryption_/models.py Show resolved Hide resolved

timgraham reviewed Jul 7, 2025

View reviewed changes

tests/encryption_/models.py Outdated Show resolved Hide resolved

aclark4life commented Jul 7, 2025

View reviewed changes

django_mongodb_backend/schema.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

timgraham reviewed Jul 9, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

aclark4life commented Jul 11, 2025

View reviewed changes

tests/encryption_/tests.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

aclark4life force-pushed the INTPYTHON-527 branch from 5b2bc07 to bddae41 Compare August 11, 2025 11:09

timgraham reviewed Aug 11, 2025

View reviewed changes

aclark4life added 9 commits August 11, 2025 19:15

INTPYTHON-527 Add Queryable Encryption support

b8b9b43

Code review fixes

379537a

- Fix rebase merge conflict edits - Remove integer field FIXME comments - Remove pos_int

Update release notes & related docs

a21f86a

Code review fixes (1/x)

76a2b08

Code review fixes (2/x)

9a388bd

Code review fixes (3/x)

b2a7f32

Don't break the build!

Code review fixes (4/x)

e1c3e43

Don't break the build Part II

Code review fixes (5/x)

917e3a8

Code review fixes (6/x)

6afce5a

aclark4life force-pushed the INTPYTHON-527 branch from 0e7b955 to 6afce5a Compare August 11, 2025 23:17

addaleax reviewed Aug 11, 2025

View reviewed changes

aclark4life and others added 2 commits August 11, 2025 19:40

Code review fixes (7/7)

a6aa8da

Fix encrypted tests crashing on non-encrypted builds

31b24f6

timgraham force-pushed the INTPYTHON-527 branch from c7f360e to 31b24f6 Compare August 12, 2025 00:17

aclark4life and others added 2 commits August 11, 2025 22:23

Update docs/source/howto/queryable-encryption.rst

fcf8e5a

Co-authored-by: Anna Henningsen <[email protected]>

Code review fixes (8/x)

c74dc4f

- Removed all signs of client side vs. server side configuration of Queryable Encryption which can be too easily confused with client-side encryption, which is what Queryable Encryption is.

aclark4life requested review from timgraham, addaleax, Copilot and Jibola August 12, 2025 03:31

Copilot AI reviewed Aug 12, 2025

View reviewed changes

aclark4life and others added 3 commits August 11, 2025 23:33

Update django_mongodb_backend/management/commands/showencryptedfields…

b1ca4c8

…map.py Co-authored-by: Copilot <[email protected]>

Update django_mongodb_backend/schema.py

cdc49c7

Co-authored-by: Copilot <[email protected]>

Update tests/encryption_/test_management.py

c1e2356

Co-authored-by: Copilot <[email protected]>

mongodb deleted a comment from Copilot AI Aug 12, 2025

Update docs/source/howto/queryable-encryption.rst

0997357

Co-authored-by: Copilot <[email protected]>


		.. _server-side-queryable-encryption-settings:

		Server-side Queryable Encryption


		.. _client-side-queryable-encryption:

		Client-side Queryable Encryption

INTPYTHON-527 Add Queryable Encryption support #329

Are you sure you want to change the base?

INTPYTHON-527 Add Queryable Encryption support #329

Conversation

aclark4life commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aclark4life commented Jun 27, 2025 •

edited

Loading