Skip to content

INTPYTHON-527 Add Queryable Encryption support #329

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 32 commits into
base: main
Choose a base branch
from
Open

INTPYTHON-527 Add Queryable Encryption support #329

Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
36bd1e5
INTPYTHON-527 Add Queryable Encryption support
aclark4life Jun 25, 2025
70c946b
encrypted fields map != encrypted fields
aclark4life Aug 27, 2025
cb3512c
Use dot separator
aclark4life Sep 16, 2025
46ca9dc
Refactor tests
aclark4life Sep 19, 2025
87b1dc9
Add embedding
aclark4life Sep 19, 2025
d39f3b3
Review feedback
aclark4life Sep 19, 2025
3f8b5c2
remove EncryptedEmbeddedModel
timgraham Sep 23, 2025
441c584
PatientRecord shouldn't be encrypted
timgraham Sep 23, 2025
c9cc301
fix linting of kms_provider() docstring line length
timgraham Sep 23, 2025
332decb
Code review fixes
aclark4life Sep 30, 2025
0317fba
Use EncryptionTestCase instead of TestCase
aclark4life Oct 2, 2025
e3da2f6
make atlas tests use encryption settings
timgraham Oct 2, 2025
fe790c7
try shared library
timgraham Oct 3, 2025
2270058
try ubuntu 22.04 just to be sure
timgraham Oct 3, 2025
e38b496
Code review fixes
aclark4life Oct 3, 2025
36c6bfd
Update QE guide with complete Python tutorial
aclark4life Oct 3, 2025
86486eb
update version added to 5.2.2
aclark4life Oct 3, 2025
59865f7
Add tests for EncryptedFieldMixin
aclark4life Oct 3, 2025
fc87e9f
Remove confusing paragraph about crypt shared
aclark4life Oct 3, 2025
75873e9
Target 5.2.3 for release and require MongoDB 8
aclark4life Oct 3, 2025
324c959
try Mongo 8.0.15 on CI
timgraham Oct 3, 2025
13ed19a
Misc updates
aclark4life Oct 4, 2025
b23c4f2
Misc updates
aclark4life Oct 6, 2025
a0cd197
Kill the helper
aclark4life Oct 7, 2025
65b96b2
Misc updates
aclark4life Oct 8, 2025
80881fa
Add assertEncrypted to verify field data is binary
aclark4life Oct 9, 2025
25e7da1
Fails on CI only
aclark4life Oct 9, 2025
ff84902
Remove create_data_keys until use case manifests
aclark4life Oct 10, 2025
23f20a4
Add partial index on keyAltNames for uniqueness
aclark4life Oct 10, 2025
01ec095
Code review updates
aclark4life Oct 10, 2025
8a3ccf5
Code review updates
aclark4life Oct 10, 2025
9bfa86d
Add EncryptedArrayField + test
aclark4life Oct 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .evergreen/run-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ set -eux
/opt/python/3.10/bin/python3 -m venv venv
. venv/bin/activate
python -m pip install -U pip
pip install ".[encryption]"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any objection to combining this line and the next line into pip install -e .[encryption]?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be best to have this line run only on builds that test encryption, so that we don't inadvertently add some top-level import for an optional dependency (similar reason to why we have a separate build for GIS that installs gdal-bin.) Perhaps the encryption build could set an environment variable like RUNNING_ENCRYPTION_TESTS that this script could consult.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK filing under CI todo

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since it doesn't appear we're going to run the encrypted tests on evergreen, we can remove this line.

pip install -e .

# Install django and test dependencies
Expand Down
18 changes: 18 additions & 0 deletions .github/workflows/encrypted_settings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
import os

from mongodb_settings import * # noqa: F403
from pymongo.encryption import AutoEncryptionOpts

DATABASES["encrypted"] = { # noqa: F405
"ENGINE": "django_mongodb_backend",
"NAME": "djangotests-encrypted",
"OPTIONS": {
"auto_encryption_opts": AutoEncryptionOpts(
key_vault_namespace="my_encrypted_database.keyvault",
kms_providers={"local": {"key": os.urandom(96)}},
# crypt_shared_lib_path="lib/mongo_crypt_v1.so",
),
"directConnection": True,
},
"KMS_CREDENTIALS": {},
}
19 changes: 19 additions & 0 deletions .github/workflows/mongodb_settings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
DATABASES = {
"default": {**db_settings, "NAME": "djangotests"},
"other": {**db_settings, "NAME": "djangotests-other"},
"encrypted": {},
}
else:
DATABASES = {
Expand All @@ -28,8 +29,26 @@
"NAME": "djangotests-other",
"OPTIONS": {"directConnection": True},
},
"encrypted": {},
}


class EncryptedRouter:
def allow_migrate(self, db, app_label, model_name=None, **hints):
# The encryption_ app's models are only created in the encrypted
# database.
if app_label == "encryption_":
return db == "encrypted"
# Don't create other app's models in the encrypted database.
if db == "encrypted":
return False
return None

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a refactor, you removed the test router than had:

    def db_for_read(self, model, **hints):
        if model._meta.app_label == "encryption_":
            return "encrypted"
        return None

    db_for_write = db_for_read

thus the current tests aren't stored objects in the encrypted database. These methods must also be added to the example router in the documentation.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed it because I thought we could rely on the test runner settings ? I'll update the docs.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These methods direct querysets where to send their queries. Without this guidance, all queries go the "default" database.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. I mean I have this in my test runner settings:

    def db_for_read(self, model, **hints):                                      
        if model_has_encrypted_fields(model):                                   
            return "encrypted"                                                  
        return "default"

So do we need a router in the tests or can we rely on the test runner's router ?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can put in the test runner's router.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment was resolved but the changes weren't made. assertEncrypted fails because the missing db_for_read / db_for_write methods on this router. Without them, the objects are created in the default database, thus there's nothing in the encrypted database.

def kms_provider(self, model, **hints):
return "local"


DATABASE_ROUTERS = [EncryptedRouter()]
DEFAULT_AUTO_FIELD = "django_mongodb_backend.fields.ObjectIdAutoField"
PASSWORD_HASHERS = ("django.contrib.auth.hashers.MD5PasswordHasher",)
SECRET_KEY = "django_tests_secret_key"
Expand Down
Loading