Add encrypted admin w/"no count" pagination #3

aclark4life · 2025-11-12T15:57:42Z

Existing approaches to "no count" pagination do not seem to include support for pagination in the Django admin, so:

Subclass Django's core Paginator
- override count
Subclass Django's contrib admin view ChangeList.get_results
- Use len instead of count
- Ruff fixes for try/except
Subclass Django's contrib ModelAdmin
- override get_paginator
- override get_changelist

Existing approaches to "no count" pagination do not seem to include support for pagination in the Django admin, so: - Subclass Django's core Paginator - override count - Subclass Django's contrib admin view ChangeList.get_results - Use len instead of count - Ruff fixes for try/except - Subclass Django's contrib ModelAdmin - override get_paginator - override get_changelist

timgraham · 2025-11-15T01:46:53Z

As I said before, len(queryset) will load the entire collection into memory. If MongoDB is supposed to be for big data, this is a nogo. It will likely take a long time and/or cause the app server to run out of memory.

Perhaps it works for testing a toy project with a small collection, but I'm unsure we should ship this approach to enterprise customers. We can check with the team before investing more time, like adding tests.

aclark4life · 2025-11-15T16:44:03Z

As I said before, len(queryset) will load the entire collection into memory. If MongoDB is supposed to be for big data, this is a nogo. It will likely take a long time and/or cause the app server to run out of memory.

Perhaps it works for testing a toy project with a small collection, but I'm unsure we should ship this approach to enterprise customers. We can check with the team before investing more time, like adding tests.

We should not ship a toy to the enterprise but we should figure out how to implement this feature so folks can use the Django admin.

timgraham · 2025-11-16T01:23:26Z

I spent a few minutes looking for solutions but haven't found anything. At first my mind went to db.collection.estimatedDocumentCount() which does work on encrypted collections, however, we have a (possibly filtered) querysets here, not necessarily the full collection.

aclark4life · 2025-11-19T20:23:55Z

For ref:

System check identified 1 issue (1 silenced).
November 19, 2025 - 14:14:28
Django version 5.2.8.dev20251021134245, using settings 'qe.settings.qe'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

WARNING: This is a development server. Do not use it in a production setting. Use a production WSGI or ASGI server instead.
For more information on production servers see: https://docs.djangoproject.com/en/dev/howto/deployment/
Internal Server Error: /admin/django_mongodb_demo/patient/
Traceback (most recent call last):
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 129, in _wrap_encryption_errors
    yield
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 472, in encrypt
    encrypted_cmd = self._auto_encrypter.encrypt(database, encoded_cmd)
  File "/Users/alex.clark/Developer/django-mongodb-cli/.venv/lib/python3.13/site-packages/pymongocrypt/synchronous/auto_encrypter.py", line 44, in encrypt
    return run_state_machine(ctx, self.callback)
  File "/Users/alex.clark/Developer/django-mongodb-cli/.venv/lib/python3.13/site-packages/pymongocrypt/synchronous/state_machine.py", line 136, in run_state_machine
    result = callback.mark_command(ctx.database, mongocryptd_cmd)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 292, in mark_command
    res = self.mongocryptd_client[database].command(
        inflated_cmd, codec_options=DEFAULT_RAW_BSON_OPTIONS
    )
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/_csot.py", line 125, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 938, in command
    return self._command(
           ~~~~~~~~~~~~~^
        connection,
        ^^^^^^^^^^^
    ...<7 lines>...
        **kwargs,
        ^^^^^^^^^
    )
    ^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/database.py", line 778, in _command
    return conn.command(
           ~~~~~~~~~~~~^
        self._name,
        ^^^^^^^^^^^
    ...<8 lines>...
        client=self._client,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/helpers.py", line 45, in inner
    return func(*args, **kwargs)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/pool.py", line 413, in command
    return command(
        self,
    ...<20 lines>...
        write_concern=write_concern,
    )
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/network.py", line 212, in command
    helpers_shared._check_command_response(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        response_doc,
        ^^^^^^^^^^^^^
    ...<2 lines>...
        parse_write_concern_error=parse_write_concern_error,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/helpers_shared.py", line 284, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: Aggregation stage $internalFacetTeeConsumer is not allowed or supported with automatic encryption., full error: RawBSONDocument(b'\xa6\x00\x00\x00\x01ok\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02errmsg\x00c\x00\x00\x00Aggregation stage $internalFacetTeeConsumer is not allowed or supported with automatic encryption.\x00\x10code\x00#y\x00\x00\x02codeName\x00\x0e\x00\x00\x00Location31011\x00\x00', codec_options=CodecOptions(document_class=<class 'bson.raw_bson.RawBSONDocument'>, tz_aware=False, uuid_representation=UuidRepresentation.UNSPECIFIED, unicode_decode_error_handler='strict', tzinfo=None, type_registry=TypeRegistry(type_codecs=[], fallback_encoder=None), datetime_conversion=DatetimeConversion.DATETIME))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django-mongodb-backend/django_mongodb_backend/query.py", line 19, in wrapper
    return func(*args, **kwargs)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django-mongodb-backend/django_mongodb_backend/query.py", line 77, in get_cursor
    return self.compiler.collection.aggregate(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.get_pipeline(), session=self.compiler.connection.session
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django-mongodb-backend/django_mongodb_backend/utils.py", line 174, in wrapper
    duration, retval = self.profile_call(func, args, kwargs)
                       ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django-mongodb-backend/django_mongodb_backend/utils.py", line 144, in profile_call
    retval = func(*args, **kwargs or {})
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/collection.py", line 3010, in aggregate
    return self._aggregate(
           ~~~~~~~~~~~~~~~^
        _CollectionAggregationCommand,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
        **kwargs,
        ^^^^^^^^^
    )
    ^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/_csot.py", line 125, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/collection.py", line 2908, in _aggregate
    return self._database.client._retryable_read(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        cmd.get_cursor,
        ^^^^^^^^^^^^^^^
    ...<3 lines>...
        operation=_Op.AGGREGATE,
        ^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/mongo_client.py", line 2048, in _retryable_read
    return self._retry_internal(
           ~~~~~~~~~~~~~~~~~~~~^
        func,
        ^^^^^
    ...<7 lines>...
        operation_id=operation_id,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/_csot.py", line 125, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/mongo_client.py", line 2014, in _retry_internal
    ).run()
      ~~~^^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/mongo_client.py", line 2763, in run
    return self._read() if self._is_read else self._write()
           ~~~~~~~~~~^^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/mongo_client.py", line 2924, in _read
    return self._func(self._session, self._server, conn, read_pref)  # type: ignore
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/aggregation.py", line 162, in get_cursor
    result = conn.command(
        self._database.name,
    ...<9 lines>...
        user_fields=self._user_fields,
    )
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/helpers.py", line 45, in inner
    return func(*args, **kwargs)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/pool.py", line 441, in command
    self._raise_connection_failure(error)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/pool.py", line 413, in command
    return command(
        self,
    ...<20 lines>...
        write_concern=write_concern,
    )
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/network.py", line 138, in command
    spec = orig = client._encrypter.encrypt(dbname, spec, codec_options)
                  ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 471, in encrypt
    with _wrap_encryption_errors():
         ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/alex.clark/.pyenv/versions/3.13.9/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 135, in _wrap_encryption_errors
    raise EncryptionError(exc) from exc
pymongo.errors.EncryptionError: Aggregation stage $internalFacetTeeConsumer is not allowed or supported with automatic encryption., full error: RawBSONDocument(b'\xa6\x00\x00\x00\x01ok\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02errmsg\x00c\x00\x00\x00Aggregation stage $internalFacetTeeConsumer is not allowed or supported with automatic encryption.\x00\x10code\x00#y\x00\x00\x02codeName\x00\x0e\x00\x00\x00Location31011\x00\x00', codec_options=CodecOptions(document_class=<class 'bson.raw_bson.RawBSONDocument'>, tz_aware=False, uuid_representation=UuidRepresentation.UNSPECIFIED, unicode_decode_error_handler='strict', tzinfo=None, type_registry=TypeRegistry(type_codecs=[], fallback_encoder=None), datetime_conversion=DatetimeConversion.DATETIME))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/contrib/admin/options.py", line 719, in wrapper
    return self.admin_site.admin_view(view)(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/utils/decorators.py", line 192, in _view_wrapper
    result = _process_exception(request, e)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/utils/decorators.py", line 190, in _view_wrapper
    response = view_func(request, *args, **kwargs)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/views/decorators/cache.py", line 80, in _view_wrapper
    response = view_func(request, *args, **kwargs)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/contrib/admin/sites.py", line 246, in inner
    return view(request, *args, **kwargs)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/utils/decorators.py", line 48, in _wrapper
    return bound_method(*args, **kwargs)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/utils/decorators.py", line 192, in _view_wrapper
    result = _process_exception(request, e)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/utils/decorators.py", line 190, in _view_wrapper
    response = view_func(request, *args, **kwargs)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/contrib/admin/options.py", line 2024, in changelist_view
    cl = self.get_changelist_instance(request)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/contrib/admin/options.py", line 867, in get_changelist_instance
    return ChangeList(
        request,
    ...<12 lines>...
        self.search_help_text,
    )
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/contrib/admin/views/main.py", line 146, in __init__
    self.get_results(request)
    ~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/contrib/admin/views/main.py", line 313, in get_results
    result_count = paginator.count
                   ^^^^^^^^^^^^^^^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/utils/functional.py", line 47, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ~~~~~~~~~^^^^^^^^^^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/core/paginator.py", line 110, in count
    return c()
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/db/models/query.py", line 604, in count
    return self.query.get_count(using=self.db)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/db/models/sql/query.py", line 644, in get_count
    return obj.get_aggregation(using, {"__count": Count("*")})["__count"]
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django/django/db/models/sql/query.py", line 626, in get_aggregation
    result = compiler.execute_sql(SINGLE)
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django-mongodb-backend/django_mongodb_backend/compiler.py", line 362, in execute_sql
    cursor = query.get_cursor()
  File "/Users/alex.clark/Developer/django-mongodb-cli/src/django-mongodb-backend/django_mongodb_backend/query.py", line 27, in wrapper
    raise DatabaseError(str(e)) from e
django.db.utils.DatabaseError: Aggregation stage $internalFacetTeeConsumer is not allowed or supported with automatic encryption., full error: RawBSONDocument(b'\xa6\x00\x00\x00\x01ok\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02errmsg\x00c\x00\x00\x00Aggregation stage $internalFacetTeeConsumer is not allowed or supported with automatic encryption.\x00\x10code\x00#y\x00\x00\x02codeName\x00\x0e\x00\x00\x00Location31011\x00\x00', codec_options=CodecOptions(document_class=<class 'bson.raw_bson.RawBSONDocument'>, tz_aware=False, uuid_representation=UuidRepresentation.UNSPECIFIED, unicode_decode_error_handler='strict', tzinfo=None, type_registry=TypeRegistry(type_codecs=[], fallback_encoder=None), datetime_conversion=DatetimeConversion.DATETIME))
[19/Nov/2025 14:14:31] "GET /admin/django_mongodb_demo/patient/ HTTP/1.1" 500 425830
[19/Nov/2025 14:14:31] "GET /static/debug_toolbar/css/toolbar.css HTTP/1.1" 304 0
[19/Nov/2025 14:14:31] "GET /static/debug_toolbar/css/print.css HTTP/1.1" 304 0
[19/Nov/2025 14:14:31] "GET /static/debug_toolbar/js/toolbar.js HTTP/1.1" 304 0
[19/Nov/2025 14:14:31] "GET /static/debug_toolbar/js/utils.js HTTP/1.1" 304 0

timgraham and others added 8 commits November 11, 2025 20:55

[temp] remove Django tests for faster CI

82a3eb2

INTPYTHON-527 Add Queryable Encryption support

e3196e2

[wip] updates to field docs

f23c383

add QuerySet tests

7d63045

use shared library instead of mongocryptd

bbf38be

adjust message and add test for missing auto_encryption_opts error

755cb20

polish howto

51da0e9

doc query limitations + docs polish + todos

ac265ce

aclark4life force-pushed the encrypted-admin branch from 6559022 to 8a1d630 Compare November 12, 2025 15:58

aclark4life force-pushed the encrypted-admin branch from 8a1d630 to 159f260 Compare November 12, 2025 16:01

aclark4life force-pushed the INTPYTHON-527 branch from 58dd814 to 301e1b4 Compare November 15, 2025 01:51

aclark4life mentioned this pull request Nov 20, 2025

Remove $facet in top level group stages mongodb/django-mongodb-backend#449

Open

timgraham force-pushed the INTPYTHON-527 branch from 0733fda to bfe70ae Compare November 22, 2025 20:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add encrypted admin w/"no count" pagination #3

Add encrypted admin w/"no count" pagination #3

Uh oh!

aclark4life commented Nov 12, 2025 •

edited

Loading

Uh oh!

timgraham commented Nov 15, 2025

Uh oh!

aclark4life commented Nov 15, 2025

Uh oh!

timgraham commented Nov 16, 2025

Uh oh!

aclark4life commented Nov 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add encrypted admin w/"no count" pagination #3

Are you sure you want to change the base?

Add encrypted admin w/"no count" pagination #3

Uh oh!

Conversation

aclark4life commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

timgraham commented Nov 15, 2025

Uh oh!

aclark4life commented Nov 15, 2025

Uh oh!

timgraham commented Nov 16, 2025

Uh oh!

aclark4life commented Nov 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aclark4life commented Nov 12, 2025 •

edited

Loading