Skip to content

Add ColumnPrefixIndex for MySQL column prefix key parts #1151

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 31 additions & 1 deletion docs/exposition.rst
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ field class allows you to interact with those fields:
Tiny integer fields
-------------------

MySQLs ``TINYINT`` type stores small integers efficiently, using just one byte.
MySQL's ``TINYINT`` type stores small integers efficiently, using just one byte.
Django-MySQL provides field classes for the ``TINYINT`` type:

.. code-block:: python
Expand Down Expand Up @@ -355,3 +355,33 @@ specific one:
self.run_it()

:ref:`Read more <test_utilities>`

-------
Indexes
-------

MySQL-specific index types for optimizing your queries.

Column Prefix Index
-------------------

An index class that allows you to create indexes with column prefix lengths,
which can be both space-efficient and useful for queries that match against
the start of strings:

.. code-block:: python

class Article(Model):
title = models.CharField(max_length=200)
content = models.TextField()

class Meta:
indexes = [
ColumnPrefixIndex(
fields=["title", "content"],
prefix_lengths=(10, 50),
name="title_content_prefix_idx",
),
]

:doc:`Read more <indexes>`
1 change: 1 addition & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ action, or get started with :doc:`installation`. Otherwise, take your pick:
migration_operations
form_fields
validators
indexes
cache
locks
status
Expand Down
64 changes: 64 additions & 0 deletions docs/indexes.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
Indexes
=======

.. currentmodule:: django_mysql.models

Django-MySQL includes a custom index class that extends Django's built-in index
functionality for MySQL-specific features.

ColumnPrefixIndex
-------------------

.. class:: ColumnPrefixIndex(*expressions, prefix_lengths, **kwargs)

A custom index class that allows you to create indexes with column prefix
lengths in MySQL. This is particularly useful for indexing ``TEXT`` or long
``VARCHAR`` columns where you want to index only the first N characters.

MySQL allows you to create indexes that only include the first N characters of
a column, which can be both space-efficient and useful for queries that match
against the start of strings, such as ``istartswith`` lookups.

For more details about column prefix indexes in MySQL, see the
`MySQL documentation on CREATE INDEX column-prefixes
<https://dev.mysql.com/doc/refman/8.0/en/create-index.html#create-index-column-prefixes>`_.

**Arguments:**

* ``expressions``: The fields or expressions to index
* ``prefix_lengths``: A sequence of integers specifying the prefix length for each column
* ``**kwargs``: Additional arguments passed to Django's Index class

**Example:**

.. code-block:: python

from django.db import models
from django_mysql.models import ColumnPrefixIndex


class Article(models.Model):
title = models.CharField(max_length=200)
content = models.TextField()

class Meta:
indexes = [
ColumnPrefixIndex(
fields=["title", "content"],
prefix_lengths=(10, 50),
name="title_content_prefix_idx",
),
]

This will create an index equivalent to the following SQL:

.. code-block:: sql

CREATE INDEX title_content_prefix_idx ON article (title(10), content(50));

Such indexes can be particularly efficient for ``istartswith`` queries:

.. code-block:: python

# This query can use the prefix index
Article.objects.filter(title__istartswith="Django")
41 changes: 41 additions & 0 deletions src/django_mysql/models/indexes.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
from __future__ import annotations

from collections.abc import Sequence
from typing import Any

from django.db.backends.base.schema import BaseDatabaseSchemaEditor
from django.db.backends.ddl_references import Statement
from django.db.models import Index, Model
from django.db.models.expressions import BaseExpression, Combinable


class ColumnPrefixIndex(Index):
def __init__(
self,
*expressions: BaseExpression | Combinable | str,
prefix_lengths: Sequence[int],
**kwargs: Any,
) -> None:
super().__init__(*expressions, **kwargs)
self.prefix_lengths = tuple(prefix_lengths)

def deconstruct(self) -> Any:
path, args, kwargs = super().deconstruct()
if self.prefix_lengths is not None:
kwargs["prefix_lengths"] = self.prefix_lengths
return path, args, kwargs

def create_sql(
self,
model: type[Model],
schema_editor: BaseDatabaseSchemaEditor,
using: str = "",
**kwargs: Any,
) -> Statement:
statement = super().create_sql(model, schema_editor, using=using, **kwargs)
qn = schema_editor.quote_name
statement.parts["columns"] = ", ".join(
f"{qn(model._meta.get_field(field).column)}({length})"
for field, length in zip(self.fields, self.prefix_lengths)
)
return statement
68 changes: 68 additions & 0 deletions tests/testapp/test_index.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
from __future__ import annotations

from django.db import connection
from django.test import SimpleTestCase, TransactionTestCase

from django_mysql.models.indexes import ColumnPrefixIndex
from tests.testapp.models import CharSetModel


class ColumnPrefixIndexTests(SimpleTestCase):
def test_deconstruct(self):
index = ColumnPrefixIndex(
fields=["field", "field2"],
prefix_lengths=(10, 20),
name="dm_field_field2_pfx",
)
path, args, kwargs = index.deconstruct()
self.assertEqual(path, "django_mysql.models.indexes.ColumnPrefixIndex")
self.assertEqual(args, ())
self.assertEqual(
kwargs,
{
"name": "dm_field_field2_pfx",
"fields": ["field", "field2"],
"prefix_lengths": (10, 20),
},
)


class SchemaTests(TransactionTestCase):
def get_constraints(self, table):
return connection.introspection.get_constraints(connection.cursor(), table)

def test_column_prefix_index_create_sql(self):
index = ColumnPrefixIndex(
fields=["field", "field2"],
prefix_lengths=(10, 20),
name="dm_name_email_pfx",
)
with connection.schema_editor() as editor:
statement = index.create_sql(CharSetModel, editor)
sql = str(statement)

self.assertIn("`field`(10)", sql)
self.assertIn("`field2`(20)", sql)

def test_column_prefix_index(self):
table = CharSetModel._meta.db_table
index_name = "dm_name_email_pfx"
index = ColumnPrefixIndex(
fields=["field", "field2"], prefix_lengths=(10, 20), name=index_name
)

# Ensure the table is there and doesn't have an index.
self.assertNotIn(index_name, self.get_constraints(table))

# Add the index.
with connection.schema_editor() as editor:
editor.add_index(CharSetModel, index)

constraints = self.get_constraints(table)
self.assertIn(index_name, constraints)
self.assertEqual(constraints[index_name]["type"], ColumnPrefixIndex.suffix)

# Drop the index.
with connection.schema_editor() as editor:
editor.remove_index(CharSetModel, index)
self.assertNotIn(index_name, self.get_constraints(table))