Fix test db name when NAME not provided in DATABASES setting.

Fix Enum error when provided an IntegerChoices value.
Add document about multiple db settings.

Author: jayvynl

CHANGELOG.md

@@ -1,3 +1,36 @@
+### 1.0.2 (2023-02-28)
+- Fix test db name when NAME not provided in DATABASES setting.
+- Fix Enum error when provided an IntegerChoices value.
+- Add document about multiple db settings.
+
+### 1.0.1 (2023-02-23)
+
+- Add `return_int` parameter to `Enum[8|16]Field` to control whether to get an int or str value when querying from the database.
+- Fix TupleField container_class.
+- Add fields documentation.
+
+
+### 1.0.0 (2023-02-21)
+
+- Add tests for migrations.
+- Fix bytes escaping.
+- Fix date and datetime lookup.
+- Add documentations.
+- Add lots of new field types:
+  - Float32/64
+  - [U]Int8/16/32/64/128/256
+  - Date/Date32/DateTime('timezone')/DateTime64('timezone')
+  - String/FixedString(N)
+  - Enum8/16
+  - Array(T)
+  - Bool
+  - UUID
+  - Decimal
+  - IPv4/IPv6
+  - LowCardinality(T)
+  - Tuple(T1, T2, ...)
+  - Map(key, value)
+
 0.2.1 (2022-10-30)
 ---
 

README.md

@@ -56,9 +56,8 @@ $ python setup.py install
 
 ### Configuration
 
-we can use the docker compose file under the project for test and try
 
-Only `ENGINE` is required, other options have default values.
+Only `ENGINE` is required in database setting, other options have default values.
 
 - ENGINE: required, set to `clickhouse_backend.backend`.
 - NAME: database name, default `default`.
@@ -67,21 +66,82 @@ Only `ENGINE` is required, other options have default values.
 - USER: database user, default `default`.
 - PASSWORD: database password, default empty.
 
-  ```python
-  DATABASES = {
-      'default': {
-          'ENGINE': 'clickhouse_backend.backend',
-          'NAME': 'default',
-          'HOST': 'localhost',
-          'USER': 'DB_USER',
-          'PASSWORD': 'DB_PASSWORD',
-          'TEST': {
-              'fake_transaction': True
-          }
-      }
-  }
-  DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
-  ```
+In the most cases, you may just use clickhouse to store some big events tables,
+and use some RDBMS to store other tables.
+Here I give an example setting for clickhouse and postgresql.
+
+```python
+DATABASES = {
+    'default': {
+        'ENGINE': 'django.db.backends.postgresql',
+        'HOST': 'localhost',
+        'USER': 'postgres',
+        'PASSWORD': '123456',
+        'NAME': 'postgres',
+    },
+    'clickhouse': {
+        'ENGINE': 'clickhouse_backend.backend',
+        'NAME': 'default',
+        'HOST': 'localhost',
+        'USER': 'DB_USER',
+        'PASSWORD': 'DB_PASSWORD',
+        'TEST': {
+            'fake_transaction': True
+        }
+    }
+}
+DATABASE_ROUTERS = ['dbrouters.ClickHouseRouter']
+DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
+```
+
+```python
+# dbrouters.py
+from clickhouse_backend.models import ClickhouseModel
+
+def get_subclasses(class_):
+    classes = class_.__subclasses__()
+
+    index = 0
+    while index < len(classes):
+        classes.extend(classes[index].__subclasses__())
+        index += 1
+
+    return list(set(classes))
+
+
+class ClickHouseRouter:
+    def __init__(self):
+        self.route_model_names = set()
+        for model in get_subclasses(ClickhouseModel):
+            if model._meta.abstract:
+                continue
+            self.route_model_names.add(model._meta.label_lower)
+
+    def db_for_read(self, model, **hints):
+        if (model._meta.label_lower in self.route_model_names
+                or hints.get('clickhouse')):
+            return 'clickhouse'
+        return None
+
+    def db_for_write(self, model, **hints):
+        if (model._meta.label_lower in self.route_model_names
+                or hints.get('clickhouse')):
+            return 'clickhouse'
+        return None
+
+    def allow_migrate(self, db, app_label, model_name=None, **hints):
+        if (f'{app_label}.{model_name}' in self.route_model_names
+                or hints.get('clickhouse')):
+            return db == 'clickhouse'
+        elif db == 'clickhouse':
+            return False
+        return None
+```
+
+You should use [database router](https://docs.djangoproject.com/en/4.1/topics/db/multi-db/#automatic-database-routing) to
+automatically route your queries to the right database. In the preceding example, I write a database router which route all
+queries from subclasses of `clickhouse_backend.models.ClickhouseModel` or custom migrations with a `clickhouse` hint key to clickhouse.
+All other queries are routed to the default database (postgresql).
 
 `DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'` is required to working with django migration.
 More details will be covered in [DEFAULT_AUTO_FIELD](https://github.com/jayvynl/django-clickhouse-backend/blob/main/docs/Configurations.md#default_auto_field).
@@ -317,33 +377,7 @@ $ tox
 Changelog
 ---
 
-### 1.0.1 (2023-02-23)
-
-- Add `return_int` parameter to `Enum[8|16]Field` to control whether to get an int or str value when querying from the database.
-- Fix TupleField container_class.
-- Add fields documentation.
-
-
-### 1.0.0 (2023-02-21)
-
-- Add tests for migrations.
-- Fix bytes escaping.
-- Fix date and datetime lookup.
-- Add documentations.
-- Add lots of new field types:
-  - Float32/64
-  - [U]Int8/16/32/64/128/256
-  - Date/Date32/DateTime('timezone')/DateTime64('timezone')
-  - String/FixedString(N)
-  - Enum8/16
-  - Array(T)
-  - Bool
-  - UUID
-  - Decimal
-  - IPv4/IPv6
-  - LowCardinality(T)
-  - Tuple(T1, T2, ...)
-  - Map(key, value)
+[All changelogs](https://github.com/jayvynl/django-clickhouse-backend/blob/main/CHANGELOG.md).
 
 
 License

clickhouse_backend/backend/base.py

@@ -151,6 +151,8 @@ def __init__(self, settings_dict, alias=DEFAULT_DB_ALIAS):
         # from flush at the end of each testcase. Only use this feature when you are
         # aware of the effect in TransactionTestCase.
         self.fake_transaction = settings_dict.get("fake_transaction", False)
+        if not self.settings_dict["NAME"]:
+            self.settings_dict["NAME"] = "default"
 
     @property
     def fake_transaction(self):

clickhouse_backend/backend/schema.py

@@ -406,8 +406,12 @@ def _alter_field(self, model, old_field, new_field, old_type, new_type,
         if old_field.null and not new_field.null:
             old_default = self.effective_default(old_field)
             new_default = self.effective_default(new_field)
+            if hasattr(self, 'skip_default_on_alter'):
+                skip = self.skip_default_on_alter(new_field)
+            else:
+                skip = self.skip_default(new_field)
             if (
-                not self.skip_default_on_alter(new_field) and
+                not skip and
                 old_default != new_default
                 and new_default is not None
             ):

clickhouse_backend/models/fields/__init__.py

@@ -3,6 +3,7 @@
 
 from django.core import checks
 from django.core import exceptions
+from django.db.models import IntegerChoices
 from django.db.models import fields
 from django.utils.functional import cached_property
 from django.utils.itercompat import is_iterable
@@ -345,6 +346,8 @@ def get_prep_value(self, value):
                 value = value.decode("utf-8")
             except UnicodeDecodeError:
                 pass
+        elif isinstance(value, IntegerChoices):
+            value = value.value
         return value
 
     def from_db_value(self, value, expression, connection):

clickhouse_backend/models/sql/compiler.py

@@ -18,10 +18,9 @@ def as_sql(self, *args, **kwargs):
             sql = "%s %s" % (prefix, sql.lstrip())
             if suffix:
                 sql = "%s %s" % (sql, suffix)
-        setting_info = getattr(self.query, "setting_info", None)
-        if setting_info:
+        if getattr(self.query, "setting_info", None):
             setting_sql, setting_params = self.connection.ops.settings_sql(
-                **setting_info
+                **self.query.setting_info
             )
             sql = "%s %s" % (sql, setting_sql)
             params = (*params, *setting_params)
@@ -169,7 +168,7 @@ def as_sql(self):
             result.append("WHERE 1")
 
         params = (*update_params, *params)
-        if hasattr(self.query, "setting_info") and self.query.setting_info:
+        if getattr(self.query, "setting_info", None):
             setting_sql, setting_params = self.connection.ops.settings_sql(
                 **self.query.setting_info
             )

clickhouse_backend/models/sql/query.py

@@ -18,7 +18,7 @@ def __init__(self, model, where=query.WhereNode, alias_cols=True):
 
     def clone(self):
         obj = super().clone()
-        obj.settings = self.setting_info.copy()
+        obj.setting_info = self.setting_info.copy()
         return obj
 
     def explain(self, using, format=None, type=None, **settings):
@@ -28,15 +28,18 @@ def explain(self, using, format=None, type=None, **settings):
         return "\n".join(compiler.explain_query())
 
 
-def clone_factory(cls):
-    old = cls.clone
+def clone_decorator(cls):
+    old_clone = cls.clone
 
     def clone(self):
-        obj = old(self)
-        obj.settings = self.setting_info.copy()
+        obj = old_clone(self)
+        if hasattr(obj, "setting_info"):
+            obj.setting_info = self.setting_info.copy()
         return obj
-    return clone
 
+    cls.clone = clone
+    return cls
 
-subqueries.UpdateQuery.clone = clone_factory(subqueries.UpdateQuery)
-subqueries.DeleteQuery.clone = clone_factory(subqueries.DeleteQuery)
+
+clone_decorator(subqueries.UpdateQuery)
+clone_decorator(subqueries.DeleteQuery)

example/config/settings.py

@@ -71,6 +71,9 @@
 DATABASES = {
     'default': {
         'ENGINE': 'clickhouse_backend.backend',
+        'TEST': {
+            'MIGRATE': False
+        }
     }
 }
 

example/testapp/models.py

@@ -10,7 +10,7 @@ class Action(IntegerChoices):
         DROP = 2
         ALERT = 3
     ip = models.GenericIPAddressField(default='::')
-    ipv4 = models.GenericIPAddressField(default="127.0.0.1")
+    ipv4 = models.IPv4Field(default="127.0.0.1")
     ip_nullable = models.GenericIPAddressField(null=True)
     port = models.UInt16Field(default=0)
     protocol = models.StringField(default='', low_cardinality=True)

example/testapp/tests.py

@@ -1,3 +1,24 @@
 from django.test import TestCase
 
-# Create your tests here.
+from . import models
+
+
+class Tests(TestCase):
+    @classmethod
+    def setUpTestData(cls):
+        models.Event.objects.bulk_create([
+            models.Event() for _ in range(10)
+        ])
+
+    def test_create(self):
+        models.Event.objects.create()
+        self.assertEqual(models.Event.objects.count(), 11)
+
+    def test_update(self):
+        event = models.Event.objects.create()
+        models.Event.objects.filter(id=event.id).settings(mutations_sync=1).update(protocol="TCP")
+        self.assertTrue(models.Event.objects.filter(protocol="TCP").exists())
+
+    def test_delete(self):
+        models.Event.objects.settings(mutations_sync=1).delete()
+        self.assertFalse(models.Event.objects.exists())