Transactions now support models routed to different databases.

Author: ses4j

CHANGELOG.md

@@ -2,26 +2,39 @@
 
 ## [Unreleased]
 
+## [3.0.1] - 2020-08-05
+
+### Updated
+
+-   Transactions now support models routed to different databases.
+
 ## [3.0.0] - 2020-07-28
+
 ### Changed
-- **Potentially breaking change** Retain PK if provided and ensure a crash if it mismatches on key_field. #10  
 
-  If your input models have the PK set already and expect it to be cleared by bulk_sync, that won't happen anymore.
-  Please clear the pks yourself if you want them to seem like "new" objects.
+-   **Potentially breaking change** Retain PK if provided and ensure a crash if it mismatches on key_field. #10
+
+    If your input models have the PK set already and expect it to be cleared by bulk_sync, that won't happen anymore.
+    Please clear the pks yourself if you want them to seem like "new" objects.
 
 ## [2.1.0] - 2020-06-05
+
 ### Added
-- Added support for skip_deletes, skip_creates, and skip_updates in bulk_sync method. #9, pull request from [@mikefreemanwd](https://github.com/mikefreemanwd).
+
+-   Added support for skip_deletes, skip_creates, and skip_updates in bulk_sync method. #9, pull request from [@mikefreemanwd](https://github.com/mikefreemanwd).
 
 ### Updated
-- Updated stats returned by bulk_sync to reflect what actually happened given the flags rather than what would have happend if all flags are false.
+
+-   Updated stats returned by bulk_sync to reflect what actually happened given the flags rather than what would have happend if all flags are false.
 
 ## [2.0.0] - 2020-05-19
+
 ### Removed
-- Removed support for Django versions before 2.2.  Please use 1.x series for Django < 2.2.
 
+-   Removed support for Django versions before 2.2. Please use 1.x series for Django < 2.2.
 
-[Unreleased]: https://github.com/mathandpencil/django-bulk-sync/compare/v3.0.0..HEAD
+[unreleased]: https://github.com/mathandpencil/django-bulk-sync/compare/v3.0.0..HEAD
+[3.0.1]: https://github.com/mathandpencil/django-bulk-sync/compare/v3.0.0..v3.0.1
 [3.0.0]: https://github.com/mathandpencil/django-bulk-sync/compare/v2.1.0..v3.0.0
 [2.1.0]: https://github.com/mathandpencil/django-bulk-sync/compare/v2.0.0..v2.1.0
-[2.0.0]: https://github.com/mathandpencil/django-bulk-sync/releases/tag/v2.0.0
+[2.0.0]: https://github.com/mathandpencil/django-bulk-sync/releases/tag/v2.0.0

README.md

@@ -6,6 +6,8 @@ Combine bulk create, update, and delete into a single call.
 
 It manages all necessary creates, updates, and deletes with as few database calls as possible to maximize performance.
 
+It can use either database PKs or `key_fields` to match up objects with existing records.
+
 ## Installation
 
 The package is available on pip as [django-bulk-sync][django-bulk-sync]. Run:
@@ -43,10 +45,12 @@ for line in company_import_file:
 	new_models.append(e)
 
 # `filters` controls the subset of objects considered when deciding to
-# update or delete.
+# update or delete.  Here we sync only company 501 employees.
 filters = Q(company_id=501)
+
 # `key_fields` matches an existing object if all `key_fields` are equal.
 key_fields = ('name', )
+
 ret = bulk_sync(
         new_models=new_models,
         filters=filters,
@@ -62,27 +66,28 @@ Under the hood, it will atomically call `bulk_create`, `bulk_update`, and a sing
 
 ## Argument Reference
 
-`def bulk_sync(new_models, key_fields, filters, batch_size=None, fields=None):`
+`def bulk_sync(new_models, key_fields, filters, batch_size=None, fields=None, skip_creates=False, skip_updates=False, skip_deletes=False):`
 Combine bulk create, update, and delete. Make the DB match a set of in-memory objects.
 
 -   `new_models`: An iterable of Django ORM `Model` objects that you want stored in the database. They may or may not have `id` set, but you should not have already called `save()` on them.
--   `key_fields`: Identifying attribute name(s) to match up `new_models` items with database rows. If a foreign key is being used as a key field, be sure to pass the `fieldname_id` rather than the `fieldname`.
+-   `key_fields`: Identifying attribute name(s) to match up `new_models` items with database rows. If a foreign key is being used as a key field, be sure to pass the `fieldname_id` rather than the `fieldname`. Use `['pk']` if you know the PKs already and want to use them to identify and match up `new_models` with existing database rows.
 -   `filters`: Q() filters specifying the subset of the database to work in. Use `None` or `[]` if you want to sync against the entire table.
--   `batch_size`: passes through to Django `bulk_create.batch_size` and `bulk_update.batch_size`, and controls how many objects are created/updated per SQL query.
+-   `batch_size`: (optional) passes through to Django `bulk_create.batch_size` and `bulk_update.batch_size`, and controls how many objects are created/updated per SQL query.
 -   `fields`: (optional) List of fields to update. If not set, will sync all fields that are editable and not auto-created.
--   `skip_creates`: If truthy, will not perform any object creations needed to fully sync. Defaults to not skip.
--   `skip_updates`: If truthy, will not perform any object updates needed to fully sync. Defaults to not skip. 
--   `skip_deletes`: If truthy, will not perform any object deletions needed to fully sync. Defaults to not skip. 
+-   `skip_creates`: (optional) If truthy, will not perform any object creations needed to fully sync. Defaults to not skip.
+-   `skip_updates`: (optional) If truthy, will not perform any object updates needed to fully sync. Defaults to not skip.
+-   `skip_deletes`: (optional) If truthy, will not perform any object deletions needed to fully sync. Defaults to not skip.
 
 -   Returns a dict:
-
+    ```
     {
     'stats': {
-    "created": number of `new_models` not found in database and so created,
-    "updated": number of `new_models` that were found in database as matched by `key_fields`,
-    "deleted": number of deleted objects - rows in database that matched `filters` but were not present in `new_models`.
-    }
+        "created": number of `new_models` not found in database and so created,
+        "updated": number of `new_models` that were found in database as matched by `key_fields`,
+        "deleted": number of deleted objects - rows in database that matched `filters` but were not present in `new_models`.
+        }
     }
+    ```
 
 `def bulk_compare(old_models, new_models, key_fields, ignore_fields=None):`
 Compare two sets of models by `key_fields`.
@@ -92,7 +97,16 @@ Compare two sets of models by `key_fields`.
 -   `key_fields`: Identifying attribute name(s) to match up `new_models` items with database rows. If a foreign key
     is being used as a key field, be sure to pass the `fieldname_id` rather than the `fieldname`.
 -   `ignore_fields`: (optional) If set, provide field names that should not be considered when comparing objects.
--   Returns dict of: `{ 'added': list of all added objects. 'unchanged': list of all unchanged objects. 'updated': list of all updated objects. 'updated_details': dict of {obj: {field_name: (old_value, new_value)}} for all changed fields in each updated object. 'removed': list of all removed objects. }`
+-   Returns dict:
+    ```
+        {
+            'added': list of all added objects.
+            'unchanged': list of all unchanged objects.
+            'updated': list of all updated objects.
+            'updated_details': dict of {obj: {field_name: (old_value, new_value)}} for all changed fields in each updated object.
+            'removed': list of all removed objects.
+        }
+    ```
 
 ## Frameworks Supported
 

bulk_sync/__init__.py

@@ -1,33 +1,45 @@
 from collections import OrderedDict
 import logging
-from django.db import transaction
+from django.db import transaction, router
 
 logger = logging.getLogger(__name__)
 
 
-def bulk_sync(new_models, key_fields, filters, batch_size=None, fields=None, skip_creates=False, skip_updates=False, skip_deletes=False):
+def bulk_sync(
+    new_models,
+    key_fields,
+    filters,
+    batch_size=None,
+    fields=None,
+    skip_creates=False,
+    skip_updates=False,
+    skip_deletes=False,
+):
     """ Combine bulk create, update, and delete.  Make the DB match a set of in-memory objects.
 
     `new_models`: Django ORM objects that are the desired state.  They may or may not have `id` set.
     `key_fields`: Identifying attribute name(s) to match up `new_models` items with database rows.  If a foreign key
             is being used as a key field, be sure to pass the `fieldname_id` rather than the `fieldname`.
     `filters`: Q() filters specifying the subset of the database to work in.  Use `None` or `[]` if you want to sync against the entire table.
-    `batch_size`: passes through to Django `bulk_create.batch_size` and `bulk_update.batch_size`, and controls
+    `batch_size`: (optional) passes through to Django `bulk_create.batch_size` and `bulk_update.batch_size`, and controls
             how many objects are created/updated per SQL query.
     `fields`: (optional) list of fields to update. If not set, will sync all fields that are editable and not auto-created.
     `skip_creates`: If truthy, will not perform any object creations needed to fully sync. Defaults to not skip.
-    `skip_updates`: If truthy, will not perform any object updates needed to fully sync. Defaults to not skip. 
-    `skip_deletes`: If truthy, will not perform any object deletions needed to fully sync. Defaults to not skip. 
+    `skip_updates`: If truthy, will not perform any object updates needed to fully sync. Defaults to not skip.
+    `skip_deletes`: If truthy, will not perform any object deletions needed to fully sync. Defaults to not skip.
     """
     db_class = new_models[0].__class__
 
     if fields is None:
         # Get a list of fields that aren't PKs and aren't editable (e.g. auto_add_now) for bulk_update
-        fields = [field.name
-                  for field in db_class._meta.fields
-                  if not field.primary_key and not field.auto_created and field.editable]
-
-    with transaction.atomic():
+        fields = [
+            field.name
+            for field in db_class._meta.fields
+            if not field.primary_key and not field.auto_created and field.editable
+        ]
+
+    using = router.db_for_write(db_class)
+    with transaction.atomic(using=using):
         objs = db_class.objects.all()
         if filters:
             objs = objs.filter(filters)
@@ -51,7 +63,7 @@ def get_key(obj):
 
         if not skip_creates:
             db_class.objects.bulk_create(new_objs, batch_size=batch_size)
-            
+
         if not skip_updates:
             db_class.objects.bulk_update(existing_objs, fields=fields, batch_size=batch_size)
 
@@ -62,9 +74,9 @@ def get_key(obj):
         assert len(existing_objs) == len(new_models) - len(new_objs)
 
         stats = {
-            "created": 0 if skip_creates else len(new_objs), 
-            "updated": 0 if skip_updates else (len(new_models) - len(new_objs)), 
-            "deleted": 0 if skip_deletes else len(obj_dict)
+            "created": 0 if skip_creates else len(new_objs),
+            "updated": 0 if skip_updates else (len(new_models) - len(new_objs)),
+            "deleted": 0 if skip_deletes else len(obj_dict),
         }
 
         logger.debug(

setup.py

@@ -5,20 +5,20 @@
 
 setuptools.setup(
     name="django-bulk-sync",
-    version='3.0.0',
+    version="3.0.1",
     description="Combine bulk add, update, and delete into a single call.",
     long_description=long_description,
-    long_description_content_type='text/markdown',
+    long_description_content_type="text/markdown",
     url="https://github.com/mathandpencil/django-bulk-sync",
     author="Scott Stafford",
     author_email="[email protected]",
     packages=setuptools.find_packages(),
     classifiers=[
-        'Programming Language :: Python :: 3',
-        'License :: OSI Approved :: MIT License',
-        'Operating System :: OS Independent',
-        'Framework :: Django',
-        'Framework :: Django :: 2.2',
+        "Programming Language :: Python :: 3",
+        "License :: OSI Approved :: MIT License",
+        "Operating System :: OS Independent",
+        "Framework :: Django",
+        "Framework :: Django :: 2.2",
     ],
     zip_safe=False,
 )