Merge pull request #23 from fsbraun/main

chore: Merge changes for versions 0.03 to 0.2 into original repo

Author: fsbraun

.github/workflows/test.yml

@@ -0,0 +1,51 @@
+name: migration test
+
+on: [push, pull_request]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  migrate-project:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        django-version: [
+          '4.2',
+        ]
+        python-version: ['3.11']
+        os: [
+          ubuntu-latest,
+        ]
+
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v4
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Create dummy project
+      run: |
+        python -m venv .venv
+        source ./.venv/bin/activate
+        python -m pip install --upgrade pip
+        pip install "djangocms-frontend[cms-3]<2" Django~=${{ matrix.django-version }}
+        django-admin startproject --template https://github.com/django-cms/cms-template/archive/migration.zip cmsproject
+        pip install -e .
+        cd cmsproject
+        python -m manage migrate
+        python -m manage cms check
+        python -m djangocms_4_migration.test
+    - name: Upgrade to django CMS 4
+      run: |
+        source ./.venv/bin/activate
+        cd cmsproject
+        pip install -U "django-cms<5" djangocms-versioning djangocms-alias
+        DJANGO_SETTINGS_MODULE="cmsproject.settings4" python -m manage cms4_migration
+    - name: Run tests
+      run: |
+        source ./.venv/bin/activate
+        cd cmsproject
+        python -m djangocms_4_migration.test

CHANGELOG.md

@@ -1,11 +1,28 @@
 # Changelog
 
-## (unreleased)
+## 0.2.0 (unreleased)
 
+* feat: Migrate permissions by @fsbraun in https://github.com/fsbraun/djangocms-4-migration/pull/7
+* feat: ensure page urls are unique by @fsbraun in https://github.com/fsbraun/djangocms-4-migration/pull/8
+
+## 0.1.0 (2025-03-14)
+
+* feature: Update foreign key relations to page objects
+* feature: Update soft relation of djangocms-link 5+
+* feature: Update soft relation of djangocms-frontend objects
+* fix: Customer model username retrieval during migration
+* fix: Crashed when running w/o `CMS_MIGRATION_USER_ID` setting
+
+## 0.0.3
+
+* feature: Update foreign key relations to page objects
+* feature: Update soft relation of djangocms-link 5+
+* feature: Update soft relation of djangocms-frontend objects
+* fix: Customer model username retrieval during migration
 * feature: Add support for custom user models
 
 ## 0.0.2 (2023-07-11)
 * fix: Added the Github project url to the setup.py file for PyPi linking
 
 ## 0.0.1 (2021-01-12)
-* Initial commit containing basic django CMS 3 page and content migration with django-cms-alias and djangocms-versioning integration. 
+* Initial commit containing basic django CMS 3 page and content migration with django-cms-alias and djangocms-versioning integration.

README.md

@@ -6,7 +6,10 @@
 This package is designed to migrate a django CMS 3.5+ project to django CMS 4.0.
 
 ## What does this package do?
-- Keeps any draft and published content, ensuring that any new draft changes are kept as a new draft version in djangocms_versioning. 
+- Keeps any draft and published content, ensuring that any new draft changes are kept as a new draft version in djangocms_versioning.
+- Creates aliases for static placeholder
+- Migrates alias plugins
+- Runs django CMS' migrations
 
 ## Limitations
 Due to the nature of the changes between django CMS 3.5+ and 4.0 the package will fail to function if an incompatible package is installed. 
@@ -27,36 +30,48 @@ Requires knowledge of django CMS Versioning
 ### Install the following packages
 The following packages are not yet officially released, they need to be installed directly from the repository. We need your help to make packages v4.0 compatible and to provide documentation for the wider community!
 
-django CMS 4.0
+django CMS 4.0+
 ```
-pip install http://github.com/divio/django-cms/tarball/release/4.0.x#egg=django-cms
+pip install django-cms
 ```
 
 djangocms-text-ckeditor
 ```
-pip install https://github.com/divio/djangocms-text-ckeditor/tarball/support/4.0.x#egg=djangocms-text-ckeditor
+pip install djangocms-text-ckeditor
 ```
 
 djangocms-versioning
 ```
-pip install https://github.com/divio/djangocms-versioning/tarball/master#egg=djangocms-versioning
+pip install djangocms-versioning
 ```
 
 djangocms-alias
 ```
-pip install https://github.com/divio/djangocms-alias/tarball/master#egg=djangocms-alias
+pip install djangocms-alias
 ```
 
 ## Installation
 **Warning**: This package can leave your DB in a corrupted state if used incorrectly, be sure to backup any databases prior to running any commands listed here!
 
 First install this package in your project
 ```
-pip install djangocms-4-migration
+pip install git+https://github.com/fsbraun/djangocms-4-migration
 ```
 
 ## Configuration
 
+Add the migration tool to `INSTALLED_APPS` temporarily. In your `settings.py` make sure that it is listed. You can remove it after the migration process.
+```
+INSTALLED_APPS = [
+    ...,
+    "djangocms_4_migration",
+    "djangocms_versioning",
+    "djangocms_alias",
+    ...,
+]
+CMS_CONFIRM_VERSION4 = True
+```
+
 If you have a custom user model, you should designate a "migration user" by specifying the user ID in your settings like so:
 
 ```
@@ -70,6 +85,13 @@ Simply run the following command to run the data migration.
 python manage.py cms4_migration
 ```
 
+You can ignore warnings of the form 
+```
+UserWarning: No user has been supplied when creating a new AliasContent object.
+No version could be created. Make sure that the creating code also creates a
+Version objects or use AliasContent.objects.with_user(user).create(...)
+```
+
 ## Common solutions for django CMS 4.0 compatibility
 
 Import PageContent in a backwards compatible way (Title).

djangocms_4_migration/__init__.py

@@ -1 +1 @@
-__version__ = "0.0.2"
+__version__ = "0.2.0"

djangocms_4_migration/helpers.py

@@ -14,7 +14,7 @@ def get_or_create_migration_user(user_model=get_user_model()):
     This is the user that is used to automatically attach to new items created as
     part of the cms migration.
     """
-    if getattr(settings, "CMS_MIGRATION_USER_ID"):
+    if getattr(settings, "CMS_MIGRATION_USER_ID", None):
         return user_model.objects.get(id=settings.CMS_MIGRATION_USER_ID), False
     return user_model.objects.get_or_create(
         username='djangocms_4_migration_user',

djangocms_4_migration/management/commands/migrate_alias_plugins.py

@@ -210,7 +210,6 @@ def process_old_alias_sources(site, language, site_plugin_queryset):
             # Create version
             changed_by = User.objects.get(**{User.USERNAME_FIELD: old_plugin.placeholder.source.changed_by})
             version = Version.objects.create(content=alias_content, created_by=changed_by)
-            version.save()
             version.publish(changed_by)
 
         # create csm4 alias plugins for cms3 alias references
@@ -254,8 +253,7 @@ class Command(BaseCommand):
     def handle(self, *args, **options):
         with transaction.atomic():
             # Alias source plugin list
-            cms3_alias_ref_ids = AliasPluginModel.objects.values('plugin_id').order_by('plugin_id').distinct('plugin_id')
-            plugin_id_list = [cms3_plugin['plugin_id'] for cms3_plugin in cms3_alias_ref_ids if cms3_plugin['plugin_id']]
+            plugin_id_list = list(AliasPluginModel.objects.values_list('plugin_id', flat=True).order_by('plugin_id'))
             alias_source_total = len(plugin_id_list)
             # Alias references list count
             alias_reference_total = AliasPluginModel.objects.count()

djangocms_4_migration/management/commands/migrate_static_placeholders.py

@@ -124,7 +124,7 @@ def _get_or_create_alias(category, static_code, site):
 
 
 def _create_alias_content(alias, name, language, user, state=PUBLISHED):
-    alias_content = AliasContent.objects.create(
+    alias_content = AliasContent.objects.with_user(user).create(
         alias=alias,
         name=name,
         language=language,

djangocms_4_migration/management/commands/migration_cleanup.py

@@ -1,5 +1,7 @@
 import logging
+from packaging.version import Version as PgkVersion
 
+from django.conf import settings
 from django.core.management.base import BaseCommand
 from django.contrib.contenttypes.models import ContentType
 from django.db import connection
@@ -17,6 +19,84 @@
 logger = logging.getLogger(__name__)
 
 
+def _fix_link_plugins(page):
+    """djangocms-link with version above 5.0.0 stores only "soft" references to pages in JSON fields which will not be
+    updated by `_fix_page_refernces`"""
+    if "djangocms_link" in settings.INSTALLED_APPS:
+        from djangocms_link import __version__
+        from djangocms_link.models import Link
+
+        if PgkVersion(__version__) >= PgkVersion("5.0.0"):
+            for link in Link.objects.all():
+                if "internal_link" in link.link and link.link["internal_link"].startswith("cms.page:"):
+                    _, linked_page_id = link.link["internal_link"].split(":")
+                    if linked_page_id == str(page.pk):
+                        replacement_page = Page.objects.filter(node_id=page.node_id).exclude(id=page.id).get()
+                        logger.info("Fixing link reference from Page %s to %s", page.id, replacement_page.id)
+                        link.link["internal_link"] = f"cms.page:{replacement_page.pk}"
+                        link.save()
+
+def _fix_frontend_refernces(page):
+    """djangocms-frontend stores only "soft" references to pages in JSON fields which will not be
+    updated by `_fix_page_refernces`"""
+    def search(json, reference, pk):
+        changed = False
+        for key, value in json.items():
+            if key == "internal_link" and value.startswith(reference + ":"):
+                # Update link field
+                _, linked_page_id = value.split(":")
+                if linked_page_id == str(pk):
+                    replacement_page = Page.objects.filter(node_id=page.node_id).exclude(id=page.id).get()
+                    json[key] = f"{reference}:{replacement_page.pk}"
+                    changed = True
+            elif isinstance(value, dict) and "model" in value and value["model"] == reference and "pk" in value:
+                # Update reference
+                if value["pk"] == pk:
+                    replacement_page = Page.objects.filter(node_id=page.node_id).exclude(id=page.id).get()
+                    value["pk"] = replacement_page.pk
+                    changed = True
+            elif isinstance(value, dict):
+                # search recursively
+                changed = changed or search(value, reference, pk)
+        return changed
+
+    if "djangocms_frontend" in settings.INSTALLED_APPS:
+        from djangocms_frontend.models import FrontendUIItem
+
+        for frontend_component in FrontendUIItem.objects.all():
+            if search(frontend_component.config, "cms.page", page.pk):
+                frontend_component.save()
+
+
+def _fix_page_references(page):
+    relations = [
+        f
+        for f in Page._meta.get_fields()
+        if (f.one_to_many or f.one_to_one or f.many_to_many)
+        and f.auto_created
+        and not f.concrete
+    ]
+
+    replacement_page = Page.objects.filter(node_id=page.node_id).exclude(id=page.id).get()
+    logger.info("Fixing reference from Page %s to %s", page.id, replacement_page.id)
+
+    for rel in relations:
+        model = rel.related_model
+        if rel.one_to_one:
+            # One to one relationships should not be duplicated, so just delete object
+            model.objects.filter(**{rel.field.name: page}).delete()
+        elif rel.many_to_many:
+            m2m_objs = model.objects.filter(**{rel.field.name: page})
+            for m2m_obj in m2m_objs:
+                m2m_rel = getattr(m2m_obj, rel.field.name)
+                m2m_rel.remove(page)
+                m2m_rel.add(replacement_page)
+        else:
+            model.objects.filter(**{rel.field.name: page}).update(
+                **{rel.field.name: replacement_page}
+            )
+
+
 def _delete_page(page):
     try:
         logger.info("Deleting Page %s" % page.id)
@@ -76,13 +156,17 @@ def handle(self, *args, **options):
             page_content_list = _get_page_contents(page)
 
             if not page_content_list.exists():
+                _fix_page_references(page)
+                _fix_link_plugins(page)
+                _fix_frontend_refernces(page)
                 _delete_page(page)
                 stats['page_deleted'] = stats['page_deleted'] + 1
                 continue
 
             stats['pagecontents_count'] = stats['pagecontents_count'] + page_content_list.count()
 
             # Find if each PageContents has versions attached.
+            languages = []
             for page_content in page_content_list:
                 # If there are no versions for the pagecontents clean them out as they are not required
                 if not Version.objects.filter(
@@ -92,5 +176,12 @@ def handle(self, *args, **options):
                     _delete_page_content_placeholders(page_content_contenttype, page_content)
                     _delete_page_content(page_content)
                     stats['pagecontents_deleted'] = stats['pagecontents_deleted'] + 1
+                else:
+                    languages.append(page_content.language)
+
+            for language in languages:
+                # Delete redundant page urls (the first is the published one - keep it)
+                for url in page.urls.filter(language=language).order_by("pk")[1:]:
+                    url.delete()
 
         logger.info("Stats: %s", str(stats))

djangocms_4_migration/migrations/0003_page_version_integration_data_migration.py

@@ -1,7 +1,7 @@
 import logging
-import os
 
 from django.conf import settings
+from django.contrib.auth import get_user_model
 from django.db import migrations
 from djangocms_versioning.constants import ARCHIVED, DRAFT, PUBLISHED
 
@@ -15,6 +15,8 @@
 # Page was marked published, but some of page parents are not.
 CMS_3_PUBLISHER_STATE_PENDING = 4
 
+USERNAME_FIELD = getattr(get_user_model(), "USERNAME_FIELD", "username")
+
 
 def forwards(apps, schema_editor):
     db_alias = schema_editor.connection.alias
@@ -64,7 +66,7 @@ def _create_version(page_content, state=DRAFT, number=1):
         # Find the user
         try:
             created_by = User.objects.using(db_alias).get(
-                **{User.USERNAME_FIELD: page_content.page.created_by}
+                **{USERNAME_FIELD: page_content.page.created_by}
             )
         except:
             # Use the first super user as the author as a fall back
@@ -88,7 +90,7 @@ def _create_version(page_content, state=DRAFT, number=1):
     for existing_title in PageData.objects.using(db_alias).all():
         """
         If Title was published keep it and create a version
-        If Title was not published 
+        If Title was not published
         """
         logger.info("Existing title: {}".format(str(existing_title.title_id)))
 
@@ -99,7 +101,6 @@ def _create_version(page_content, state=DRAFT, number=1):
         # If this is the draft page
         if existing_title.publisher_is_draft:
             _handle_draft_page(existing_title)
-
         # Otherwise this is the published page
         else:
             _handle_public_page(existing_title)