Auto-resolve trivial migrations

[pavelzw]

We are using copier quite a bit at my company and it works really well, thanks a lot!
One thing we often stumbled upon was that we have trivial merge conflicts like this

<<<<<<< before updating
        uses: prefix-dev/setup-pixi@19eac09b398e3d0c747adc7921926a6d802df4da # v0.8.8
=======
        uses: prefix-dev/setup-pixi@14c8aabd75893f83f4ab30c03e7cf853c8208961 # v0.8.10
>>>>>>> after updating

or

<<<<<<< before updating
        uses: prefix-dev/setup-pixi@8ca4608ef7f4daeb54f5205b20d0b7cb42f11143 # v0.8.14
||||||| last update
        uses: prefix-dev/setup-pixi@14c8aabd75893f83f4ab30c03e7cf853c8208961 # v0.8.10
=======
        uses: prefix-dev/setup-pixi@fef5c9568ca6c4ff7707bf840ab0692ba3f08293 # v0.9.0
>>>>>>> after updating

This is because next to a copier updater, we also have dependabot running for github workflow files that are not tracked by copier.
These merge conflicts are quite annoying to deal with on a large scale as it's stupid work that needs to be repeated over and over.

For this, we have written a custom after migration that handles these things.

# copier.yml
# ...

_migrations:
  - command: '"{{ _copier_python }}" -I -c "import sys; sys.path.insert(0, \"{{ _copier_conf.src_path }}\"); import runpy;
      runpy.run_module(\"migrations\", run_name=\"__main__\")"'
    when: true

# migrations/__main__.py
import logging
import os

from packaging.version import Version

from migrations.conflict_resolution import ResolveActionConflict

# Migrations that are always run
always_migrations = [ResolveActionConflict()]

# Migrations that are only run on specific version transitions
version_migrations = {
    Version("v0.4"): ...
}


def main():
    logging.basicConfig(level=logging.INFO)

    from_version = Version(os.environ["VERSION_PEP440_FROM"])
    to_version = Version(os.environ["VERSION_PEP440_TO"])
    stage = os.environ["STAGE"]

    logging.info(
        f"Running migrations from {from_version} to {to_version} in {stage} stage"
    )

    # Select migrations that are applicable to the current version transition
    applicable_migrations = always_migrations
    for version, migrations in version_migrations.items():
        if from_version < version <= to_version:
            applicable_migrations.extend(migrations)

    # Execute migrations if they implement the current stage
    for migration in applicable_migrations:
        if hasattr(migration, stage):
            print(f"Running '{migration.name}' ({stage})")
            getattr(migration, stage)()


if __name__ == "__main__":
    main()

# migrations/conflict_resolution.py

import re
from dataclasses import dataclass
from pathlib import Path

from .migration import Migration

ACTION_NAME_REGEX = r"[a-zA-Z0-9_-]+/[a-zA-Z0-9_-]+)"
ACTION_HASH_REGEX = r"[a-z0-9]{40}"
ACTION_VERSION_REGEX = r"v?([0-9]+[\.0-9]*[0-9])+"
ACTION_LINE_REGEX = (
    rf" *uses: *({ACTION_NAME_REGEX}@{ACTION_HASH_REGEX} *# *{ACTION_VERSION_REGEX} *"  # noqa: E501
)


class ResolveActionConflict(
    Migration, name="Resolve conflicts in GitHub action versions if possible"
):
    def after(self):
        files = (Path(".github") / "workflows").glob("*.yml")
        for file in files:
            _resolve_file(file)


def _resolve_file(file: Path) -> bool:
    """Resolve merge conflicts on a file and write updated version."""
    lines = file.read_text().splitlines()
    new_lines = _resolve_all_blocks(lines)

    # Only write if we changed something
    if changed := lines != new_lines:
        file.write_text("\n".join(new_lines) + "\n")
    return changed


def _resolve_all_blocks(lines: list[str]):
    """Resolve merge conflicts on blocks found within the given lines."""

    # When we find a line that looks like the start of the merge
    # conflict, we submit the 5-line block starting on this line
    # to the transformation function.
    new_lines = []

    row = 0
    while row < len(lines):
        line = lines[row]

        # Found a conflict block. Replace it with resolved lines
        if line == "<<<<<<< before updating":
            if lines[row + 4] == ">>>>>>> after updating":
                new_lines.extend(_resolve_block(lines[row : row + 5]))
                row += 5
                continue
            elif lines[row + 6] == ">>>>>>> after updating":
                new_lines.extend(_resolve_block(lines[row : row + 7]))
                row += 7
                continue

        # Otherwise, we just keep the line
        new_lines.append(line)
        row += 1

    return new_lines


def _resolve_block(block: list[str]) -> list[str]:
    """Resolve one merge conflict block.

    If the block does not look like it's a relevant conflict, or we
    don't know how to resolve it, we just return the input as-is
    """
    if len(block) == 5:
        # Block must have the right structure with metadata lines
        if not (
            block[0] == "<<<<<<< before updating"
            and block[2] == "======="
            and block[4] == ">>>>>>> after updating"
        ):
            return block

        first_action_line = block[1]
        second_action_line = block[3]

        info1 = _parse_uses_line(first_action_line)

        info2 = _parse_uses_line(second_action_line)

        if info1 is None or info2 is None:
            return block

        # Sanity check: Action must match
        if info1.action != info2.action:
            return block

        if info1.version > info2.version:
            return [first_action_line]
        if info1.version < info2.version:
            return [second_action_line]
    elif len(block) == 7:
        # Block must have the right structure with metadata lines
        if not (
            block[0] == "<<<<<<< before updating"
            and block[2] == "||||||| last update"
            and block[4] == "======="
            and block[6] == ">>>>>>> after updating"
        ):
            return block
        first_action_line = block[1]
        second_action_line = block[3]
        third_action_line = block[5]

        info1 = _parse_uses_line(first_action_line)
        info2 = _parse_uses_line(second_action_line)
        info3 = _parse_uses_line(third_action_line)

        if info1 is None or info2 is None or info3 is None:
            return block

        # Sanity check: Action must match
        if info1.action != info2.action or info1.action != info3.action:
            return block

        if info1.version >= info2.version and info1.version >= info3.version:
            return [first_action_line]
        if info2.version >= info1.version and info2.version >= info3.version:
            return [second_action_line]
        if info3.version >= info1.version and info3.version >= info2.version:
            return [third_action_line]
    return block


@dataclass
class ActionVersion:
    action: str
    version: tuple[int, ...]


def _parse_uses_line(line) -> ActionVersion | None:
    """Determines the action name and version from a 'uses:' line found in
    GitHub Action workflow files."""
    m = re.match(ACTION_LINE_REGEX, line)
    if not m:
        return None

    groups = m.groups()
    version_tuple = tuple(int(part) for part in groups[1].split("."))
    return ActionVersion(action=groups[0], version=version_tuple)

This basically trivially resolves github actions merge conflicts by choosing the newest one, only if it's of the form uses: ...@<sha> # <version>.

I think this might be useful for other users as well. @sisp what do you think would be a good place for this? We could also refine this or make a custom python package out of this that people could install.

[pawamoy]

Nice, thanks for sharing!

I could see this being an installable package, yes. This could be packaged as a Jinja extension, to be loaded thanks to copier-template-extensions.

_jinja_extensions:
- copier_template_extensions.TemplateExtensionLoader
- copier_resolve_conflicts.ResolveConflicts

The Jinja extension would just add a variable to the context, so that users can use it in _migrations.

_migrations:
  - command: >
    {{ resolve_conflicts }} --regex '^\s+uses: [\w_-]+/[\w_-]+@[a-zA-Z0-9]{40} # v(?P<version>.+)$'

...where {{ resolve_conflicts }} renders to {{ _copier_python }} -m your_module_name or something like this.

This executable would be a Python script or runnable module, which would accept args like a regex to match lines and find a PEP 440 version within them, used to compare lines in conflicts and keep only the ones with latest versions.

Or maybe there's an even simpler approach!