Add Python CICD fixes and rollback handling to server.py #13

Tomajari · 2026-01-02T18:44:02Z

Add a CI/CD check function to server.py
Steps on how to test:

Run the server using "python3 server.py"
Ensure that config.yml includes your repo, branch, and path.
My example:
repos:

name: sce-cicd
branch: main
path: /Users/Brian/sce-cicd
force_recreate: false

Triggered a webhook event using a curl command
Ex:
curl command to test deployment successful:
curl -X POST
-H "Content-Type: application/json"
-H "X-GitHub-Event: push"
--data '{
"ref": "refs/heads/main",
"repository": {"name": "sce-cicd"},
"head_commit": {
"id": "abc123def456",
"message": "Deployment successful test commit",
"branch": "main",
"url": "abc123def456",
"author": {"name": "Test User", "username": "testuser", "url": "https://github.com/testuser"}
}
}'
http://127.0.0.1:3000/webhook
curl command to test rollback:
curl -X POST
-H "Content-Type: application/json"
-H "X-GitHub-Event: push"
--data '{
"ref": "refs/heads/main",
"repository": {"name": "sce-cicd"},
"head_commit": {
"id": "abc123def456",
"message": "Test commit",
"branch": "main",
"url": "abc123def456",
"author": {"name": "Test User", "username": "testuser", "url": "https://github.com/testuser"}
}
}'
http://127.0.0.1:3000/webhook
Terminal Output:
You should see logs indicating that a push was detected. If docker-compose fails a rollback log message will appear in the terminal like this:
" ROLLBACK INITIATED
docker-compose failed (exit code ...), git pull exit code .... Rollback would be performed here (mocked in dev).
Rollback skipped for local development. No destructive git reset performed.
ROLLBACK COMPLETE"
If your discord webhook is set up, you will see a deployment status embed in your Discord channel. To create a webhook to test the embed, follow this tutorial: https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks

Output:
The first screenshot shows when a commit is sucessful. The second screenshot shows when a commit is failed and invalid, which proceeds to run the rollback command:

Add a CI/CD check function to server.py

Add git rollback if docker-compose fails with python in server.py

Updated rollback handling to log messages without executing git reset in development in the terminal

Removed mock rollback message for local development.

Added rollback functionality with backup branch creation and deletion.

Removed unused import for datetime module.

evanugarte · 2026-01-04T07:02:52Z

server.py

    name: str
    branch: str
    path: str
+    force_recreate: bool = False


can we remove this for now

evanugarte · 2026-01-04T07:03:43Z

server.py

+            if 'enable_rollback' not in config:
+                config['enable_rollback'] = False


do we have to do this, would enable_rollback just default to false if we dont pass anything in to the dataclass

evanugarte · 2026-01-04T07:04:35Z

server.py

+    backup_branch_name = None
+    # Backup branch logic
+    if getattr(repo_config, 'enable_rollback', False):
+        from datetime import datetime


lets import this at the top of the file, and import it as import datetime

i know we have to then do datetime.datetime.whatever, but it makes it easier to follow as we are using the base module, and write out the full paths to each function

evanugarte · 2026-01-04T07:05:44Z

server.py

+        backup_branch_name = datetime.now().strftime('%Y%m%d-%H%M%S') + f'-{repo_config.branch}'
+        try:
+            # double check if on the main branch
+            subprocess.run(["git", "checkout", repo_config.branch], cwd=repo_config.path, check=True)
+            # create backup branch
+            subprocess.run(["git", "checkout", "-b", backup_branch_name], cwd=repo_config.path, check=True)
+            # switch back to main
+            subprocess.run(["git", "checkout", repo_config.branch], cwd=repo_config.path, check=True)
+            logger.info(f"Created backup branch {backup_branch_name} for rollback protection.")
+        except Exception as e:
+            logger.error(f"Failed to create backup branch {backup_branch_name}: {e}")
+            result.rollback_message = f"Failed to create backup branch: {e}"
+        result.backup_branch_name = backup_branch_name


could this be its own function, like create_copy_of_cicd_branch and we pass in repo_config as a param

the function could then return the name of the branch we created as the copy

evanugarte · 2026-01-04T07:07:39Z

server.py

+        if docker_result.returncode != 0:
+            logger.warning("\nROLLBACK INITIATED\n")
+            logger.warning(f"docker-compose failed (exit code {docker_result.returncode}), git pull exit code {git_result.returncode}.")
+            rollback_success = False
+            rollback_error = None
+            if getattr(repo_config, 'enable_rollback', False) and hasattr(result, 'backup_branch_name') and result.backup_branch_name:
+                try:
+                    #reset main to backup branch
+                    subprocess.run(["git", "checkout", repo_config.branch], cwd=repo_config.path, check=True)
+                    subprocess.run(["git", "reset", "--hard", result.backup_branch_name], cwd=repo_config.path, check=True)
+                    #delete backup branch
+                    subprocess.run(["git", "branch", "-D", result.backup_branch_name], cwd=repo_config.path, check=True)
+                    logger.info(f"Rolled back {repo_config.branch} to {result.backup_branch_name} and deleted backup branch.")
+                    rollback_success = True
+                except Exception as e:
+                    logger.error(f"Rollback failed: {e}")
+                    rollback_error = str(e)
+            else:
+                logger.warning("Rollback skipped: no backup branch available.")
+            logger.warning("\nROLLBACK COMPLETE\n")
+            result.git_exit_code = 1
+            result.docker_exit_code = 1
+            if rollback_success:
+                result.rollback_message = f"Deployment failed, rollback performed using backup branch {getattr(result, 'backup_branch_name', '')}."
+                # Retry docker-compose up after rollback
+                try:
+                    retry_result = subprocess.run(
+                        ["docker-compose", "up", "--build", "-d"],
+                        cwd=repo_config.path,
+                        capture_output=True,
+                        text=True,
+                    )
+                    logger.info(f"Docker compose retry stdout: {retry_result.stdout}")
+                    logger.info(f"Docker compose retry stderr: {retry_result.stderr}")
+                    result.docker_stdout += f"\n[rollback retry]\n{retry_result.stdout}"
+                    result.docker_stderr += f"\n[rollback retry]\n{retry_result.stderr}"
+                    result.docker_exit_code = retry_result.returncode
+                except Exception as e:
+                    logger.error(f"Retry after rollback failed: {e}")
+            else:
+                result.rollback_message = f"Deployment failed, rollback logic triggered but failed: {rollback_error or 'No backup branch.'}"
+            push_update_success_as_discord_embed(repo_config, result)
+            return result
+        if getattr(repo_config, 'enable_rollback', False) and hasattr(result, 'backup_branch_name') and result.backup_branch_name:
+            try:
+                subprocess.run(["git", "branch", "-D", result.backup_branch_name], cwd=repo_config.path, check=True)
+                logger.info(f"Deleted backup branch {result.backup_branch_name} after successful deployment.")
+            except Exception as e:
+                logger.error(f"Failed to delete backup branch {result.backup_branch_name}: {e}")


lets make this its own function

and we can call it like

if docker_result.returncode != 0 and repo_config.enable_rollback: do_rollback(repo_config, copy_branch_name)

Added import datetime to the top, removed force_recreate and enable_rollback, made own functions for create_copy_of_cicd_branch and do rollback

evanugarte

lets add one more field, bool | None called maybe_rollback_result

it is default None meaning it didnt happen. but if we did a rollback, we set it to true or false, the return value of do_rollback

we should also have a section in the embed to check for this field being non-none and have some text saying a rollback happened and it worked/didnt work

server.py

evanugarte · 2026-01-04T23:50:14Z

server.py

+    except Exception as e:
+        logger.error(f"Failed to create backup branch {copy_branch_name}: {e}")
+        return None
+def do_rollback(repo_config, copy_branch_name, result):


could do_rollback instead just return True or False if it was able to rollback

we can skip passing the result param in

server.py

Co-authored-by: Evan Ugarte <[email protected]>

Added maybe_rollback_result attribute to track rollback status.

Refactor rollback logic to simplify return statements and remove unnecessary error handling.

Tomajari · 2026-01-05T01:20:44Z

lets add one more field, bool | None called maybe_rollback_result

it is default None meaning it didnt happen. but if we did a rollback, we set it to true or false, the return value of do_rollback

we should also have a section in the embed to check for this field being non-none and have some text saying a rollback happened and it worked/didnt work

fixed in @Tomajari
Add maybe_rollback_result to server.py

server.py

Co-authored-by: Evan Ugarte <[email protected]>

Tomajari added 7 commits January 2, 2026 10:43

Add Python CICD fixes and rollback handling to server.py

cb670f6

Add a CI/CD check function to server.py

Added git rollback if docker-compose fails with python

606c577

Add git rollback if docker-compose fails with python in server.py

Added a rollback command for terminal

78288bb

Updated rollback handling to log messages without executing git reset in development in the terminal

removed "mock rollback" comment

214d763

Removed rollback warning message in server.py

9589fc9

Removed mock rollback message for local development.

added "rollback feature if docker step fails

f035656

Added rollback functionality with backup branch creation and deletion.

Remove unused datetime import

2f46df1

Removed unused import for datetime module.

evanugarte reviewed Jan 4, 2026

View reviewed changes

Implement rollback functionality with backup branch

d7a267d

Added import datetime to the top, removed force_recreate and enable_rollback, made own functions for create_copy_of_cicd_branch and do rollback

evanugarte reviewed Jan 4, 2026

View reviewed changes

Tomajari and others added 4 commits January 4, 2026 16:12

changed logger.error to logger.exception

4ee6c25

Co-authored-by: Evan Ugarte <[email protected]>

fixed code to assume that copy_branch_name is set

d767205

Co-authored-by: Evan Ugarte <[email protected]>

Add maybe_rollback_result to server.py

4690508

Added maybe_rollback_result attribute to track rollback status.

Simplify rollback logic in server.py

13c3a5e

Refactor rollback logic to simplify return statements and remove unnecessary error handling.

removed the hasattr function

2aa2f00

evanugarte approved these changes Jan 6, 2026

View reviewed changes

server.py Outdated Show resolved Hide resolved

Tomajari and others added 2 commits January 5, 2026 21:24

removed "as e"

fca9ef5

Co-authored-by: Evan Ugarte <[email protected]>

Change type hint for maybe_rollback_result to Optional

5c9e1eb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Python CICD fixes and rollback handling to server.py #13

Add Python CICD fixes and rollback handling to server.py #13

Tomajari commented Jan 2, 2026 •

edited

Loading

Uh oh!

evanugarte Jan 4, 2026

Uh oh!

evanugarte Jan 4, 2026

Uh oh!

evanugarte Jan 4, 2026

Uh oh!

evanugarte Jan 4, 2026

Uh oh!

evanugarte Jan 4, 2026

Uh oh!

evanugarte left a comment

Uh oh!

Uh oh!

evanugarte Jan 4, 2026

Uh oh!

Uh oh!

Tomajari commented Jan 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		if 'enable_rollback' not in config:
		config['enable_rollback'] = False

Add Python CICD fixes and rollback handling to server.py #13

Are you sure you want to change the base?

Add Python CICD fixes and rollback handling to server.py #13

Conversation

Tomajari commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

evanugarte Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

evanugarte Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

evanugarte Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

evanugarte Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

evanugarte Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

evanugarte left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

evanugarte Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Tomajari commented Jan 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Tomajari commented Jan 2, 2026 •

edited

Loading