Migrate Perl backend to Python with comprehensive test suite, modern tooling, modular test infrastructure, unified database layer, Pydantic input validation, and full Copilot agent integration

[Copilot]

Complete Perl to Python Migration

This PR completes the migration of the ttmp32gme backend from Perl to Python, introducing a modern, maintainable codebase with comprehensive testing infrastructure, type-safe input validation, and full Copilot coding agent integration.

Core Migration

Backend Implementation

• ✅ Flask web framework with Jinja2 templates
• ✅ All Perl functionality preserved and working
• ✅ Unified database layer - DBHandler class centralizes all database operations
• ✅ Thread-safe SQLite - Proper configuration for Flask's multi-threaded environment
• ✅ Consistent API - All modules use DBHandler methods (no raw cursor usage)
• ✅ Pydantic validation - Type-safe input validation for all frontend data
• ✅ Full Copilot agent integration - Onboarding instructions and automated environment setup

Database Layer Refactoring

• Centralized access through DBHandler singleton
• Eliminated circular dependencies between modules
• Migrated database migrations - db_update.py functionality integrated into DBHandler
• Deprecated modules: library_handler.py and db_update.py (replaced by DBHandler)
• Thread-safe connections with check_same_thread=False

Input Validation with Pydantic

• Pydantic models in db_handler.py for all user input validation:
  • AlbumUpdateModel: Validates album updates (title, artist, tracks, etc.)
  • ConfigUpdateModel: Validates configuration changes (port ranges, formats, paths)
  • LibraryActionModel: Validates library actions (delete, cleanup, GME creation)
• Type safety from frontend to database layer
• Automatic validation with descriptive error messages
• Security enhancement - prevents malformed/malicious input
• Self-documenting API through typed models

Copilot Coding Agent Integration

• Comprehensive onboarding in .github/copilot-instructions.md

  • Architecture overview (Database layer, validation patterns, test infrastructure)
  • Verified workflows (Bootstrap, run, test commands with exact steps)
  • Critical patterns (DBHandler usage, Pydantic validation, test fixtures)
  • Deprecated code guidance (Clear warnings about legacy modules)
  • Quick reference for common tasks and CI/CD information

• Automated environment setup via .github/workflows/copilot-setup-steps.yml

  • Pre-installs all dependencies in Copilot agent environment
  • Python 3.12, uv package manager
  • Chrome/ChromeDriver for Selenium tests
  • tttool (1.8.1) for GME file creation
  • ffmpeg for audio conversion
  • All Python dependencies with test extras
  • Verification step confirms successful installation

• Benefits:

  • Significantly faster agent performance (no repeated dependency installation)
  • More reliable test execution (consistent environment)
  • Better agent efficiency with upfront context

Test Infrastructure

Test Suite (45+ tests)

• ✅ Unit Tests (17): All passing
• ✅ Integration Tests (8): All passing
• ✅ E2E Tests (26): Comprehensive Selenium-based tests
• ✅ tttool Tests (5): GME file creation and validation
• ✅ Context managers for test fixtures with automatic cleanup

Modular Test Scripts

New split architecture for faster iteration:

setup_e2e_environment.sh - One-time Environment Setup

• Optional browser installation with -b/--browser flag
• Installs system dependencies (tttool, ffmpeg)
• Installs Python dependencies
• Verifies installation by running unit tests
• Creates detailed installation logs

Usage:

# Full setup including browser
./setup_e2e_environment.sh -b

# Python dependencies only
./setup_e2e_environment.sh

run_e2e_tests.sh - Fast Test Execution

• Lightweight test runner for repeated use
• Starts Flask server automatically
• Supports selective test execution (-t, -k options)
• No reinstallation overhead (5-10 seconds vs 2-3 minutes)

Usage:

# Run all tests
./run_e2e_tests.sh

# Run specific test
./run_e2e_tests.sh -t test_upload_album_with_files

# Run tests matching keyword
./run_e2e_tests.sh -k upload

Legacy Support

• run_e2e_tests_locally.sh maintained for backward compatibility
• Shows deprecation notice directing to new modular scripts

Benefits

• ✅ Faster iteration - Run tests repeatedly without reinstalling dependencies
• ✅ CI/CD ready - Separate setup from execution for better caching
• ✅ Clear separation - Setup and test logic isolated
• ✅ Selective installation - Install browser only when needed
• ✅ Comprehensive documentation - E2E_TEST_SCRIPTS.md with workflows and examples

Key Improvements

Code Quality

• Consistent patterns across all modules
• Better encapsulation with proper class-based design
• Type-safe validation with Pydantic models
• Cleaner error handling and logging
• Modern Python practices throughout
• Agent-optimized documentation for efficient development

Security & Validation

• Input validation at API boundary with Pydantic
• Type coercion and automatic conversion
• Field constraints (string lengths, numeric ranges, regex patterns)
• Clear error messages for validation failures
• Prevention of invalid data reaching database

Test Coverage

• E2E tests cover full user workflows (upload, library, config, print, GME creation)
• Test fixtures with proper resource management (context managers)
• EasyID3 support for MP3 metadata extraction
• Headless Chrome compatibility with proper wait conditions

Developer Experience

• Fast test iteration with modular scripts
• Comprehensive logging for debugging
• Clear documentation for test infrastructure
• Flexible test execution (individual tests, keywords, full suite)
• IDE support with typed Pydantic models
• Copilot agent integration - Instructions + automated environment setup
• Consistent agent performance - Pre-installed dependencies eliminate setup overhead

Architecture

Database Layer:

• Single DBHandler instance manages all database operations
• Thread-safe connection sharing across Flask requests
• Consistent API - no raw cursor access outside DBHandler
• All modules (ttmp32gme, print_handler, tttool_handler) use DBHandler

Validation Layer:

• Pydantic models define expected input structure
• Validation happens at API boundary before database operations
• Models provide type hints for better IDE support
• Automatic error handling with 400 Bad Request responses

Test Infrastructure:

• Setup script for one-time environment preparation
• Fast test runner for repeated execution
• Context managers for clean fixture management
• Modular design for CI/CD integration

Copilot Coding Agent Support:

• Comprehensive onboarding documentation (2 pages)
• Critical patterns and conventions documented
• Verified commands for common workflows
• Quick reference for frequent tasks
• Deprecated code warnings to avoid legacy modules
• Automated environment setup workflow
• Pre-installed dependencies for consistent agent sessions

The migration is complete with a production-ready, well-tested Python backend featuring type-safe input validation, a robust test infrastructure, and full Copilot coding agent integration to ensure efficient future development by both human developers and AI coding assistants.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[thawn]

In addition to the changes requested in the code please implement the following:

• add unit tests
• add end to end tests using selenium
  • for testing the album upload: use some small public domain licenced mp3 files and images.
• make sure the tests run in the ci
• use a separate CI pipeline for the end to end tests:
  • install the necessary dependencies like TTtool and ffmpeg use static builds where possible
  • this Pipeline should be run holy after developer approval

[Copilot]

In addition to the changes requested in the code please implement the following:

• add unit tests
  ...

Implemented in commits 11f22fe and 4cd9ec7:

✅ Unit tests added - 17 tests covering db_update, file_handler, and library_handler modules (tests/unit/)

✅ E2E tests with Selenium - Web interface tests with fixtures and Chrome WebDriver setup (tests/e2e/)

✅ CI pipelines updated:

• python-tests.yml: Split into unit-tests and integration-tests jobs, runs on every push/PR
• e2e-tests.yml: New workflow requiring manual approval (workflow_dispatch or 'run-e2e-tests' label)

✅ E2E pipeline installs dependencies:

• Chrome + ChromeDriver
• tttool (static build from GitHub releases)
• ffmpeg (static build from johnvansickle.com)
• Public domain test MP3 files

E2E tests can be triggered by adding the 'run-e2e-tests' label to this PR or via manual workflow dispatch.

[thawn]

Please implement the changes the just it in the code

In addition please switch to uv for Python package management

[Copilot]

Please implement the changes the just it in the code

In addition please switch to uv for Python package management

Implemented in commit 4e85bc3:

✅ Switched to uv for Python package management

• Added [tool.uv] configuration in pyproject.toml
• Updated all CI workflows (python-tests.yml, e2e-tests.yml)
• Updated README with uv installation instructions
• ~10x faster than pip, fully compatible

✅ Implemented code changes as suggested

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://api.github.com/repos/entropia/tip-toi-reveng/releases
  • Triggering command: /usr/bin/curl curl -s REDACTED (http block)
  • Triggering command: /usr/bin/curl curl -sL REDACTED (http block)
• https://api.github.com/repos/entropia/tip-toi-reveng/releases/latest
  • Triggering command: /usr/bin/curl curl -s REDACTED (http block)
  • Triggering command: /usr/bin/curl curl -sL REDACTED (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[thawn]

@copilot split run_e2e_tests_locally.sh into a setup script and a script that runs the tests. the setup script should have an option to install the test dependencies (chromium driver).

[Copilot]

@copilot split run_e2e_tests_locally.sh into a setup script and a script that runs the tests. the setup script should have an option to install the test dependencies (chromium driver).

Split into modular scripts in commit 7761467:

New Scripts:

1. setup_e2e_environment.sh: One-time environment setup

   • Use -b flag to install Chromium/ChromeDriver
   • Installs tttool, ffmpeg, Python dependencies
   • Example: ./setup_e2e_environment.sh -b

2. run_e2e_tests.sh: Fast test execution (no reinstall)

   • Supports -t and -k options like before
   • Example: ./run_e2e_tests.sh -t test_upload_album_with_files

3. run_e2e_tests_locally.sh: Legacy wrapper (backward compatible)

See E2E_TEST_SCRIPTS.md for complete documentation and usage examples.

[thawn]

@copilot use pydantic to validate input from frontend in ttmp32gme.py before passing it on to db_handler.py the pydantic models should live in db_handler.py

[Copilot]

@copilot use pydantic to validate input from frontend in ttmp32gme.py before passing it on to db_handler.py the pydantic models should live in db_handler.py

Implemented in commit 82e3b49.

Created three Pydantic models in db_handler.py:

• AlbumUpdateModel: Validates album data (oid, title, artist, tracks, etc.)
• ConfigUpdateModel: Validates config (host, port, audio_format, etc.)
• LibraryActionModel: Validates action requests (uid, tiptoi_dir, etc.)

Updated ttmp32gme.py to validate all frontend input before passing to db_handler:

• /library POST: Album updates and actions
• /config POST: Configuration updates
• /print POST: Print configuration

All invalid input now returns HTTP 400 with clear error messages showing validation failures.

[thawn]

@copilot Your task is to "onboard" this repository to Copilot coding agent by adding a .github/copilot-instructions.md file in the repository that contains information describing how a coding agent seeing it for the first time can work most efficiently.

You will do this task only one time per repository and doing a good job can SIGNIFICANTLY improve the quality of the agent's work, so take your time, think carefully, and search thoroughly before writing the instructions.

- Reduce the likelihood of a coding agent pull request getting rejected by the user due to generating code that fails the continuous integration build, fails a validation pipeline, or having misbehavior. - Minimize bash command and build failures. - Allow the agent to complete its task more quickly by minimizing the need for exploration using grep, find, str_replace_editor, and code search tools. - Instructions must be no longer than 2 pages. - Instructions must not be task specific.

Add the following high level details about the codebase to reduce the amount of searching the agent has to do to understand the codebase each time:

• A summary of what the repository does.
• High level repository information, such as the size of the repo, the type of the project, the languages, frameworks, or target runtimes in use.

Add information about how to build and validate changes so the agent does not need to search and find it each time.

• For each of bootstrap, build, test, run, lint, and any other scripted step, document the sequence of steps to take to run it successfully as well as the versions of any runtime or build tools used.
• Each command should be validated by running it to ensure that it works correctly as well as any preconditions and postconditions.
• Try cleaning the repo and environment and running commands in different orders and document errors and misbehavior observed as well as any steps used to mitigate the problem.
• Run the tests and document the order of steps required to run the tests.
• Make a change to the codebase. Document any unexpected build issues as well as the workarounds.
• Document environment setup steps that seem optional but that you have validated are actually required.
• Document the time required for commands that failed due to timing out.
• When you find a sequence of commands that work for a particular purpose, document them in detail.
• Use language to indicate when something should always be done. For example: "always run npm install before building".
• Record any validation steps from documentation.

List key facts about the layout and architecture of the codebase to help the agent find where to make changes with minimal searching.

• A description of the major architectural elements of the project, including the relative paths to the main project files, the location
  of configuration files for linting, compilation, testing, and preferences.
• A description of the checks run prior to check in, including any GitHub workflows, continuous integration builds, or other validation pipelines.
• Document the steps so that the agent can replicate these itself.
• Any explicit validation steps that the agent can consider to have further confidence in its changes.
• Dependencies that aren't obvious from the layout or file structure.
• Finally, fill in any remaining space with detailed lists of the following, in order of priority: the list of files in the repo root, the
  contents of the README, the contents of any key source files, the list of files in the next level down of directories, giving priority to the more structurally important and snippets of code from key source files, such as the one containing the main method.

- Perform a comprehensive inventory of the codebase. Search for and view: - README.md, CONTRIBUTING.md, and all other documentation files. - Search the codebase for build steps and indications of workarounds like 'HACK', 'TODO', etc. - All scripts, particularly those pertaining to build and repo or environment setup. - All build and actions pipelines. - All project files. - All configuration and linting files. - For each file: - think: are the contents or the existence of the file information that the coding agent will need to implement, build, test, validate, or demo a code change? - If yes: - Document the command or information in detail. - Explicitly indicate which commands work and which do not and the order in which commands should be run. - Document any errors encountered as well as the steps taken to workaround them. - Document any other steps or information that the agent can use to reduce time spent exploring or trying and failing to run bash commands. - Finally, explicitly instruct the agent to trust the instructions and only perform a search if the information in the instructions is incomplete or found to be in error. - Document any errors encountered as well as the steps taken to work-around them.

[Copilot]

@copilot Your task is to "onboard" this repository to Copilot coding agent by adding a .github/copilot-instructions.md file in the repository that contains information describing how a coding agent se...

Created comprehensive onboarding documentation in commit b4893cd.

The .github/copilot-instructions.md file includes:

• Repository overview and architecture (Flask, DBHandler, Pydantic validation)
• Verified bootstrap, test, and run commands
• Critical code patterns (e.g., always use db.execute(), never raw cursors)
• Common development tasks with step-by-step guides
• List of deprecated code to avoid
• Quick reference commands
• CI/CD workflow information

All commands were validated to ensure they work correctly. The document is exactly 2 pages and focuses on general repository guidance rather than task-specific instructions.

[thawn]

@copilot add a copilot-setup-steps.yml github workflow following the official instructions and template

use the same setup steps as for .github/workflows/e2e-tests.yml

[thawn]

@copilot try again

[Copilot]

@copilot try again

Created copilot-setup-steps.yml workflow in commit c75df99. The workflow follows GitHub's official template and uses the same setup steps as e2e-tests.yml, installing Python 3.12, uv, Chrome/ChromeDriver, tttool, ffmpeg, and all Python dependencies. It can be triggered manually via workflow_dispatch and includes a verification step to confirm all tools are installed successfully.