Skip to content

Commit 6507efd

Browse files
thawnthawn
authored
Merge pull request #74 from thawn/copilot/migrate-perl-backend-to-python
Migrate Perl backend to Python with comprehensive test suite, modern tooling, modular test infrastructure, unified database layer, Pydantic input validation, and full Copilot agent integration
2 parents 64e2caf + df4c09b commit 6507efd

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

47 files changed

+6006
-2411
lines changed

‎.github/copilot-instructions.md‎

Lines changed: 215 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,215 @@
1+
# Copilot Coding Agent Instructions for ttmp32gme
2+
3+
## Repository Overview
4+
5+
**ttmp32gme** is a cross-platform tool that converts MP3/audio files into TipToi GME (Game Music Engine) files playable on the TipToi pen. It generates printable control sheets with OID codes for music/audiobook playback control.
6+
7+
### Key Stats
8+
- **Languages**: Python (backend), JavaScript (frontend), Shell scripts (testing)
9+
- **Size**: ~500 files, primarily Python source in `src/ttmp32gme/`
10+
- **Framework**: Flask 3.0+ web application with Jinja2 templates
11+
- **Database**: SQLite with custom DBHandler class
12+
- **Python Version**: 3.11+ required
13+
- **Testing**: 45+ tests (unit, integration, E2E with Selenium)
14+
15+
### Architecture
16+
- **Backend**: Python Flask application migrated from Perl
17+
- **Database Layer**: Unified DBHandler class (`src/ttmp32gme/db_handler.py`)
18+
- All database operations go through DBHandler singleton
19+
- Thread-safe SQLite with `check_same_thread=False`
20+
- Never use raw cursors outside DBHandler - always use `db.execute()`, `db.fetchone()`, etc.
21+
- **Validation**: Pydantic models in `db_handler.py` validate all frontend input
22+
- **Dependencies**: tttool (external binary), ffmpeg (optional for OGG)
23+
24+
## Development Workflow
25+
26+
### Bootstrap & Setup
27+
28+
```bash
29+
# Clone repository
30+
git clone https://github.com/thawn/ttmp32gme.git && cd ttmp32gme
31+
32+
# Install Python dependencies (recommended: use uv)
33+
uv pip install -e ".[test]"
34+
# OR
35+
pip install -e ".[test]"
36+
37+
# Install external dependencies
38+
# - tttool: https://github.com/entropia/tip-toi-reveng#installation
39+
# - ffmpeg: sudo apt-get install ffmpeg (Ubuntu) or brew install ffmpeg (macOS)
40+
```
41+
42+
**Verification**: `python -m ttmp32gme.ttmp32gme --help` should show usage info.
43+
44+
### Running the Application
45+
46+
```bash
47+
# Start server (default: localhost:10020)
48+
python -m ttmp32gme.ttmp32gme
49+
50+
# Or use entry point
51+
ttmp32gme
52+
53+
# Custom host/port
54+
ttmp32gme --host 0.0.0.0 --port 8080
55+
56+
# Access UI at http://localhost:10020
57+
```
58+
59+
**Verification**: `curl http://localhost:10020/` should return HTML.
60+
61+
### Testing
62+
63+
#### Unit Tests (Fast, no dependencies)
64+
```bash
65+
# Run all unit tests
66+
pytest tests/unit/ -v
67+
68+
# Run specific test file
69+
pytest tests/unit/test_library_handler.py -v
70+
```
71+
72+
#### Integration Tests (Requires running server)
73+
```bash
74+
# Terminal 1: Start server
75+
python -m ttmp32gme.ttmp32gme
76+
77+
# Terminal 2: Run integration tests
78+
pytest tests/test_web_frontend.py -v
79+
```
80+
81+
#### E2E Tests (Selenium, requires full setup)
82+
```bash
83+
# One-time setup (installs Chrome, tttool, etc.)
84+
./ttmp32gme > /tmp/server.log 2>&1 & sleep(2) # Start server in background
85+
86+
# Run all E2E tests
87+
./pytest tests/e2e/ -v
88+
89+
# Run specific test
90+
pytest tests/e2e/test_upload_album_with_files.py -v
91+
92+
# Run tests matching keyword
93+
pytest -k upload -v
94+
```
95+
96+
**E2E Test Markers**:
97+
- Skip E2E tests: `pytest -m "not e2e"`
98+
- Skip slow tests: `pytest -m "not slow"`
99+
100+
### Building & Linting
101+
102+
**No formal linting configured** - follow existing code style:
103+
- 4-space indentation
104+
- Type hints encouraged (especially with Pydantic)
105+
- Descriptive variable names
106+
107+
**No build step required** - Python source runs directly.
108+
109+
## Code Patterns & Conventions
110+
111+
### Database Access (CRITICAL)
112+
❌ **NEVER do this**:
113+
```python
114+
cursor = db.cursor()
115+
cursor.execute("SELECT ...")
116+
```
117+
118+
✅ **ALWAYS do this**:
119+
```python
120+
result = db.fetchone("SELECT ...") # or db.fetchall()
121+
```
122+
All database operations MUST go through DBHandler methods (except in unit tests).
123+
124+
### Input Validation
125+
All frontend input MUST be validated with Pydantic models in `db_handler.py`:
126+
```python
127+
from pydantic import ValidationError
128+
129+
try:
130+
validated = AlbumUpdateModel(**data)
131+
db.update_album(validated.model_dump(exclude_none=True))
132+
except ValidationError as e:
133+
return jsonify({"success": False, "error": str(e)}), 400
134+
```
135+
136+
### Test Fixtures
137+
Use context managers for test file management:
138+
```python
139+
with test_audio_files_context() as test_files:
140+
# Files created automatically
141+
driver.find_element(...).send_keys(test_files[0])
142+
# Files cleaned up automatically on exit
143+
```
144+
145+
### Threading
146+
SQLite connection uses `check_same_thread=False` for Flask's multi-threaded environment. DBHandler is safe to use across request threads.
147+
148+
## Common Tasks
149+
150+
### Adding a New Database Operation
151+
1. Add method to `DBHandler` class in `src/ttmp32gme/db_handler.py`
152+
2. Use `self.execute()`, `self.fetchone()`, `self.commit()` internally
153+
3. Call from Flask route: `db.new_method()`
154+
155+
### Adding Input Validation
156+
1. Create/extend Pydantic model in `db_handler.py`
157+
2. Add field constraints (`Field()`, regex patterns, value ranges)
158+
3. Validate in Flask route before calling DBHandler
159+
160+
### Adding a New Route
161+
1. Add route in `src/ttmp32gme/ttmp32gme.py`
162+
2. Validate input with Pydantic
163+
3. Use `db` (DBHandler instance) for database operations
164+
4. Return JSON for AJAX or render template for pages
165+
166+
### Fixing E2E Test Issues
167+
1. Before re-running specific test, start the server manually:
168+
```bash
169+
./ttmp32gme > /tmp/server.log 2>&1 & sleep(2) # Start server in background
170+
```
171+
2. Check server logs in `/tmp/server.log` for errors
172+
3. Add debug statements to test for element visibility
173+
4. Use explicit waits: `WebDriverWait(driver, 5).until(...)`
174+
175+
## File Locations
176+
177+
- **Source**: `src/ttmp32gme/` (main application)
178+
- **Templates**: `resources/` (Jinja2 HTML templates)
179+
- **Tests**: `tests/unit/`, `tests/e2e/`, `tests/test_web_frontend.py`
180+
- **Static files**: `resources/assets/` (CSS, JS, images)
181+
- **Config**: `pyproject.toml` (dependencies, pytest config)
182+
183+
## Quick Reference Commands
184+
185+
```bash
186+
# Install dependencies
187+
uv pip install -e ".[test]"
188+
189+
# Run app
190+
python -m ttmp32gme.ttmp32gme
191+
192+
# Run all tests
193+
pytest tests/ -v
194+
195+
# Run unit tests only
196+
pytest tests/unit/ -v
197+
198+
# Setup E2E environment
199+
./setup_e2e_environment.sh -b
200+
201+
# Run E2E tests
202+
./run_e2e_tests.sh
203+
204+
# Run specific E2E test
205+
./run_e2e_tests.sh -t test_upload_album_with_files
206+
```
207+
208+
## CI/CD
209+
210+
GitHub Actions workflows run automatically on PRs:
211+
- **python-tests.yml**: Unit and integration tests (Python 3.11, 3.12, 3.13)
212+
- **javascript-tests.yml**: Frontend Jest tests (Node 18.x, 20.x)
213+
- **e2e-tests.yml**: Full E2E suite (manual trigger or workflow_dispatch)
214+
215+
Tests must pass before merging.

‎.github/workflows/copilot-setup-steps.yml‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
name: Copilot Setup Steps
2+
3+
on:
4+
workflow_dispatch:
5+
push:
6+
paths:
7+
- .github/workflows/copilot-setup-steps.yml
8+
pull_request:
9+
paths:
10+
- .github/workflows/copilot-setup-steps.yml
11+
12+
jobs:
13+
copilot-setup-steps:
14+
runs-on: ubuntu-latest
15+
16+
permissions:
17+
contents: read
18+
19+
steps:
20+
- name: Checkout repository
21+
uses: actions/checkout@v4
22+
# fetch-depth will be overridden
23+
24+
- name: Set up Python
25+
uses: actions/setup-python@v5
26+
with:
27+
python-version: '3.12'
28+
29+
- name: Install uv
30+
uses: astral-sh/setup-uv@v4
31+
with:
32+
version: "latest"
33+
34+
- name: Install system dependencies
35+
run: |
36+
sudo apt-get update
37+
sudo apt-get install -y wget unzip
38+
39+
- name: Install Chrome and ChromeDriver
40+
run: |
41+
# Install Chrome
42+
wget -q https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
43+
sudo dpkg -i google-chrome-stable_current_amd64.deb || sudo apt-get -f install -y
44+
45+
# Install ChromeDriver using Chrome for Testing
46+
CHROME_VERSION=$(google-chrome --version | awk '{print $3}' | cut -d '.' -f 1)
47+
CHROMEDRIVER_URL=$(curl -s "https://googlechromelabs.github.io/chrome-for-testing/last-known-good-versions-with-downloads.json" | \
48+
grep -o '"chromedriver".*"linux64".*"url":"[^"]*' | \
49+
grep -o 'https://[^"]*' | head -1)
50+
wget -q "$CHROMEDRIVER_URL" -O chromedriver-linux64.zip
51+
unzip chromedriver-linux64.zip
52+
sudo mv chromedriver-linux64/chromedriver /usr/local/bin/
53+
sudo chmod +x /usr/local/bin/chromedriver
54+
rm -rf chromedriver-linux64*
55+
56+
- name: Install tttool (from releases)
57+
run: |
58+
# Download a specific working version from GitHub releases
59+
# Using the correct URL pattern: tttool-{version}.zip
60+
TTTOOL_VERSION="1.8.1"
61+
echo "Installing tttool version $TTTOOL_VERSION"
62+
63+
# Use system temp directory for extraction
64+
TEMP_DIR=$(mktemp -d)
65+
cd "$TEMP_DIR"
66+
67+
wget -q "https://github.com/entropia/tip-toi-reveng/releases/download/${TTTOOL_VERSION}/tttool-${TTTOOL_VERSION}.zip"
68+
69+
# Extract into temp directory
70+
unzip -q "tttool-${TTTOOL_VERSION}.zip"
71+
72+
# Move the binary to /usr/local/bin
73+
chmod +x tttool
74+
sudo mv tttool /usr/local/bin/
75+
76+
# Cleanup - change permissions recursively then remove
77+
cd "$GITHUB_WORKSPACE"
78+
chmod -R u+w "$TEMP_DIR" || true
79+
rm -rf "$TEMP_DIR"
80+
81+
# Verify installation
82+
tttool --help || { echo "Error: tttool installation failed"; exit 1; }
83+
84+
- name: Install ffmpeg (static build)
85+
run: |
86+
wget -q https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz
87+
tar -xf ffmpeg-release-amd64-static.tar.xz
88+
sudo mv ffmpeg-*-amd64-static/ffmpeg /usr/local/bin/
89+
sudo mv ffmpeg-*-amd64-static/ffprobe /usr/local/bin/
90+
rm -rf ffmpeg-*-amd64-static*
91+
92+
- name: Install Python dependencies
93+
run: |
94+
uv pip install --system -e ".[test]"
95+
96+
- name: Prepare test fixtures
97+
run: |
98+
# Bundled test audio file already exists in repo
99+
ls -lh tests/fixtures/test_audio.mp3
100+
101+
# Create test cover image if needed
102+
if [ ! -f tests/fixtures/test_cover.jpg ]; then
103+
python3 -c "from PIL import Image; img = Image.new('RGB', (100, 100), color='green'); img.save('tests/fixtures/test_cover.jpg')"
104+
fi
105+
106+
# Verify files exist
107+
ls -lh tests/fixtures/
108+
109+
- name: Verify setup
110+
run: |
111+
echo "Setup complete. Verifying installations..."
112+
python --version
113+
google-chrome --version
114+
chromedriver --version
115+
tttool --help | head -5
116+
ffmpeg -version | head -5
117+
uv --version
118+
echo "All tools installed successfully!"

0 commit comments

Comments
 (0)