-
Notifications
You must be signed in to change notification settings - Fork 0
Description
Problem
The application currently uses a pre-seeded SQLite database file (storage/players-sqlite3.db) that is copied into the Docker container and mounted as a persistent volume. While this approach works for a Proof of Concept, it presents several challenges for production-ready development:
- Manual Schema Changes: Any database schema modification requires manually editing the SQLite file or recreating it from scratch, which is error-prone and not version-controlled.
- No Version Control for Schema: The database structure is embedded in a binary file (
.db), making it impossible to track schema changes through git history. - Team Collaboration Issues: Multiple developers cannot easily sync database schema changes. Each dev must manually replicate schema modifications.
- No Rollback Capability: There's no built-in way to revert schema changes if an update causes issues.
- Deployment Complexity: Applying schema updates to existing production databases requires custom scripts and manual intervention.
- Testing Challenges: Integration tests rely on the pre-seeded database state, making it harder to test schema changes independently.
Current Approach:
- Pre-seeded
storage/players-sqlite3.dbfile committed to repository - Docker entrypoint copies database from
/app/hold/to persistent volume/storage/on first run - Schema defined implicitly in Sequelize model (
src/models/player-model.ts) - No automated way to create, update, or rollback schema changes
Problems for Future Development:
- Adding new columns/tables requires manual SQL or full DB recreation
- No audit trail for schema evolution
- Cannot easily support multiple environments (dev, staging, prod) with different schema states
- Difficult to coordinate schema changes with code deployments
Proposed Solution
Implement Sequelize's migration system to manage database schema changes in a version-controlled, automated, and reversible manner. Migrations provide:
- Version-Controlled Schema: Each migration is a timestamped file in git, providing full schema history
- Automated Application: Run
npx sequelize-cli db:migrateto apply pending migrations - Rollback Support: Use
db:migrate:undoto revert changes safely - Team Synchronization: Developers pull migration files and run migrations locally to sync schemas
- CI/CD Integration: Migrations can run automatically as part of deployment pipelines
- Environment Parity: Ensure dev, staging, and prod databases stay in sync
Benefits:
- β Track schema changes in git with commit messages
- β Apply schema updates consistently across environments
- β Rollback problematic changes without data loss
- β Generate initial schema from existing models
- β Support future features (adding columns, indexes, foreign keys)
- β Enable database seeding as separate step from schema creation
Migration Workflow:
- Developer creates migration:
npx sequelize-cli migration:generate --name add-player-nationality - Developer writes
up()anddown()functions in migration file - Migration committed to git and pushed
- Other developers pull and run:
npx sequelize-cli db:migrate - Production deployment runs migrations automatically
Suggested Approach
1. Install Sequelize CLI and Dependencies
npm install --save-dev sequelize-cli
npm install --save-dev @types/nodeNote: sequelize is already installed as a dependency.
2. Create .sequelizerc Configuration File
File: .sequelizerc (root directory)
This tells Sequelize CLI where to find migrations, models, and config files:
const path = require('path');
module.exports = {
'config': path.resolve('config', 'database.json'),
'models-path': path.resolve('src', 'models'),
'seeders-path': path.resolve('database', 'seeders'),
'migrations-path': path.resolve('database', 'migrations'),
};Directory Structure:
database/
βββ migrations/ # Migration files (timestamped)
βββ seeders/ # Seed data files (optional)
config/
βββ database.json # Database configuration for each environment
3. Create Database Configuration File
File: config/database.json
{
"development": {
"dialect": "sqlite",
"storage": "./storage/players-sqlite3.db",
"logging": console.log
},
"test": {
"dialect": "sqlite",
"storage": ":memory:",
"logging": false
},
"production": {
"dialect": "sqlite",
"storage": "/storage/players-sqlite3.db",
"logging": false
}
}Explanation:
- development: Local SQLite file in
storage/directory - test: In-memory database for fast test execution
- production: Docker volume mount path (
/storage/in container)
Alternative: Use TypeScript config file (config/database.ts) and export dynamic configuration:
import path from 'node:path';
const storagePath = process.env.STORAGE_PATH ?? path.join(process.cwd(), 'storage', 'players-sqlite3.db');
export default {
development: {
dialect: 'sqlite',
storage: storagePath,
logging: console.log,
},
// ... other environments
};4. Generate Initial Migration from Existing Schema
Create a migration that reflects the current Player model schema:
npx sequelize-cli migration:generate --name create-players-tableFile: database/migrations/YYYYMMDDHHMMSS-create-players-table.js
'use strict';
module.exports = {
async up(queryInterface, Sequelize) {
await queryInterface.createTable('players', {
id: {
type: Sequelize.INTEGER,
autoIncrement: true,
primaryKey: true,
allowNull: false,
},
firstName: {
type: Sequelize.STRING,
allowNull: false,
},
middleName: {
type: Sequelize.STRING,
allowNull: true,
},
lastName: {
type: Sequelize.STRING,
allowNull: false,
},
dateOfBirth: {
type: Sequelize.DATE,
allowNull: true,
},
squadNumber: {
type: Sequelize.INTEGER,
allowNull: false,
unique: true,
},
position: {
type: Sequelize.STRING,
allowNull: false,
},
abbrPosition: {
type: Sequelize.STRING,
allowNull: true,
},
team: {
type: Sequelize.STRING,
allowNull: true,
},
league: {
type: Sequelize.STRING,
allowNull: true,
},
starting11: {
type: Sequelize.BOOLEAN,
allowNull: true,
},
});
// Add index on squadNumber for faster lookups
await queryInterface.addIndex('players', ['squadNumber'], {
unique: true,
name: 'players_squad_number_unique',
});
},
async down(queryInterface, Sequelize) {
await queryInterface.dropTable('players');
},
};Key Points:
up()creates the table and indexesdown()reverts changes (drops table)- Sequelize CLI tracks which migrations have run in a
SequelizeMetatable
5. Create Seed Data Migration (Optional)
Generate a seeder to populate initial player data:
npx sequelize-cli seed:generate --name seed-initial-playersFile: database/seeders/YYYYMMDDHHMMSS-seed-initial-players.js
'use strict';
module.exports = {
async up(queryInterface, Sequelize) {
await queryInterface.bulkInsert('players', [
{
id: 1,
firstName: 'Emiliano',
middleName: 'Viviano',
lastName: 'MartΓnez',
dateOfBirth: '1992-08-02',
squadNumber: 23,
position: 'Goalkeeper',
abbrPosition: 'GK',
team: 'Aston Villa',
league: 'Premier League',
starting11: true,
},
// ... more players
], {});
},
async down(queryInterface, Sequelize) {
await queryInterface.bulkDelete('players', null, {});
},
};Run Seeds:
npx sequelize-cli db:seed:all6. Update npm Scripts
Add migration commands to package.json:
"scripts": {
"db:migrate": "sequelize-cli db:migrate",
"db:migrate:undo": "sequelize-cli db:migrate:undo",
"db:migrate:undo:all": "sequelize-cli db:migrate:undo:all",
"db:seed": "sequelize-cli db:seed:all",
"db:seed:undo": "sequelize-cli db:seed:undo:all",
"db:create": "sequelize-cli db:create",
"db:drop": "sequelize-cli db:drop",
"db:reset": "npm run db:migrate:undo:all && npm run db:migrate && npm run db:seed"
}Usage:
npm run db:migrate- Apply pending migrationsnpm run db:migrate:undo- Revert last migrationnpm run db:seed- Populate seed datanpm run db:reset- Reset and rebuild database
7. Update Docker Entrypoint Script
Modify scripts/entrypoint.sh to run migrations instead of copying pre-seeded DB:
#!/bin/sh
set -e
echo "β Executing entrypoint script..."
VOLUME_STORAGE_PATH="/storage/players-sqlite3.db"
echo "β Starting container..."
# Check if database exists, create if not
if [ ! -f "$VOLUME_STORAGE_PATH" ]; then
echo "β οΈ No existing database file found in volume."
echo "Running migrations to create schema..."
NODE_ENV=production npx sequelize-cli db:migrate
echo "β Database initialized at $VOLUME_STORAGE_PATH"
else
echo "β Existing database file found."
echo "Running pending migrations..."
NODE_ENV=production npx sequelize-cli db:migrate
fi
echo "β Ready!"
echo "π Launching app..."
exec "$@"Key Changes:
- Remove pre-seeded database copy logic
- Run
db:migrateon every container start (safe: applies only pending migrations) - Use
NODE_ENV=productionto use correct database config
8. Update Dockerfile to Include Migration Files
Modify Dockerfile to copy migration files and CLI:
# Stage 1: Builder
FROM node:krypton-alpine AS builder
WORKDIR /app
COPY package.json package-lock.json tsconfig.json ./
RUN npm ci
COPY src/ ./src/
COPY database/ ./database/ # Add migrations and seeders
COPY config/ ./config/ # Add database config
COPY .sequelizerc ./ # Add Sequelize CLI config
RUN npm run build && \
npm run swagger:docs && \
npm prune --omit=dev
# Stage 2: Runtime
FROM node:krypton-alpine AS runtime
# ... existing runtime setup ...
# Copy migrations and config for runtime execution
COPY --from=builder /app/database/ ./database/
COPY --from=builder /app/config/ ./config/
COPY --from=builder /app/.sequelizerc ./.sequelizerc
# ... rest of Dockerfile ...Important: Keep sequelize-cli as a regular dependency (not dev-only) or install it globally in the runtime image:
RUN npm install -g sequelize-cli9. Remove Pre-Seeded Database from Repository
After migration setup is complete and tested:
# Remove pre-seeded database (keep directory)
git rm storage/players-sqlite3.db
echo "players-sqlite3.db" >> storage/.gitignore
git add storage/.gitignore
git commit -m "chore: migrate to Sequelize migrations, remove pre-seeded DB"Note: Keep storage/ directory with a .gitkeep file for local development:
touch storage/.gitkeep
git add storage/.gitkeep10. Update Documentation
File: README.md
Add section on database setup:
## Database Setup
This project uses Sequelize migrations to manage the database schema.
### Initial Setup (First Time)
```bash
# Run migrations to create schema
npm run db:migrate
# (Optional) Seed initial data
npm run db:seedCreating New Migrations
# Generate migration file
npx sequelize-cli migration:generate --name add-nationality-column
# Edit migration file in database/migrations/
# Run migration
npm run db:migrateRollback Migrations
# Undo last migration
npm run db:migrate:undo
# Undo all migrations
npm run db:migrate:undo:allDocker
Migrations run automatically on container startup via entrypoint.sh.
Acceptance Criteria
- β
Sequelize CLI Installed:
sequelize-cliand@types/nodeinstalled as dev dependencies - β
Configuration Files Created:
.sequelizercandconfig/database.jsonexist with correct paths - β
Initial Migration Created: Migration file exists in
database/migrations/that createsplayerstable matching current schema - β
Migration Scripts Added: npm scripts (
db:migrate,db:migrate:undo,db:seed) added topackage.json - β
Migrations Run Successfully:
npm run db:migratecreates database from scratch without errors - β
Rollback Works:
npm run db:migrate:undosuccessfully reverts the last migration - β Docker Integration: Dockerfile and entrypoint script updated to run migrations on container startup
- β
Local Development Works: Developers can clone repo, run
npm run db:migrate, and start app without manual DB setup - β CI/CD Compatible: Migrations run successfully in GitHub Actions workflow
- β Documentation Updated: README.md includes clear instructions for migration usage
- β
Pre-Seeded DB Removed:
storage/players-sqlite3.dbremoved from git (with proper.gitignore) - β Seed Data Optional: Seed data extracted to separate seeder file (optional step)
References
Sequelize Documentation
- π Sequelize Migrations - Official migration guide
- π Sequelize CLI - CLI tool documentation
- π Query Interface API - Migration methods reference
- π Sequelize Configuration - Config file structure
Related Issues
- [FEATURE] Add test coverage for Service and Database layers with Jest Mock FunctionsΒ #20 - Test coverage for Service/Database layers (integration tests may need migration setup)
- [FEATURE] Refactor by adding an interface to decouple from ORM ModelΒ #104 - Interface to decouple from ORM (migrations work with or without interface)
- Migrate Player
idfromINTEGERtoUUIDΒ #23 - UUID migration (future: add UUID column via migration)
Migration Best Practices
- Database Migrations Best Practices - General principles
- Sequelize Migration Examples - Community examples
- Zero-Downtime Migrations - Production deployment strategies
TypeScript Integration
- Use
.sequelizercto specify TypeScript paths - Consider using
ts-nodefor TypeScript migration files (optional advanced setup) - Example: Sequelize + TypeScript + Migrations
Testing with Migrations
- Integration tests can use
:memory:database with migrations applied before each test suite - Ensures tests always run against correct schema
- Example pattern:
beforeAll(async () => { await sequelize.sync({ force: true }); // Or run migrations });
CI/CD Integration Examples
-
Run migrations in GitHub Actions before running tests
-
Use separate database configs for test environment
-
Example workflow step:
- name: Run Database Migrations run: npm run db:migrate env: NODE_ENV: test