Current Situation:
- 15+ year old codebase (Java/Struts from ~2008)
- Critical scientific data: 500K+ specimens, 200K+ images
- Mixed technology stack: Java, Python, PHP, MySQL 5
- Known security vulnerabilities requiring attention
- Struts framework is effectively end-of-life (last major update 2013)
Goal: Modernize AntWeb for 20+ year sustainability without breaking existing functionality or losing data.
Strategy: Incremental modernization with clear phases, prioritizing security and long-term maintainability. Each phase builds on the previous, creating a stable foundation for future development.
Development priorities are ordered by:
- Dependencies - What must be completed before other work can begin
- Risk - Security and data integrity concerns
- Impact - Value delivered to users and maintainability
- Isolation - Can be developed without affecting other systems
🔴 Must be completed first - addresses immediate risks
Duration: 12-16 weeks
Detailed Plan: See PHASE1.md for week-by-week implementation guide
-
Release Management & Versioning (NEW - Start Immediately)
- Establish semantic versioning
- GitHub Releases integration
- Website links to releases and roadmap
- Automated changelog generation
-
Security & Authentication
- Modern authentication system (Spring Security)
- Password hashing (bcrypt)
- Role-based access control
- Audit logging
-
Dependency Updates
- Update all Java dependencies
- Update Python packages
- Update JavaScript libraries
- Automated security scanning
-
Database Modernization
- MySQL 5 → MySQL 8 migration
- MyISAM → InnoDB conversion
- Character set updates (UTF-8MB4)
- Performance optimization
🟡 Enables frontend independence and future development 4. Complete REST API 5. Asynchronous Processing Pipeline
🟢 Improves user experience and developer workflow 6. Modern Frontend Framework 7. Progressive Enhancement
🟢 Long-term code maintainability 8. Struts Replacement 9. Microservices Architecture (Optional)
🔵 Throughout all phases
- Documentation & Knowledge Transfer
- Testing & Quality Assurance
- Performance Optimization
Why First:
- Current login system likely has vulnerabilities
- Foundation for all other changes
- Relatively isolated (doesn't affect other systems)
- Clear scope for volunteers
Current State:
LoginDb.java - Basic authentication
Password storage: Likely MD5 or SHA1 (outdated)
Session management: Basic
No 2FA, no OAuthTarget State:
Modern authentication:
- bcrypt password hashing
- JWT tokens for API
- OAuth2 support (optional)
- Role-based access control (RBAC)
- Audit logging
- Password reset workflow
- 2FA (optional but recommended)
Implementation Plan:
Step 1.1: Audit Current System (Week 1-2)
- Document current login flow
- Identify all authentication points
- List all user roles and permissions
- Security vulnerability assessment
Step 1.2: Design New System (Week 3-4)
- Choose auth library (Spring Security or standalone)
- Design database schema for new auth
- API authentication strategy (JWT)
- Migration plan for existing users
Step 1.3: Implement Core Auth (Week 5-12)
- Create new
authenticationmodule - Implement bcrypt password hashing
- JWT token generation/validation
- Role-based permission system
- Audit logging system
Step 1.4: Migrate Users (Week 13-14)
- Force password reset for all users
- Migrate to new auth system
- Test thoroughly
- Document new system
Step 1.5: Add Advanced Features (Week 15-16)
- Password reset via email
- Account lockout after failed attempts
- Session timeout management
- Optional: 2FA
Technical Complexity:
- Core Implementation: Authentication library integration, password migration
- Database Changes: New authentication tables, user migration scripts
- Testing Requirements: Security testing, penetration testing, user acceptance testing
Success Criteria:
- Zero critical authentication vulnerabilities
- All passwords using modern hashing (bcrypt/argon2)
- Comprehensive audit logging
- Password reset and account recovery workflows
- Role-based access control fully functional
Estimated Timeline: 3-4 months
Why Second:
- Known security vulnerabilities requiring attention
- Affects entire system
- Required for security compliance
- Foundation for future updates
Current State:
- Struts 2.x (likely 2.3.x - end of life)
- MySQL 5.x (end of life 2023)
- Java dependencies (10+ years old)
- Python packages (various ages)
- JavaScript libraries (jQuery, YUI)
Target State:
- Latest security patches applied
- Dependency versions documented
- Automated dependency scanning
- Regular update schedule
Implementation Plan:
Step 2.1: Inventory & Analysis (Week 1-2)
- List all dependencies (Java, Python, JS)
- Check versions against CVE databases
- Prioritize by severity
- Test in dev environment
Step 2.2: Java Dependencies (Week 3-6)
- Update Apache Commons libraries
- Update database connectors
- Update logging frameworks
- Update XML parsers
- Test after each update
Step 2.3: Python Dependencies (Week 7-8)
- Update Flask to latest
- Update SQLAlchemy
- Update PyMySQL
- Update all API dependencies
Step 2.4: JavaScript Libraries (Week 9-10)
- Replace YUI uploader (dead project)
- Update jQuery to 3.x
- Add dependency scanning to CI/CD
Step 2.5: Database (Week 11-12)
- Plan MySQL 5 → MySQL 8 migration
- Test compatibility
- Document changes needed
Technical Complexity:
- Dependency Analysis: CVE scanning, compatibility testing
- Breaking Changes: Code updates required for new versions
- Testing Requirements: Regression testing across all modules
Success Criteria:
- All dependencies less than 2 years old
- Zero critical or high-severity vulnerabilities
- Automated dependency scanning in CI/CD
- Documented update procedures
Estimated Timeline: 3-4 months
Why Third:
- MySQL 5 end of life (2023)
- Foundation for better performance
- Required before other changes
- Complex but well-defined scope
Current State:
MySQL 5.x
- MyISAM tables (old storage engine)
- Limited JSON support
- No window functions
- Slower than modern MySQL
Target State:
MySQL 8.0+
- InnoDB tables (ACID compliant)
- JSON column support
- Window functions
- Better performance
- Future PostgreSQL option
Implementation Plan:
Step 3.1: Assessment (Week 1-2)
- Full database audit
- Identify MyISAM tables
- List stored procedures/triggers
- Check character encoding issues
- Performance baseline
Step 3.2: Dev Environment Upgrade (Week 3-4)
- Install MySQL 8 in dev
- Test application compatibility
- Fix breaking changes
- Document issues found
Step 3.3: Schema Updates (Week 5-8)
- Convert MyISAM → InnoDB
- Update character sets (utf8mb4)
- Add missing indexes
- Optimize slow queries
- Add database constraints
Step 3.4: Migration Testing (Week 9-10)
- Backup production database
- Test restore on MySQL 8
- Run full test suite
- Performance testing
- Rollback plan
Step 3.5: Production Migration (Week 11-12)
- Schedule downtime window
- Migrate production database
- Verify data integrity
- Monitor performance
- Document lessons learned
Technical Complexity:
- Data Migration: Live migration with minimal downtime
- Schema Changes: MyISAM to InnoDB conversion, character set updates
- Testing Requirements: Data integrity verification, performance benchmarking
Success Criteria:
- MySQL 8.0+ in production
- All tables using InnoDB storage engine
- UTF-8MB4 character encoding throughout
- Improved query performance (measurable benchmarks)
- Zero data loss during migration
Estimated Timeline: 4-6 months
Prerequisites: Dependency updates completed
Why This Phase:
- Already have partial API (Python Flask)
- Foundation for frontend independence
- Enables mobile apps, integrations
- Modern development pattern
Current State:
Python Flask API (v3)
- Partial coverage (specimens, taxa, images)
- Missing: uploads, user management, admin functions
- Inconsistent response formats
- Limited documentation
Target State:
Comprehensive REST API
- 100% feature coverage
- Consistent JSON responses
- OpenAPI/Swagger documentation
- Rate limiting
- Versioning strategy
- GraphQL consideration
Implementation Plan:
Step 4.1: API Coverage Analysis (Month 1)
- List all web application features
- Map to existing API endpoints
- Identify gaps
- Prioritize missing endpoints
Step 4.2: Core API Expansion (Month 2-6)
- User management endpoints
- Upload endpoints (specimens, images)
- Search endpoints (advanced)
- Statistics/analytics endpoints
- Admin functions API
- Batch operations API
Step 4.3: API Documentation (Month 7-8)
- OpenAPI 3.0 specification
- Swagger UI integration
- Code examples in multiple languages
- Interactive API explorer
- Postman collection
Step 4.4: API Enhancement (Month 9-12)
- GraphQL endpoint (optional)
- WebSocket support for real-time updates
- Bulk export capabilities
- API analytics/monitoring
- Developer portal
Technical Complexity:
- API Development: RESTful endpoint design, authentication integration
- Documentation: OpenAPI 3.0 specification, interactive documentation
- Testing Requirements: API integration tests, performance testing
Success Criteria:
- 100% feature parity with web application
- Comprehensive OpenAPI documentation
- API response times < 200ms (95th percentile)
- Rate limiting and authentication fully functional
- Client libraries in at least 2 languages (Python, JavaScript)
Estimated Timeline: 6-8 months
Prerequisites: Database modernization completed
Why Important:
- Separate data processing from web serving
- Better reliability
- Scalability
- Modern architecture
Current State:
Tightly coupled:
- Upload processing blocks UI
- Statistics calculated synchronously
- Image processing inline
- Search index updates inline
Target State:
Asynchronous processing:
- Background job queue (Celery/RabbitMQ)
- Scheduled tasks for statistics
- Event-driven architecture
- Independent services
Implementation Plan:
Step 5.1: Job Queue Setup (Month 1-2)
- Install RabbitMQ or Redis
- Set up Celery workers
- Create job monitoring dashboard
- Define job priority levels
Step 5.2: Move Processing to Background (Month 3-8)
- Specimen upload processing
- Image processing pipeline
- Statistics calculations
- Search index updates
- Email notifications
Step 5.3: Scheduled Jobs (Month 9-10)
- Nightly statistics recalculation
- Weekly full reindex
- Monthly data exports
- Database maintenance tasks
Step 5.4: Monitoring & Alerting (Month 11-12)
- Job status dashboard
- Failed job alerts
- Performance metrics
- Resource monitoring
Technical Complexity:
- Infrastructure: Message queue setup (RabbitMQ/Redis), worker processes
- Code Refactoring: Move synchronous operations to background jobs
- Monitoring: Job status tracking, failure handling, alerting
Success Criteria:
- All long-running operations moved to background processing
- Job monitoring dashboard operational
- Failed job retry logic implemented
- Processing time improvements (measurable)
Estimated Timeline: 4-6 months
Prerequisites: API completion
Why This Phase:
- Separate frontend from backend
- Modern developer experience
- Better user experience
- Easier to maintain long-term
Current State:
JSP pages (Java Server Pages)
- Server-side rendering
- Tightly coupled to Java backend
- Limited interactivity
- Difficult to test
- Old JavaScript (jQuery, YUI)
Target State:
Modern SPA (Single Page Application)
- React, Vue, or Svelte
- API-driven
- Component-based
- Fast, responsive
- Mobile-friendly
- Progressive Web App (PWA)
The Big Decision:
Option A: React ✅ RECOMMENDED
- Pros: Largest ecosystem, most developers, best job market
- Cons: Steeper learning curve, more complex
- Best for: Long-term sustainability, volunteer recruitment
Option B: Vue
- Pros: Easier learning curve, great documentation
- Cons: Smaller ecosystem
- Best for: Rapid development, simpler apps
Option C: Svelte
- Pros: Smallest bundle size, fastest performance
- Cons: Smallest ecosystem, fewer developers
- Best for: Performance-critical apps
Implementation Plan:
Step 6.1: Frontend Foundation (Month 1-3)
- Choose framework (recommend React)
- Set up build system (Vite/Webpack)
- Create design system/component library
- Establish coding standards
- Set up testing framework
Step 6.2: Parallel Development (Month 4-12)
- Build new frontend on subdomain (next.antweb.org)
- Implement page by page
- Use existing API
- A/B testing
- User feedback collection
Priority Pages (in order):
- Search results
- Specimen detail page
- Taxon description page
- Browse/navigation
- Upload interface (last - most complex)
Step 6.3: Gradual Migration (Month 13-24)
- Redirect one page at a time
- Monitor performance/errors
- Collect user feedback
- Fix issues before proceeding
- Complete migration
Step 6.4: Deprecate Old Frontend (Month 25-36)
- Keep old JSP as fallback
- Gradual removal of Java UI code
- Full switch to new frontend
- Remove old code
Technical Complexity:
- Framework Selection: React/Vue/Svelte evaluation and setup
- Component Development: Design system, reusable components
- Migration Strategy: Parallel development, gradual rollout
- Testing: Component tests, end-to-end tests, accessibility testing
Success Criteria:
- Modern, responsive web application
- Page load times < 2 seconds
- Lighthouse score > 90
- Zero accessibility violations (WCAG 2.1 AA)
- Mobile-friendly interface
Estimated Timeline: 12-18 months
Prerequisites: Complete REST API operational
Why Last:
- Most complex change
- Frontend migration must be complete first
- Can be done gradually
- Minimal user impact if API is solid
Current State:
Apache Struts 2.x
- Action classes handle web requests
- JSP rendering (being replaced in Phase 3)
- Business logic mixed with presentation
- ~200+ Action classes
Target State:
Spring Boot microservices
- REST API only (no views)
- Clear separation of concerns
- Microservice architecture option
- Easy to deploy and scale
The Big Decision:
Option A: Spring Boot Monolith ✅ RECOMMENDED
- Pros: Well-supported, huge ecosystem, easy migration path
- Cons: Still Java (might want to diversify)
- Best for: Incremental migration, minimal disruption
Option B: Python FastAPI
- Pros: Modern, fast, same language as existing API
- Cons: Larger rewrite, fewer Java developers
- Best for: Clean break, performance-critical
Option C: Keep Struts, Modernize Around It
- Pros: Least work, no migration risk
- Cons: Technical debt, hard to find developers
- Best for: Resource-constrained scenarios
Implementation Plan (Assuming Spring Boot):
Step 7.1: Preparation (Month 1-3)
- Set up Spring Boot skeleton
- Migrate database access layer (Db classes)
- Create Spring repositories
- Test database connectivity
- Establish patterns
Step 7.2: Service Layer Migration (Month 4-12)
- Extract business logic from Actions
- Create Spring services
- Migrate piece by piece
- Test each service
- Keep Struts for now
Step 7.3: API Migration (Month 13-24)
- Move Python API endpoints to Spring
- Or: Keep Python API (microservice)
- Ensure feature parity
- Performance testing
- Gradual traffic shift
Step 7.4: Remove Struts (Month 25-36)
- Delete Struts Actions (frontend replaced)
- Remove Struts dependencies
- Clean up remaining Java code
- Full Spring Boot app
- Deployment simplification
Technical Complexity:
- Architecture Redesign: Service layer extraction, dependency injection
- Framework Migration: Spring Boot setup, configuration management
- Code Refactoring: Action classes to controllers/services
- Testing: Unit tests, integration tests, regression testing
Success Criteria:
- Zero Struts dependencies
- Modern Spring Boot application
- Deployment time < 5 minutes
- Code test coverage > 70%
- All functionality preserved
Estimated Timeline: 12-18 months
Prerequisites: Frontend migration complete (frontend no longer depends on JSP)
Activities:
- Maintain technical documentation (already started!)
- Create video tutorials
- Onboarding guides for volunteers
- Architecture decision records (ADRs)
- Regular "state of the codebase" reports
Frequency: Monthly reviews, quarterly updates
Activities:
- Expand test coverage (target: 70%+)
- Automated integration tests
- Performance regression tests
- Security scanning (OWASP ZAP)
- Load testing
Frequency: Every code change
Activities:
- Database query optimization
- Image delivery optimization (CDN)
- Caching strategy (Redis)
- API response time monitoring
- Frontend bundle size reduction
Frequency: Quarterly performance audits
If you have resources for a bigger change:
- Modern stack: Next.js (React) + FastAPI (Python) + PostgreSQL
- Read-only access to old database
- Build new features on new stack
- No migration yet
- Gradual data sync old → new
- Dual-write period (write to both)
- Extensive testing
- Soft launch new platform
- Redirect users to new platform
- Keep old as backup for 1 year
- Complete shutdown of old system
Pros:
- Clean, modern codebase
- No technical debt
- Easier to understand
Cons:
- Expensive (2-3x more work)
- Feature parity challenge
- Risk of scope creep
- Requires dedicated team
Phase 1: Security & Foundation (Months 1-12)
- New authentication system
- Dependency updates to latest stable versions
- MySQL 5 → MySQL 8 migration
- Milestone: Secure, modern database foundation
Phase 2: API-First Architecture (Months 13-24)
- Complete REST API (100% feature coverage)
- Background job processing system
- OpenAPI documentation
- Milestone: Backend services decoupled from frontend
Phase 3: Frontend Modernization (Months 25-48)
- Modern JavaScript framework (React recommended)
- Component-based architecture
- Progressive migration from JSP
- Milestone: Modern user experience
Phase 4: Backend Modernization (Months 49-60)
- Struts → Spring Boot migration
- Service-oriented architecture
- Deployment automation
- Milestone: Complete technical modernization
Release Management Comes First:
- Establishes process for tracking all changes
- Can be done quickly (2 weeks)
- No code dependencies
- Benefits all subsequent work
- Improves project professionalism
Phase 1 Must Come First:
- Security vulnerabilities must be addressed immediately
- Database modernization is foundation for all other work
- Dependencies must be current before framework changes
Phase 2 Enables Phase 3:
- Frontend cannot be independent without complete API
- Background jobs prevent UI blocking during development
- API documentation guides frontend development
Phase 3 Enables Phase 4:
- Must remove JSP dependency before replacing Struts
- Frontend-backend decoupling reduces migration risk
- Modern frontend provides fallback during backend changes
Phase 4 Comes Last:
- Most complex and risky change
- Requires all previous phases complete
- Internal change with minimal user impact if API is solid
✅ YES - TOP PRIORITY
Reasons:
- Security risk is highest here
- Isolated system (low risk to other features)
- Clear scope for volunteers
- Foundation for future work
- Quick win (2-3 months)
⏸️ LATER - BUT NOT LAST
Reasons:
- Complex, time-consuming
- Requires API and frontend first
- Can wait until those are done
- Struts can limp along if API handles load
- Do in Phase 4 (months 36-48)
✅ YES - PHASE 3
Reasons:
- Modern development pattern
- Better user experience
- Easier to maintain
- Mobile-friendly
- But only AFTER API is complete
⏸️ PARTIAL - CONSIDER
If you mean static site generation:
- Pros: Fast, cheap hosting, scales infinitely
- Cons: No dynamic features, complex updates
- Best for: Mostly read-only sites
- AntWeb challenge: Upload functionality requires dynamic backend
Recommendation:
- Read pages: Static generation (Next.js SSG)
- Interactive pages: Dynamic (API-driven)
- Best of both worlds
- 🔴 Database migration (data loss potential)
- 🔴 Authentication changes (lockout risk)
- 🟡 Struts removal (breaking everything)
- 🟡 Frontend cutover (user confusion)
- Always maintain rollback capability
- Extensive testing in staging
- Gradual migrations (not "big bang")
- Keep old system running in parallel
- User communication plan
- 0 critical vulnerabilities
- All dependencies < 2 years old
- MySQL 8 in production
- 100% users migrated to new auth
- 100% feature coverage in API
- API response time < 200ms (95th percentile)
- OpenAPI docs complete
- 5+ external integrations using API
- Page load time < 2 seconds
- Lighthouse score > 90
- Mobile-friendly (responsive)
- 0 accessibility violations
- 0 Struts dependencies
- Spring Boot in production
- Deployment time < 5 minutes
- Code coverage > 70%
Development Environment:
- Staging environment that mirrors production
- CI/CD pipeline for automated testing
- Code review process
- Version control best practices
Testing Infrastructure:
- Automated test suite (unit, integration, end-to-end)
- Performance benchmarking tools
- Security scanning tools
- Rollback procedures for each phase
Documentation:
- Architecture decision records (ADRs)
- API documentation (OpenAPI)
- Deployment procedures
- Troubleshooting guides
For Each Major Change:
- Develop in parallel (don't break existing system)
- Test extensively in staging
- Plan rollback procedures
- Gradual rollout to production
- Monitor closely after deployment
Data Protection:
- Automated backups before any migration
- Test restore procedures
- Data validation after migrations
- Audit trails for all changes
Communication:
- User notifications for planned changes
- Status page for system health
- Change log for all updates
- Support channels for issues
Security Audit:
# Run security scanner
./scripts/security-scan.sh
# Check dependency vulnerabilities
npm audit
pip-audit
./gradlew dependencyCheckAnalyzeCurrent Authentication Review:
-- Check password storage method
SELECT password FROM login LIMIT 1;
-- If you see plain MD5/SHA1 hashes, authentication upgrade is critical
-- Count users and roles
SELECT role, COUNT(*) FROM login GROUP BY role;
-- Review access patterns
SELECT * FROM login WHERE last_login > DATE_SUB(NOW(), INTERVAL 6 MONTH);Database Assessment:
-- Check MySQL version
SELECT VERSION();
-- Check table engines
SELECT TABLE_NAME, ENGINE FROM information_schema.TABLES
WHERE TABLE_SCHEMA = 'ant';
-- Identify slow queries
SELECT * FROM mysql.slow_log LIMIT 10;- Set up development environment (see DEVELOPMENT.md)
- Create staging environment (exact copy of production)
- Implement backup automation (if not already done)
- Document current architecture (use ARCHITECTURE.md as template)
- Choose first project from Phase 1
Years 1-2: Foundation
- ✅ Modern, secure authentication
- ✅ Current dependencies and database
- ✅ Complete API enabling future development
Years 3-4: User Experience
- ✅ Modern, responsive frontend
- ✅ Improved performance and usability
- ✅ Mobile-friendly interface
Years 5+: Internal Modernization
- ✅ Clean, maintainable backend
- ✅ Easy deployment and scaling
- ✅ Long-term sustainability
Incremental Value: Each phase delivers concrete improvements while maintaining system stability.
Risk Mitigation: Gradual changes with rollback capabilities at each step.
Dependency-Driven: Later phases depend on earlier ones, preventing waste and rework.
Future-Proof: Modern architecture allows for continued evolution beyond this roadmap.
Phase 1 Complete When:
- Zero critical security vulnerabilities
- MySQL 8 in production
- All dependencies current
Phase 2 Complete When:
- 100% API feature coverage
- All long-running operations backgrounded
- Comprehensive API documentation
Phase 3 Complete When:
- Modern frontend serving all pages
- Lighthouse score > 90
- Mobile usage supported
Phase 4 Complete When:
- Zero Struts dependencies
- Deployment time < 5 minutes
- Code coverage > 70%
The most important first step: Begin with authentication security. It's the highest priority, has the clearest scope, and provides the foundation for all future work.
- Spring Security (if staying with Java)
- Auth0 (if you want managed service)
- Keycloak (if you want self-hosted SSO)
- React (recommended - largest ecosystem)
- Next.js (React framework with SSG)
- Tailwind CSS (styling)
- Spring Boot (if keeping Java)
- FastAPI (if moving to Python)
- PostgreSQL (if moving from MySQL)
- Docker (already using - good!)
- Kubernetes (if scaling needed)
- GitHub Actions (CI/CD)
- Cloudflare (CDN for images)
- Sentry (error tracking)
- Prometheus + Grafana (metrics)
- ELK Stack (logging)
- JUnit (Java backend)
- pytest (Python API)
- Jest + React Testing Library (frontend)
- Cypress (end-to-end)
On GitHub:
- Releases: https://github.com/calacademy-research/antweb/releases
- This Roadmap: https://github.com/calacademy-research/antweb/blob/master/docs/ROADMAP.md
On AntWeb.org: Once Phase 1 is complete, these links will appear in:
- Website footer (all pages)
- About page
- Admin/curator dashboard
Review and Understand:
- Read the complete roadmap
- Review technical documentation in
/docs - Set up development environment (see DEVELOPMENT.md)
- Understand current architecture (see ARCHITECTURE.md)
Pick a Task:
- Check GitHub Issues for roadmap-related tasks
- Start with Phase 1 items (highest priority)
- Review success criteria before beginning
- Coordinate with maintainers to avoid duplicate work
Development Process:
- Create feature branch from master
- Follow coding standards in DEVELOPMENT.md
- Write tests for all changes
- Update documentation
- Submit pull request with clear description
- Open a GitHub Issue with tag
roadmap - Discuss in project forum or mailing list
- Contact maintainers for major architectural questions
This roadmap is a living document. As technology evolves and project needs change, priorities may shift. Major updates will be documented in the CHANGELOG.
Last Updated: January 2026
Next Review: July 2026