Bioinfotoolsweb2 (BITW2)

A comprehensive Spring Boot web application for managing bioinformatics workflows, pipeline executions, and data analysis for the GenPat-COHESIVE platform.

Overview

Bioinfotoolsweb2 is the core web service that provides REST APIs for:

• Analysis Management: Create, execute, and monitor bioinformatics analyses
• Pipeline Execution: Integration with external pipeline servers and workflow engines
• Data Cart: Manage sample selections and datasets for batch processing
• File Upload/Download: Handle large genomic files (up to 5GB) with efficient streaming
• Result Management: Store and retrieve analysis results with metadata
• Surveillance: Specialized modules for epidemiological surveillance (ReporTree integration)
• Workflow Orchestration: Multi-step analysis workflows with validation and notifications

Features

• RESTful API: Complete REST API with OpenAPI/Swagger documentation
• Spring Security: Pre-authenticated security model with session-based authorization
• Large File Handling: Support for multipart uploads up to 5GB
• Database Integration: PostgreSQL with HikariCP connection pooling and batch insert optimization
• Webhook Support: External integrations and event notifications
• Result Archival: Automated cleanup with configurable expiration policies
• Multi-Module Support: Extensible architecture for different analysis types
• Mustache Templates: Dynamic HTML rendering for reports and notifications
• Wiki Integration: Documentation and help system integration
• Health Monitoring: Spring Boot Actuator endpoints for application monitoring

API Modules

The application exposes the following REST controller endpoints:

• /analysis - Analysis lifecycle management (create, update, execute, delete)
• /cart - Sample cart and dataset management
• /dataset - Dataset operations and exports
• /discriminator - Entity discriminator management
• /download - File download and result retrieval
• /execution - Pipeline execution monitoring
• /menu - Dynamic menu generation
• /module - Bioinformatics module registry
• /notification - User notifications and alerts
• /result - Analysis result storage and queries
• /step - Workflow step definitions
• /submission - External submission tracking (e.g., ENA, GISAID)
• /surveillance - Surveillance-specific endpoints
• /tag - Metadata tagging system
• /upload - File upload handling
• /user - User profile and preferences
• /validation - Input validation and data quality checks
• /version - Module version management
• /webhook - External webhook handlers
• /workflow - Workflow definitions and orchestration

Technology Stack

• Spring Boot 2.6.3 - Core application framework
• Spring Security - Authentication and authorization
• Spring Data JDBC - Database access layer
• PostgreSQL - Primary data store
• OpenAPI 3 (SpringDoc) - API documentation
• Jackson - JSON serialization
• Mustache - Template engine
• OpenCSV - CSV parsing and generation
• JSoup - HTML parsing
• Apache Commons (Text, Compress) - Utility libraries
• Lombok - Code generation

Prerequisites

• Java 11 or higher
• PostgreSQL 9.6 or higher
• Maven 3.6 or higher
• External configuration file at /conf/application.yml

Configuration

The application requires an external configuration file located at /conf/application.yml with the following settings:

Database Configuration

spring:
  datasource:
    url: jdbc:postgresql://localhost:5432/cmdbuild
    username: your_db_username
    password: your_db_password

Security Configuration

app:
  security:
    session-key: CMDBuild-Authorization
    service-token: your_service_token
    allowed-origins:
      - "http://localhost:1841"

Pipeline Integration

app:
  pipeline:
    server-url: 'http://your-pipeline-server'
    working-folder: '/path/to/workdir'
    job-queue-folder: '/path/to/queue'
    lock-folder: '/path/to/locks'
    trash-folder: '/path/to/trash'
    expire-in-days: 45

Download Service

app:
  download:
    folder: '/path/to/downloads'
    server-url: 'http://your-download-server'
    context-path: '/downloads'
    expire-in-days: 45

File Upload Limits

spring:
  servlet:
    multipart:
      max-file-size: 5GB
      max-request-size: 5GB

Building

Build the WAR file using Maven:

mvn clean package

The WAR file will be generated at target/bitw2.war.

Deployment

Standalone Mode

Run as a standalone Spring Boot application:

java -jar target/bitw2.war

Servlet Container

Deploy the WAR file to a servlet container (Tomcat 9+, Jetty, etc.):

1. Copy bitw2.war to the container's webapps directory
2. Ensure the external configuration file exists at /conf/application.yml
3. Start the container

The application will be available at http://localhost:8080/bitw2/ (adjust port and context as needed).

API Documentation

Once deployed, interactive API documentation is available via Swagger UI:

http://localhost:8080/bitw2/swagger-ui.html

OpenAPI specification (JSON):

http://localhost:8080/bitw2/v3/api-docs

Monitoring

Spring Boot Actuator endpoints are enabled for application monitoring:

• /actuator/health - Application health status
• /actuator/info - Application information
• /actuator/metrics - Application metrics
• /actuator/loggers - Logger configuration
• /actuator/threaddump - Thread dump
• /actuator/heapdump - Heap dump

Additional endpoints can be enabled via configuration.

Database Schema

The application expects a PostgreSQL database with the CMDBuild schema. Key entities include:

• Analysis definitions and executions
• Sample carts and datasets
• Module configurations
• Result metadata
• User profiles and permissions
• Workflow definitions

Integration Points

External Pipeline Server

The application integrates with external pipeline execution services via:

• Job submission to queue folders
• File-based locking mechanisms
• Result polling and status updates
• Legacy Jenkins API support

Wiki Documentation

Integration with wiki systems for contextual help and documentation:

• Confluence or similar wiki platforms
• REST API for content retrieval
• Configurable base paths and timeouts

Download Service

Automated file download using wget:

• Recursive directory downloads
• Authentication support
• Automatic archival and cleanup

Security Considerations

• Authentication: Pre-authenticated security model (requires upstream authentication)
• Session Management: Cookie-based sessions with configurable keys
• CORS: Configurable allowed origins for cross-origin requests
• File Uploads: Size limits enforced (default 5GB)
• External Configuration: Credentials stored outside WAR file in /conf/application.yml

Logging

Application logs are written to:

• logs/bitw2.log (configurable)
• Console output with color-coded levels
• Log pattern: yy-MM-dd HH:mm:ss LEVEL logger : message

Troubleshooting

Database Connection Issues

Verify PostgreSQL is running and accessible:

psql -h localhost -U postgres -d cmdbuild

Check HikariCP connection pool settings in configuration.

File Upload Failures

Ensure:

• Temporary directory /tmp is writable
• Multipart size limits are configured correctly
• Sufficient disk space for large files

Pipeline Execution Problems

Check:

• Pipeline server URL is accessible
• Working folders exist and have correct permissions
• Lock folder is writable

Related Projects

This application is part of the GenPat-COHESIVE bioinformatics platform:

• cohesive-zuul - API Gateway
• cohesive-bioinfotools2 - Data Import Service
• cohesive-auspice-server - Auspice Visualization
• cohesive-common-resources - Shared UI Resources

Contact

For questions or support, contact: cohesive@izs.it

License

This project is licensed under the MIT License - see the LICENSE file for details.