The File-Transfer service provides upload and download handling for large genomic file data. This repository provides the source code for the File-Transfer server and command line interface applications.
File-Transfer service is a fork of Score, part of Overture's collection of open-source software suite. It has been extended to integrate with PCGL's federated Authorization service.
- Multi-cloud Support: Compatible with AWS S3, Azure Storage, and any object storage with an S3 compliant API (Minio, Ceph, etc.)
- High-performance Transfers: Implements multipart uploads and downloads for optimal throughput
- Genomic Data Handling (SamTools): Supports BAM/CRAM file slicing by genomic region and provides built-in samtools operations for BAM file handling
- Data Integrity: Ensures file integrity through MD5 checksum validation on uploads and downloads
- Security: Implements ACL-based security using OAuth2 with study code-scoped access
- Metadata Integration: Integrates with the Song metadata management system for comprehensive data tracking
- File Bundling: Enables efficient transfer of multiple files in a single bundle
- Resumable Downloads: Supports resuming downloads after network interruptions
- FUSE Support: Offers file system in Userspace (FUSE) support for enhanced file operations
- Interactive API Documentation: Built-in Swagger UI for easy API interaction and exploration
Technical resources for those working with or contributing to the project are available from our official documentation site, the following content can also be read and updated within the /docs folder of this repository.
- Java 11 (OpenJDK)
- Maven 3.5+ (or use provided wrapper)
- VS Code or preferred Java IDE
- Docker Container platform
- For support, feature requests, and bug reports, please see our Support Guide.
- For detailed information on how to contribute to this project, please see our Contributing Guide.
The Overture Platform includes the following Overture Components:
| Software | Description |
|---|---|
| Score | Transfer data to and from any cloud-based storage system |
| Song | Catalog and manage metadata associated to file data spread across cloud storage systems |
| Maestro | Organizing your distributed data into a centralized Elasticsearch index |
| Arranger | A search API with reusable search UI components |
| Stage | A React-based web portal scaffolding |
| Lyric | A model-agnostic, tabular data submission system |
| Lectern | Schema Manager, designed to validate, store, and manage collections of data dictionaries. |
If you'd like to get started using our platform check out our quickstart guides
Overture is supported by grant #U24CA253529 from the National Cancer Institute at the US National Institutes of Health, and additional funding from Genome Canada, the Canada Foundation for Innovation, the Canadian Institutes of Health Research, Canarie, and the Ontario Institute for Cancer Research.
