Support for further databases / public repositories and formats or content

[sreichl]

Only implement if required by one of the projects.

• support download of count matrices/processed data from e.g., GEO
• support download from URLs
• Metadatamapping by DM
  • For annotation DM recently developed a Python package that makes it easier to download metadata from the SRA https://github.com/dmalzl/metadatamapping. It currently only supports biosample metadata but also has code for extracting specific sample data from the ARCHS4 so for expression data it might come in handy.
• EGA
  • https://snakemake-wrappers.readthedocs.io/en/stable/wrappers/ega/fetch.html
• NCBI and Genbank
  • https://snakemake-wrappers.readthedocs.io/en/stable/wrappers/entrez/efetch.html
  • uses this: https://www.ncbi.nlm.nih.gov/books/NBK179288/
• xrootd?
  • https://xrootd.slac.stanford.edu/
  • https://github.com/snakemake/snakemake-storage-plugin-xrootd
• NCBI tools
  • https://www.ncbi.nlm.nih.gov/home/tools/
• Encode
• Human cell atlas data portal
• ArchS4 for GEO count matrices
• MetaSRA all GEO metadata
• CZ: CellxGene (single-cell data from HCA and Chan Zuckerberg)
  • https://www.biorxiv.org/content/10.1101/2023.10.30.563174v1
  • Census
    • R: https://chanzuckerberg.github.io/cellxgene-census/articles/2023/20230808-r_api_release.html
    • python version also exists (probably faster)
• Elixier data platform
• UK Biobank released WGS of 500k individuals
• And for GEO, if it could also support the --ngc flag
  • https://www.ncbi.nlm.nih.gov/sra/docs/sra-dbgap-download/
  • NGC (NCBI Genome Collections) refers to a credential file format used by the NCBI (National Center for Biotechnology Information) to provide authorized access to controlled-access genomic data, particularly from the dbGaP (database of Genotypes and Phenotypes) repository. The NGC file contains encrypted authentication credentials
• Human Aging and Longevity Landscape (HALL)
  • https://ngdc.cncb.ac.cn/hall/index
• The Compendium of Cancer Genome Aberrations (CCGA)
  • https://ccga.io/index.php/The_Compendium_of_Cancer_Genome_Aberrations
• Curated Cancer Cell Atlas (3CA) by Weizmann Institute
  • https://www.weizmann.ac.il/sites/3CA/
• Mendelian Randomization web platform
  • https://www.mrbase.org/
• ReGEO: Restructured Gene Expression Omnibus
  • https://regeo.org/
• scPerturb - Single Cell Perturbation Datasets
  • http://projects.sanderlab.org/scperturb/
• PerturBase: a comprehensive database for single-cell perturbation data analysis and visualization
  • http://www.perturbase.cn/