Merge pull request #1914 from mamtagiri/main

Jill Grant · web-flow · commit 44c4179fc57a · 2024-12-10T16:52:21.000-07:00
update genomics open data lake doc
diff --git a/articles/open-datasets/dataset-1000-genomes.md b/articles/open-datasets/dataset-1000-genomes.md
@@ -9,8 +9,6 @@ ms.date: 07/10/2024
 
 # 1000 Genomes
 
-[!INCLUDE [Open Dataset access change notice](./includes/open-datasets-change-note.md)]
-
 The 1000 Genomes Project ran between 2008 and 2015, to create the largest public catalog of human variation and genotype data. The final data set contains data for 2,504 individuals from 26 populations and 84 million identified variants. For more information, visit the 1000 Genome Project [website](https://www.internationalgenome.org/) and these publications:
 
 [Pilot Analysis: A map of human genome variation from population-scale sequencing Nature 467, 1061-1073 (28 October 2010)](https://www.nature.com/articles/nature09534)
@@ -33,6 +31,16 @@ This dataset is a mirror of [this](ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/) FT
 
 This dataset contains approximately 815 TB of data. It receives daily updates.
 
+## Storage location
+
+This dataset is stored in the West US 2 and West Central US Azure regions. We recommend locating compute resources in West US 2 or West Central US for affinity.
+
+## Data access
+
+West US 2:"https://dataset1000genomes.blob.core.windows.net/dataset'"
+
+West Central US: "https://dataset1000genomes-secondary.blob.core.windows.net/dataset"
+
 ## Use Terms
 
 Following the final publications, data from the 1000 Genomes Project is publicly available, without embargo, to anyone for use under the terms provided by the [dataset source](http://www.internationalgenome.org/data). Use of the data should be cited per details available in the 1000 Genome Project [FAQ resource](https://www.internationalgenome.org/faq).
diff --git a/articles/open-datasets/dataset-clinvar-annotations.md b/articles/open-datasets/dataset-clinvar-annotations.md
@@ -9,8 +9,6 @@ ms.date: 06/13/2024
 
 # ClinVar Annotations
 
-[!INCLUDE [Open Dataset access change notice](./includes/open-datasets-change-note.md)]
-
 The [ClinVar](https://www.ncbi.nlm.nih.gov/clinvar/) resource is a freely accessible, public archive of reports - with supporting evidence - about the relationships among human variations and phenotypes. It facilitates access to and communication about the claimed relationships between human variation and observed health status, and about the history of that interpretation. It provides access to a broader set of clinical interpretations that researchers can incorporate into genomics workflows and applications.
 
 Visit the [Data Dictionary](https://www.ncbi.nlm.nih.gov/projects/clinvar/ClinVarDataDictionary.pdf) and the [FAQ resource](https://www.ncbi.nlm.nih.gov/clinvar/docs/faq/) for more information about the data.
@@ -20,26 +18,31 @@ Visit the [Data Dictionary](https://www.ncbi.nlm.nih.gov/projects/clinvar/ClinVa
 ## Data source
 
 This dataset is a mirror of the National Library of Medicine ClinVar [FTP resource](https://ftp.ncbi.nlm.nih.gov/pub/clinvar/xml/).
+[FTP resource](https://ftp.ncbi.nlm.nih.gov/pub/clinvar/)
+
+[FTP Overview](https://www.ncbi.nlm.nih.gov/clinvar/docs/ftp_primer/)
 
 ## Data update frequency
 
 This dataset receives daily updates.
 
-## Data Access
+## Storage location
 
-[FTP resource](https://ftp.ncbi.nlm.nih.gov/pub/clinvar/)
+This dataset is stored in the West US 2 and West Central US Azure regions. We recommend locating compute resources in West US 2 or West Central US for affinity.
 
-[FTP Overview](https://www.ncbi.nlm.nih.gov/clinvar/docs/ftp_primer/)
+## Data Access
+
+West US 2:"https://datasetclinvar.blob.core.windows.net/dataset'"
+West Central US: "https://datasetclinvar-secondary.blob.core.windows.net/dataset"
 
 ## Use Terms
+
 Data is available without restrictions. More information and citation details, see [Accessing and using data in ClinVar](https://www.ncbi.nlm.nih.gov/clinvar/docs/maintenance_use/).
 
 ## Contact
 
 For any questions or feedback about this dataset, contact [clinvar@ncbi.nlm.nih.gov](mailto:clinvar@ncbi.nlm.nih.gov).
 
-## Data access
-
 ### Azure Notebooks
 
 # [azure-storage](#tab/azure-storage)
diff --git a/articles/open-datasets/dataset-encode.md b/articles/open-datasets/dataset-encode.md
@@ -8,8 +8,6 @@ ms.date: 04/16/2021
 
 # ENCODE: Encyclopedia of DNA Elements
 
-[!INCLUDE [Open Dataset access change notice](./includes/open-datasets-change-note.md)]
-
 The [Encyclopedia of DNA Elements (ENCODE) Consortium](https://www.encodeproject.org/help/project-overview/) is an ongoing international collaboration of research groups funded by the National Human Genome Research Institute (NHGRI). ENCODE's goal is to build a comprehensive parts list of functional elements in the human genome, including elements that act at the protein and RNA levels, and regulatory elements that control cells and circumstances in which a gene is active.
 
 ENCODE investigators employ various assays and methods to identify functional elements. The discovery and annotation of gene elements is accomplished primarily by sequencing a diverse range of RNA sources, comparative genomics, integrative bioinformatic methods, and human curation. Regulatory elements are typically investigated through DNA hypersensitivity assays, assays of DNA methylation, and immunoprecipitation (IP) of proteins that interact with DNA and RNA, that is, modified histones, transcription factors, chromatin regulators, and RNA-binding proteins, followed by sequencing.
@@ -34,8 +32,6 @@ West US 2: 'https://datasetencode.blob.core.windows.net/dataset'
 
 West Central US: 'https://datasetencode-secondary.blob.core.windows.net/dataset'
 
-[SAS Token](/azure/storage/common/storage-sas-overview): ?sv=2019-10-10&si=prod&sr=c&sig=9qSQZo4ggrCNpybBExU8SypuUZV33igI11xw0P7rB3c%3D
-
 ## Use Terms
 
 External data users may freely download, analyze, and publish results based on any ENCODE data without restrictions, regardless of type or size, and includes no grace period for ENCODE data producers, either as individual members or as part of the Consortium. Researchers using unpublished ENCODE data are encouraged to contact the data producers to discuss possible publications. The Consortium will continue to publish the results of its own analysis efforts in independent publications.
diff --git a/articles/open-datasets/dataset-gatk-resource-bundle.md b/articles/open-datasets/dataset-gatk-resource-bundle.md
@@ -8,7 +8,6 @@ ms.date: 04/16/2021
 
 # GATK Resource Bundle
 
-[!INCLUDE [Open Dataset access change notice](./includes/open-datasets-change-note.md)]
 
 The [GATK resource bundle](https://gatk.broadinstitute.org/hc/articles/360035890811-Resource-bundle) is a collection of standard files for working with human resequencing data with the GATK.
 
@@ -39,49 +38,35 @@ This dataset is stored in the West US 2 and West Central US Azure regions. Alloc
     West US 2: 'https://datasetgatkbestpractices.blob.core.windows.net/dataset'
     
     West Central US: 'https://datasetgatkbestpractices-secondary.blob.core.windows.net/dataset'
-    
-    [SAS Token](/azure/storage/common/storage-sas-overview): ?sv=2020-04-08&si=prod&sr=c&sig=6SaDfKtXAIfdpO%2BkvNA%2FsTNmNij%2Byh%2F%2F%2Bf98WAUqs7I%3D
 
 2. datasetgatklegacybundles
 
     West US 2: 'https://datasetgatklegacybundles.blob.core.windows.net/dataset'
     
     West Central US: 'https://datasetgatklegacybundles-secondary.blob.core.windows.net/dataset'
-    
-    [SAS Token](/azure/storage/common/storage-sas-overview): ?sv=2020-04-08&si=prod&sr=c&sig=xBfxOPBqHKUCszzwbNCBYF0k9osTQjKnZbEjXCW7gU0%3D
 
 3. datasetgatktestdata
 
     West US 2: 'https://datasetgatktestdata.blob.core.windows.net/dataset'
     
     West Central US: 'https://datasetgatktestdata-secondary.blob.core.windows.net/dataset'
-    
-    [SAS Token](/azure/storage/common/storage-sas-overview): ?sv=2020-04-08&si=prod&sr=c&sig=fzLts1Q2vKjuvR7g50vE4HteEHBxTcJbNvf%2FZCeDMO4%3D
 
 4. datasetpublicbroadref
     
     West US 2: 'https://datasetpublicbroadref.blob.core.windows.net/dataset'
     
     West Central US: 'https://datasetpublicbroadref-secondary.blob.core.windows.net/dataset'
-    
-    [SAS Token](/azure/storage/common/storage-sas-overview): ?sv=2020-04-08&si=prod&sr=c&sig=DQxmjB4D1lAfOW9AxIWbXwZx6ksbwjlNkixw597JnvQ%3D
 
     South Central US: 'https://datasetpublicbroadrefsc.blob.core.windows.net/dataset'
     
-    [SAS Token](/azure/storage/common/storage-sas-overview): ?sv=2023-01-03&st=2024-02-12T19%3A56%3A11Z&se=2029-02-13T19%3A56%3A00Z&sr=c&sp=rl&sig=oGiNUGZ08PaabHVNtIiVEpJ1kcyqcL6ZadQcuN2ns%2FM%3D
-
 6. datasetbroadpublic
 
     West US 2: 'https://datasetbroadpublic.blob.core.windows.net/dataset'
     
     West Central US: 'https://datasetbroadpublic-secondary.blob.core.windows.net/dataset'
-    
-    [SAS Token](/azure/storage/common/storage-sas-overview): ?sv=2020-04-08&si=prod&sr=c&sig=u%2Bg2Ab7WKZEGiAkwlj6nKiEeZ5wdoJb10Az7uUwis%2Fg%3D
 
     South Central US: 'https://datasetbroadpublicsc.blob.core.windows.net/dataset'
-         
-    [SAS Token](/azure/storage/common/storage-sas-overview): ?sv=2023-01-03&st=2024-02-12T19%3A58%3A33Z&se=2029-02-13T19%3A58%3A00Z&sr=c&sp=rl&sig=C2lDhe1uwu%2FJnC9rbQO65G6%2BdEUQ%2Fl0VheXrlnIQVAs%3D
-
+   
 ## Use Terms
 
 Visit the [GATK resource bundle official site](https://gatk.broadinstitute.org/hc/articles/360035890811-Resource-bundle).
diff --git a/articles/open-datasets/dataset-human-reference-genomes.md b/articles/open-datasets/dataset-human-reference-genomes.md
@@ -8,8 +8,6 @@ ms.date: 04/16/2021
 
 # Human Reference Genomes
 
-[!INCLUDE [Open Dataset access change notice](./includes/open-datasets-change-note.md)]
-
 This dataset includes two human-genome references assembled by the [Genome Reference Consortium](https://www.ncbi.nlm.nih.gov/grc): Hg19 and Hg38.
 
 For more information on Hg19 (GRCh37) data, see the [GRCh37 report at NCBI](https://www.ncbi.nlm.nih.gov/assembly/GCF_000001405.13/).
@@ -44,12 +42,8 @@ West US 2: 'https://datasetreferencegenomes.blob.core.windows.net/dataset'
 
 West Central US: 'https://datasetreferencegenomes-secondary.blob.core.windows.net/dataset'
 
-[SAS Token](/azure/storage/common/storage-sas-overview): sv=2019-02-02&se=2050-01-01T08%3A00%3A00Z&si=prod&sr=c&sig=JtQoPFqiC24GiEB7v9zHLi4RrA2Kd1r%2F3iFt2l9%2FlV8%3D
-
 South Central US: 'https://datasetreferencegenomesc.blob.core.windows.net/dataset'
 
-[SAS Token](/azure/storage/common/storage-sas-overview): sv=2023-01-03&st=2024-02-12T20%3A07%3A21Z&se=2029-02-13T20%3A07%3A00Z&sr=c&sp=rl&sig=ASZYVyhqLOXKsT%2BcTR8MMblFeI4uZ%2Bnno%2FCnQk2RaFs%3D
-
 ## Use Terms
 
 Data is available without restrictions. For more information and citation details, see the [NCBI Reference Sequence Database site](https://www.ncbi.nlm.nih.gov/refseq/).
diff --git a/articles/open-datasets/dataset-illumina-platinum-genomes.md b/articles/open-datasets/dataset-illumina-platinum-genomes.md
@@ -8,9 +8,7 @@ ms.date: 04/16/2021
 
 # Illumina Platinum Genomes
 
-[!INCLUDE [Open Dataset access change notice](./includes/open-datasets-change-note.md)]
-
-Whole-genome sequencing is enabling researchers worldwide to characterize the human genome more fully and accurately. This requires a comprehensive, genome-wide catalog of high-confidence variants called in a set of genomes as a benchmark. Illumina has generated deep, whole-genome sequence data of 17 individuals in a three-generation pedigree. Illumina has called variants in each genome using a range of currently available algorithms.
+Whole-genome sequencing is enabling researchers worldwide to characterize the human genome more fully and accurately. This effort requires a comprehensive, genome-wide catalog of high-confidence variants called in a set of genomes as a benchmark. Illumina generated deep, whole-genome sequence data of 17 individuals in a three-generation pedigree. Illumina called variants in each genome using a range of currently available algorithms.
 
 For more information on the data, see the official [Illumina site](https://www.illumina.com/platinumgenomes.html).
 
@@ -34,8 +32,6 @@ West US 2: 'https://datasetplatinumgenomes.blob.core.windows.net/dataset'
 
 West Central US: 'https://datasetplatinumgenomes-secondary.blob.core.windows.net/dataset'
 
-[SAS Token](/azure/storage/common/storage-sas-overview): sv=2019-02-02&se=2050-01-01T08%3A00%3A00Z&si=prod&sr=c&sig=FFfZ0QaDcnEPQmWsshtpoYOjbzd4jtwIWeK%2Fc4i9MqM%3D
-
 ## Use Terms
 
 Data is available without restrictions. For more information and citation details, see the [official Illumina site](https://www.illumina.com/platinumgenomes.html).
@@ -55,7 +51,7 @@ For any questions or feedback about the dataset, contact platinumgenomes@illumin
 
 ## Getting the Illumina Platinum Genomes from Azure Open Datasets and Doing Initial Analysis 
 
-Use Jupyter notebooks, GATK, and Picard to do the following:
+Use Jupyter notebooks, GATK, and Picard in analyses such as:
 
 1. Annotate genotypes using VariantFiltration
 2. Select Specific Variants
@@ -77,7 +73,7 @@ This notebook requires the following libraries:
 
 ## Getting the Genomics data from Azure Open Datasets
 
-Several public genomics data has been uploaded as an Azure Open Dataset [here](https://azure.microsoft.com/services/open-datasets/catalog/). We create a blob service linked to this open dataset. You can find examples of data calling procedure from Azure Open Dataset for `Illumina Platinum Genomes` datasets in below:
+Several public genomics data has been uploaded as an Azure Open Dataset [here](https://azure.microsoft.com/services/open-datasets/catalog/). We create a blob service linked to this open dataset. You can find examples of data calling procedure from Azure Open Dataset for `Illumina Platinum Genomes` datasets as:
 
 ### Downloading the specific 'Illumina Platinum Genomes'
 
@@ -164,7 +160,7 @@ Extract fields from a VCF file to a tab-delimited table. This tool extracts spec
 
 INFO/site-level fields:
 
-Use the `-F` argument to extract INFO fields; each field will occupy a single column in the output file. The field can be any standard VCF column (for example, CHROM, ID, QUAL) or any annotation name in the INFO field (for example, AC, AF). The tool also supports the following fields:
+Use the `-F` argument to extract INFO fields; each field occupies a single column in the output file. The field can be any standard VCF column (for example, CHROM, ID, QUAL) or any annotation name in the INFO field (for example, AC, AF). The tool also supports the following fields:
 
 EVENTLENGTH (length of the event)
 TRANSITION (1 for a bi-allelic transition (SNP), 0 for bi-allelic transversion (SNP), -1 for INDELs and multi-allelics)
diff --git a/articles/open-datasets/dataset-immunecode.md b/articles/open-datasets/dataset-immunecode.md
@@ -38,8 +38,6 @@ West US 2: 'https://dataset1000genomes.blob.core.windows.net/dataset'
 
 West Central US: 'https://dataset1000genomes-secondary.blob.core.windows.net/dataset'
 
-[SAS Token](/azure/storage/common/storage-sas-overview): sv=2019-10-10&si=prod&sr=c&sig=9nzcxaQn0NprMPlSh4RhFQHcXedLQIcFgbERiooHEqM%3D
-
 ## Use terms
 
 To learn more about the data use terms refer the [publication](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7418738/) and [Terms of Use](https://clients.adaptivebiotech.com/terms-of-use).
diff --git a/articles/open-datasets/dataset-open-cravat.md b/articles/open-datasets/dataset-open-cravat.md
@@ -8,8 +8,6 @@ ms.date: 04/16/2021
 
 # OpenCravat: Open Custom Ranked Analysis of Variants Toolkit
 
-[!INCLUDE [Open Dataset access change notice](./includes/open-datasets-change-note.md)]
-
 OpenCRAVAT is a Python package that performs genomic variant interpretation including variant impact, annotation, and scoring. OpenCRAVAT has a modular architecture with a wide variety of analysis modules and annotation resources that can be selected and installed/run based on the needs of a given study.
 
 For more information on the data, see the [OpenCravat](https://opencravat.org/).
@@ -34,8 +32,6 @@ West US 2: 'https://datasetopencravat.blob.core.windows.net/dataset'
 
 West Central US: 'https://datasetopencravat-secondary.blob.core.windows.net/dataset'
 
-[SAS Token](/azure/storage/common/storage-sas-overview): sv=2020-04-08&st=2021-03-11T23%3A50%3A01Z&se=2025-07-26T22%3A50%3A00Z&sr=c&sp=rl&sig=J9J9wnJOXsmEy7TFMq9wjcxjXDE%2B7KhGpCUL4elsC14%3D
-
 ## Use Terms
 
 OpenCRAVAT is available with a GPLv3 license. Most data sources are free for non-commercial use. For commercial use, consult the institutional contacts for each data source.
diff --git a/articles/open-datasets/dataset-open-targets.md b/articles/open-datasets/dataset-open-targets.md
@@ -8,8 +8,6 @@ ms.date: 04/16/2021
 
 # Open Targets
 
-[!INCLUDE [Open Dataset access change notice](./includes/open-datasets-change-note.md)]
-
 The Open Targets Platform is a data resource to facilitate the systematic identification and prioritization of potential therapeutic drug targets. This resource integrates publicly available datasets, including those datasets that are generated by the Open Targets consortium, to build and score target-disease associations, aiding in the identification and prioritization of drug targets. Additionally, it incorporates pertinent annotation information about targets, diseases, phenotypes, drugs, and their key relationships.
 
 The Open Targets Genetics highlights variant-centric statistical evidence to allow both prioritization of candidate causal variants at trait-associated loci and identification of potential drug targets. It collects and combines genetic associations gathered from published literature as well as newly derived data from sources like UK Biobank and FinnGen. Additionally, it includes functional genomics information such as chromatin conformation and interactions, along with quantitative trait loci (eQTLs, pQTLs, and sQTLs). Large-scale pipelines apply statistical fine-mapping across thousands of trait-associated loci to resolve association signals and link each variant to its proximal and distal target genes using a 'Locus2Gene' assessment. Integrated cross-trait colocalisation analyses and linking to detailed pharmaceutical compounds extend the capacity of Open Targets Genetics to explore drug repositioning opportunities and shared genetic architecture.
@@ -35,8 +33,6 @@ This dataset is stored in the West US 2 Azure region. Allocating compute resourc
 
 West US 2: `https://datasetopentargets.blob.core.windows.net/dataset`
 
-[SAS Token](/azure/storage/common/storage-sas-overview): sv=2019-10-10&si=prod&sr=c&sig=9nzcxaQn0NprMPlSh4RhFQHcXedLQIcFgbERiooHEqM%3D
-
 
 ## Use terms
 
diff --git a/articles/open-datasets/dataset-panancestry-uk-bio-bank.md b/articles/open-datasets/dataset-panancestry-uk-bio-bank.md
@@ -29,8 +29,6 @@ This dataset is stored in the East US Azure region. We recommend locating comput
 
 East US: 'https://datasetpanukbb.blob.core.windows.net/dataset'
 
-[SAS Token](/azure/storage/common/storage-sas-overview): ?sp=rl&st=2023-05-17T21:26:19Z&se=2050-05-18T05:26:19Z&spr=https&sv=2022-11-02&sr=c&sig=MGvVbVHbmkGKmWmfkHzpcaJEf5G0ljLnBQy6cbrmR%2FA%3D
-
 ## Use Terms
 
 The GWAS results data produced by the Pan-UKB are available free of restrictions under the Creative Commons Attribution 4.0 International (CC BY 4.0). The team requests that you acknowledge and give attribution to both the Pan-UKB project and UK Biobank, and link back to the relevant page, wherever possible. Full terms of use can be found [here](https://pan.ukbb.broadinstitute.org/downloads)
diff --git a/articles/open-datasets/dataset-snpeff.md b/articles/open-datasets/dataset-snpeff.md
@@ -32,8 +32,6 @@ West US 2: 'https://datasetsnpeff.blob.core.windows.net/dataset'
 
 West Central US: 'https://datasetsnpeff-secondary.blob.core.windows.net/dataset'
 
-[SAS Token](/azure/storage/common/storage-sas-overview): sv=2019-10-10&st=2020-09-01T00%3A00%3A00Z&se=2050-09-01T00%3A00%3A00Z&si=prod&sr=c&sig=isafOa9tGnYBAvsXFUMDGMTbsG2z%2FShaihzp7JE5dHw%3D
-
 ## Use Terms
 Data is available without restrictions. More information and citation details, see [Accessing and using data in ClinVar](https://pcingola.github.io/SnpEff/snpeff/introduction/).
 
diff --git a/articles/open-datasets/dataset-the-cancer-genome-atlas.md b/articles/open-datasets/dataset-the-cancer-genome-atlas.md
@@ -12,8 +12,6 @@ ms.date: 09/22/2022
 
 # TCGA Open Data
 
-[!INCLUDE [Open Dataset access change notice](./includes/open-datasets-change-note.md)]
-
 The Cancer Genome Atlas (TCGA), a landmark cancer genomics program, molecularly characterized over 20,000 primary cancer and matched normal samples spanning 33 cancer types[[1]](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga). The TCGA cancer data made available publically are two tiers: open or controlled access. 
 
 - Open access [available on Azure]: This dataset contains deindentified clinical and biospecimen data or summarized data that doesn't contain any individually identifiable information. The data types included are Gene expression, methylation beta values and protein quantification. DNA level datatype includes gene level copy number and masked copy number segment.
@@ -37,9 +35,6 @@ This dataset is stored in the East US 2 Azure regions. Allocating compute resour
 
 East US 2: 'https://datasettcga.blob.core.windows.net/dataset'
 
-[SAS Token](/azure/storage/common/storage-sas-overview): ?sp=rl&st=2022-10-07T19:43:37Z&se=2030-10-02T03:43:37Z&spr=https&sv=2021-06-08&sr=c&sig=9YgXjisOpHJNgdeMb5lOOzBhA38PWGM8g2DHjo9A5Cs%3D
-
-
 ## Use terms
 
 Data is available without restrictions. For more information and citation details, see the [TCGA Program page](https://www.cancer.gov/about-nci/organization/ccg/research/structural-genomics/tcga/using-tcga/citing-tcga)
