diff --git a/docs/release-notes/v1.60-v1.69.md b/docs/release-notes/v1.60-v1.69.md index 215e07c..c8dc70e 100644 --- a/docs/release-notes/v1.60-v1.69.md +++ b/docs/release-notes/v1.60-v1.69.md @@ -16,6 +16,142 @@ **Admins** - users who manage the organization in ODM, including overseeing users, groups, and permissions. They are also responsible for creating, defining, and editing metadata templates as needed. +## 1.62 + +### Single-Cell functionality + +* **New `Cell` entity**: ODM now supports storing and managing **per-cell metadata and expression** for single-cell datasets. +* **Cell Groups**: Each Cell record belongs to a **Cell Group**, representing a single-cell table/group. +* **Cell metadata import (TSV only)**: Import cell metadata via **Jobs API** (`/api/v1/jobs/import/cells` or `/api/v1/jobs/import/cells/multipart`) or the **`odm-import-data` script** (`--cells/-c`) and link to Samples/Libraries/Preparations. +* **Cell expression import**: Import expression via **Jobs API** (`/api/v1/jobs/import/expression` or `/api/v1/jobs/import/expression/multipart`); `.br` / `.lz4` archives recommended. Link expression to a Cell metadata group (1:1). +* **Validation**: Enforces required fields (e.g., `barcode`, `batch`) and rejects duplicate barcodes within a group; invalid types are ignored with warnings. +* **[BETA] Analytics**: New endpoints for **Cell Ratio**, **Gene Summary**, and **Differential Expression** calculations over filtered single-cell populations. +* **Deletion**: Remove Cell metadata or expression groups via the **manage-data/data** endpoint. + + Read more on **[Working with Single Cell Data page](../user-guide/doc-odm-user-guide/single-cell.md)**. + + ![Data Model](../assets/data_model.svg) + +### Multipart/form-data job endpoints + +We have added **multipart form-data upload endpoints** to the `jobs` import API, enabling **direct file uploads** (no `dataLink` required) for common ODM import flows: + +* `POST /api/v1/jobs/import/samples/multipart` — upload **Sample metadata** (TSV) +* `POST /api/v1/jobs/import/libraries/multipart` — upload **Library metadata** (TSV) +* `POST /api/v1/jobs/import/preparations/multipart` — upload **Preparation metadata** (TSV) +* `POST /api/v1/jobs/import/cells/multipart` — upload **Cell metadata** (TSV) +* `POST /api/v1/jobs/import/expression/multipart` — upload **tabular expression** data (TSV, GCT) +* `POST /api/v1/jobs/import/variant/multipart` — upload **variation data/metadata** (VCF, TSV) +* `POST /api/v1/jobs/import/flow-cytometry/multipart` — upload **flow cytometry data/metadata** (FACS, TSV) +* `POST /api/v1/jobs/import/file/multipart` — upload a **file attachment** via Jobs import + +### Attachments transformation + +ODM now supports an **attachment transformation workflow** that converts uploaded attachments into **ODM-indexable formats** and can automatically kick off the relevant **jobs import** flow. + +* **Transform attachments into supported formats** + + Use transformations to convert files ODM can’t ingest directly (e.g., **CSV**) into formats it can index (e.g., **TSV** for metadata imports). + +* **Image + configuration model** + + Each transformation uses: + + * a **transformation image** (script) defining how to convert input → output, and + * a **configuration** defining the **destination** (e.g., Samples, Libraries, Preparations, Cells, Expression, Variants). + +* **Discover available transformation images** + + New API to list available images via **Processors Controller**: + `GET /transformations/images`. + +* **Create and manage transformation configurations** + + Define reusable configs via: + `POST /transformations/configurations` (and retrieve via `GET /transformations/configurations`) + Example: config to route **csv → samples**. + +* **Run transformations as jobs** + + Trigger a transformation run using an **attachment accession**, **configuration**, and **image reference**: + `POST /transformations/jobs` + Track progress via: `GET /transformations/jobs/{id}`. + +* **Automatic import after conversion** + + On successful transformation, ODM **automatically uploads the converted file** and triggers the corresponding **Jobs import multipart endpoint** (e.g., CSV → TSV → `POST /api/v1/jobs/import/samples/multipart`), creating a new metadata group in the same Study where the original attachment was stored. + + Read more on **[Attachments transformation page](../user-guide/doc-odm-user-guide/attachment-transformation.md)**. + + ![AT Diagram](../user-guide/doc-odm-user-guide/doc-odm-user-guide/images/attachments-transformation-diagram.png) + +### API search by latest committed (published) data + +* **Updated search behavior across all endpoints**: Metadata search endpoints now query against the **latest committed metadata**, ensuring results reflect the most up-to-date committed changes. +* **Filters and queries remain supported**: All existing **filter and query parameters** continue to work as before, but are now applied to the **latest committed dataset**. + +### New Data Classes + +New Data classes were introduced: + +* Spatial transcriptomics `SPT` +* Phenomics `PHE` +* Copy number alterations `CNA` +* Microbiome / Metagenomics `MIC` +* Immune repertoire `IMR` +* Genetic screens (CRISPR / RNAi) `GSC` +* Cell imaging `CIM` +* Nanopore data class was renamed to Long-read sequencing (Nanopore, PacBio) `LRS` +* `MTX` label for Metabolomics was renamed to `MTB` + +### Samples + +* Starting with this release, creating a Study no longer automatically creates four empty Sample entries. Instead, use the **Add Samples** button to upload a Samples metadata file. + +![Add Samples](../user-guide/doc-odm-user-guide/doc-odm-user-guide/images/add-samples-button.png) + +* Sample deletion is no longer available via the GUI. Instead, a new option is available to delete a single Sample object via the **manageData** endpoint. + +![Single Sample deletion](../user-guide/doc-odm-user-guide/doc-odm-user-guide/images/sample-object-deletion.png) + +### Templates + +Starting with this release, the Template Editor includes a new feature to view the list of Studies that use the template you’re viewing. In the new Study Browser window, you’ll see the list of Studies available to you. + +![Template applied](../user-guide/doc-odm-user-guide/doc-odm-user-guide/images/template-applied-to-studies.png) + +### Preview PDF attachments + +* **Preview PDFs directly from ME**: Attachments with filenames ending in **`.pdf`** can now be previewed from the attachment list. +* **One-click preview**: A **Preview** action/button is available for PDF attachments. +* **Opens in a new browser tab**: Clicking **Preview** opens the file in a **new tab** using the browser’s built-in PDF viewer. + +![PDF Preview](../user-guide/doc-odm-user-guide/doc-odm-user-guide/images/pdf-preview.png) + +### Permanent tokens + +* **Instant token display**: When creating an ODM personal access token from the Profile page, the **token value is now shown immediately** after the user confirms creation (and enters their password, if prompted). +* **Email step removed**: Token creation no longer requires an email with a secure link/code exchange. + +![New token](../user-guide/doc-odm-user-guide/doc-odm-user-guide/images/token.png) + +### Access (Bearer) tokens + +1. The user identifier for Azure token authentication was changed from `subject` to `oid`, enabling login without defining an Azure app scope. +2. As a result, all users registered in ODM via Azure SSO will need to log in via the UI once again to have their records updated in the database. + +### Metadata validity facet + +A new default facet has been added to filter Studies by **valid** or **invalid** metadata, validated against the applied template. Metadata is considered **invalid** if at least one field is invalid in any Study entity. + +![Validity Facet](../user-guide/doc-odm-user-guide/doc-odm-user-guide/images/validity-facet.png) + +### Key security improvements + +1. Removing the email-based token delivery flow to eliminate a potential attack vector. +2. Spring Boot upgrade to v4 with updated transitive dependencies resolved plenty of security vulnerabilities. +3. Three vulnerabilities (SQL injection, path traversal, log injection) identified and fixed by code analysis tools. + ## 1.61 Release 1.61 introduces powerful enhancements focused on improving data governance and preventing common user errors. diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/add-samples-button.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/add-samples-button.png new file mode 100644 index 0000000..a7d17dc Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/add-samples-button.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/pdf-preview.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/pdf-preview.png new file mode 100644 index 0000000..3c45818 Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/pdf-preview.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/sample-object-deletion.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/sample-object-deletion.png new file mode 100644 index 0000000..faf55c5 Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/sample-object-deletion.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/template-applied-to-studies.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/template-applied-to-studies.png new file mode 100644 index 0000000..b3a59c4 Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/template-applied-to-studies.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/token.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/token.png new file mode 100644 index 0000000..c9c207d Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/token.png differ diff --git a/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/validity-facet.png b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/validity-facet.png new file mode 100644 index 0000000..d12e2bb Binary files /dev/null and b/docs/user-guide/doc-odm-user-guide/doc-odm-user-guide/images/validity-facet.png differ diff --git a/docs/user-guide/doc-odm-user-guide/single-cell.md b/docs/user-guide/doc-odm-user-guide/single-cell.md index 722121a..07a0cbc 100644 --- a/docs/user-guide/doc-odm-user-guide/single-cell.md +++ b/docs/user-guide/doc-odm-user-guide/single-cell.md @@ -93,7 +93,7 @@ Learn more about [uploading data to ODM via API here](../doc-odm-user-guide/impo ### Uploading via script -Curators can upload and link Cell metadata groups to ODM using the [import_ODM_data script](../../tools/odm-sdk/terminal/study/uploading-study.md). +Curators can upload and link Cell metadata groups to ODM using the [odm_import_data script](../../tools/odm-sdk/terminal/study/uploading-study.md). This extension allows you to include Cell groups in the same import workflow as other metadata entities (Studies, Samples, Libraries, and Preparations), ensuring a consistent and automated data-loading process.