adding variant check progress bar

Author: pnrobinson

book/src/help/img/ageParse.png

@@ -5,17 +5,72 @@ Some articles present information about groups (cohorts) of individuals in table
 Then, each column is processed by right clicking on the column header or as needed on individual cells.
 
 <figure>
-  <img src="img/external-age.png" alt="Cohort editor" width="600">
+  <img src="img/table-editor.png" alt="Cohort editor" width="600">
   <figcaption>
-    <strong>External table editor</strong>. Here, the user has right-clicked on the table header and is transforming the contents to Age entries.
+    <strong>External table editor</strong>. Users should work through the columns one by one. Successfully transformed columns are shown in green. The follow sections explain how to use the tool.
   </figcaption>
 </figure>
 
 
-## Functionality
+## Excel import 
+Clinical data about cohorts is often provided in tabular form in the medical genetics literature. Often, supplemental tables with clinical data about the cohort are provided as Excel files.  Use the *Excel (Cols)* button to import Excel files in which 
+data for each individual is represented by a column. Use *Excel (Rows)* button to import Excel files in which 
+data for each individual is represented by a row. 
 
-The functions of phenoboard can be explored by right-clicking on column headers or cells. 
+> **Warning**  
+> Some excel files encountered in the literature can be difficult to parse. For instance, sometimes there are formatting errors such that the contents of one cell "spills over" into the next row. Sometimes, information about two different items
+is presented in the same cell (e.g., Age and Sex). Phenoboard provides functionality to merge and split cells that may help to 
+deal with this, but it may be easier to manually edit the excel file in some cases.
 
-## Saving
+The Excel import buttons are disactivated if there is no current cohort. Thus, if you are trying to create a cohort from scratch based on an Excel file, you first need to create the cohort data using the [New Cohort](newcohort.md) page.
 
-When all columns have been processed, the user can add all rows to the current cohort (which must be previously entered!).
+## Saving and Loading ETL files
+
+If you would like to save your work and come back later to finish, use the *Save ETL* button to save the file that contains the current state of the ETL file. Use the *Load ETL* button to load the file and resume work where you left off.
+
+> **ETL**: Extract transform load
+> ETL is a three-phase computing process where data is extracted from an input source, transformed (including cleaning), and loaded into an output data container. In our case, we are extracting data from the original Excel file, transforming it into HPO terms and other data required for the phenopacket, and loading the transformed data into the Cohort data structure that phenoboard uses to store data about cohorts of individuals (phenopackets).
+
+## PMID
+
+Assign the PubMed identifier of the article from which the cohort data was taken.
+
+## Add to cohort
+
+When you are finished transforming each column, the next step is to load the data into the current cohort. This is not
+possible unless all columns have been transformed or marked as "Ignored" (except for the HPO Text Mining column, which is optional).
+
+## Transforming columns
+Phenoboard offers a number of different functions for transforming columns, each of which can be started from the context menu that appears upon right click. The following sections describe the major functionalities. Many of the dialogs also provide help that can be accessed by clicking on the **?** symbol.
+
+### Demographic information
+Each individual can be annotated with an individual identifier, sex, age of onset, age at last encounter, deceased status. Optionally, a column with a family identifier can be marked and merged with the individual identifier column. To do this, right click on the column header and navegate the context menu as shown below.
+
+<figure>
+  <img src="img/ageParse.png" alt="Ingesting age" width="600">
+  <figcaption>
+    <strong>Importing age entries</strong>. 
+  </figcaption>
+</figure>
+
+
+### Individual ID
+This entry must be unique in the cohort. For instance, it would be an error to have two rows with the identifier "Individual A". 
+
+### Sex column
+This column is used to specify the biological sex of the individual. A variety of formats are transformed into the phenopacket standard entries:
+- **M**: Male
+- **F**: Female
+- **O**: Other
+- **U**: Unknown (or not recorded, not available)
+
+### Age of onset/Age at last encounter
+Enter the age at first manifestation of any clinical manifestation related to the disease (Onset) and the age at which the individual was last medically examined (last encounter). The ingest will try to transform the data in the column into a Gestational age, HPO term, or ISO 8601 string as appropriate. It is common to see that the age is provide in years (just the number without "y" or "years"). In this case, use the "Assume years" option to ingest the data.
+
+### Deceased
+The data in this column must be in the phenopacket format
+- **yes**: deceased
+- **no**: alive
+- **na**: unknown/not available
+
+Further information about these fields can be found in the [Phenopacket Schema documentation](https://phenopacket-schema.readthedocs.io/en/latest/).

book/src/help/img/external-age.png

@@ -1,6 +1,6 @@
 {
   "name": "phenoboard",
-  "version": "0.5.99",
+  "version": "0.5.100",
   "scripts": {
     "ng": "ng",
     "start": "nx serve phenoboard --port 1420",
@@ -24,16 +24,16 @@
     "@angular/platform-browser": "^20.0.6",
     "@angular/platform-browser-dynamic": "^20.0.6",
     "@angular/router": "^20.0.6",
-    "@tauri-apps/api": "^2.5.0",
-    "@tauri-apps/plugin-dialog": "^2.4.0",
-    "@tauri-apps/plugin-fs": "^2.4.2",
-    "@tauri-apps/plugin-opener": "2.5.0",
+    "@tauri-apps/api": "^2.10.1",
+    "@tauri-apps/plugin-dialog": "^2.6.0",
+    "@tauri-apps/plugin-fs": "^2.4.5",
+    "@tauri-apps/plugin-opener": "2.5.3",
     "material-icons": "^1.13.14",
     "rxjs": "~7.8.0",
     "tslib": "^2.3.0",
     "tw-elements": "^2.0.0",
     "xlsx": "^0.18.5",
-    "zone.js": "^0.15.1"
+    "zone.js": "^0.16.0"
   },
   "devDependencies": {
     "@angular-devkit/build-angular": "^20.0.5",

book/src/help/img/table-editor.png

@@ -1,6 +1,6 @@
 [package]
 name = "phenoboard"
-version = "0.5.99"
+version = "0.5.100"
 description = "Curate cohorts of GA4GH Phenopackets"
 authors = ["Peter N Robinson"]
 edition = "2021"
@@ -22,10 +22,10 @@ dirs = "6.0"
 html-escape = "0.2.13"
 fuzzy-matcher = "0.3"
 ontolius = "0.7.2" 
-reqwest = { version = "0.13.1", features = ["json", "blocking"] }
+reqwest = { version = "0.13.2", features = ["json", "blocking"] }
 rfd = { version = "0.16.0", default-features = false}
 fenominal = { git = 'https://github.com/P2GX/fenominal.git',  tag = '0.1.18' }
-ga4ghphetools = { git = 'https://github.com/P2GX/ga4ghphetools.git', tag = '0.5.21' }
+ga4ghphetools = { git = 'https://github.com/P2GX/ga4ghphetools.git', tag = '0.5.23' }
 #ga4ghphetools = { path = "../../ga4ghphetools" }
 serde = { version = "1.0.219", features = ["derive"] }
 serde_json = "1.0.148"

book/src/help/table-editor.md

@@ -9,9 +9,9 @@ use ga4ghphetools::{dto::{cohort_dto::{CohortData, CohortType, DiseaseData, Indi
 use ga4ghphetools::dto::intergenic_variant::IntergenicHgvsVariant;
 use ontolius::ontology::MetadataAware;
 use phenoboard::PhenoboardSingleton;
-use tauri::{AppHandle, Emitter, Manager, WindowEvent};
+use tauri::{AppHandle, Emitter, Manager, Window, WindowEvent};
 use tauri_plugin_dialog::{DialogExt};
-use std::{collections::HashMap, fs, sync::{Arc, Mutex}};
+use std::{collections::{HashMap, HashSet}, fs, sync::{Arc, Mutex}};
 use tauri_plugin_fs::{init};
 
 
@@ -737,15 +737,39 @@ async fn get_variant_analysis(
     singleton.get_variant_analysis(cohort_dto)
 }
 
+
+#[derive(Clone, serde::Serialize)]
+struct ProgressPayload {
+    current: u32,
+    total: u32,
+}
+
+/// Check all alleles in an ETL column and emit signals to show progress.
 #[tauri::command]
-fn process_allele_column(
+async fn process_allele_column(
     state: tauri::State<'_, Arc<AppState>>,
+    window: Window,
     etl: EtlDto,
     col: usize
 ) -> Result<EtlDto, String> {
-    let singleton = state.phenoboard.lock()
-        .map_err(|_| "Failed to acquire lock on HPO State".to_string())?;
-    singleton.process_allele_column(etl, col)
+    let app_handle = state.inner().clone();
+    if col >= etl.table.columns.len() {
+        return Err(format!("Attempt to access invalid column {} for table with {} columns", col, etl.table.columns.len()));
+    }
+    // Move work to background task so we can still send emits to front-end!
+    tokio::task::spawn_blocking(move || {
+        let singleton = app_handle.phenoboard.lock()
+            .map_err(|_| "Failed to acquire lock".to_string())?;
+        let total_alleles = etl.table.columns[col].values.len() as u32;
+        let pb = |current: u32, q: u32| {
+            let _ = window.emit("progress-update", ProgressPayload { 
+                current, 
+                total: total_alleles 
+            });
+        };
+
+    singleton.process_allele_column(etl, col, pb)
+    }).await.map_err(|e| e.to_string())?
 }
 
 /// This command creates a CohortData object from the current EtlDto and should

package.json

@@ -636,7 +636,7 @@ impl PhenoboardSingleton {
     }
 
     pub fn save_biocurator_orcid(&mut self, orcid: String) -> Result<StatusDto, String> {
-        self.settings.save_biocurator_orcid(orcid);
+        self.settings.save_biocurator_orcid(orcid)?;
         Ok(self.get_status())
     }
 
@@ -647,13 +647,15 @@ impl PhenoboardSingleton {
         ga4ghphetools::variant::analyze_variants(cohort_dto)
     }
 
-    pub fn process_allele_column(
+    pub fn process_allele_column<F>(
         &self,
         etl: EtlDto,
-        col: usize
-    ) -> Result<EtlDto, String> {
+        col: usize,
+        progress_cb: F
+    ) -> Result<EtlDto, String> where F: FnMut(u32, u32) {
+   
         match &self.ontology {
-            Some(hpo) =>  ga4ghphetools::etl::process_allele_column(hpo.clone(),etl, col),
+            Some(hpo) =>  ga4ghphetools::etl::process_allele_column(hpo.clone(),etl, col, progress_cb),
             None => Err("HPO not initialized".to_string()),
         }
 

src-tauri/Cargo.toml

@@ -1,7 +1,7 @@
 {
   "$schema": "https://schema.tauri.app/config/2",
   "productName": "phenoboard",
-  "version": "0.5.99",
+  "version": "0.5.100",
   "identifier": "org.p2gx.phenoboard",
   "build": {
     "beforeDevCommand": "npx nx serve phenoboard --configuration=development  --no-cloud",

src-tauri/src/lib.rs

@@ -17,10 +17,13 @@ export class AppStatusService {
   // Derived UI states
   readonly hpoLoading = signal<boolean>(false);
   readonly hpoLoaded = computed(() => this.state().hpoLoaded);
+  progress = signal<number>(0);
+  
 
   constructor() {
     this.init();
     this.setupListeners();
+    this.listen_alleles();
   }
 
   private async init() {
@@ -32,6 +35,17 @@ export class AppStatusService {
     }
   }
 
+  private async listen_alleles() {
+    await listen("progress-update", (event) => {
+      const {current, total} = event.payload as {current: number, total: number};
+      const percent = Math.round((current / total) * 100);
+      this.ngZone.run(() => {
+        this.progress.set(percent)
+      });
+       ;
+    });
+  }
+
   private async setupListeners() {
     await listen("hpo-load-event", (event) => {
       const { status, message, data } = event.payload as {