481 document new data ingestion model based on fhir documentation and generate sandbox data (#486)

* New data ingestion model documentation added

* sandbox generator examples added to data ingestion

* updating comparator values to rates

* sandbox generator added. preference and history read from ingestion data folder. config added for ingestion data folder. PractitionerRole.identifier used instead of PractitionerRole.practitioner.

Author: faridseifi

.env.local

@@ -1,9 +1,19 @@
-preferences=file:///path/to/knowledge-base/preferences.json 
+default_preferences=file:///path/to/knowledge-base/preferences.json 
 mpm=/path/to/knowledge-base/prioritization_algorithms/motivational_potential_model.csv
-manifest=file:////path/to/knowledge-base/mpog_local_manifest.yaml
-log_level=DEBUG
-generate_image=0
-cache_image=0
-outputs=0
-plot_goal_line=1
-display_window=6
+manifest=file:///path/to/knowledge-base/manifest.yaml
+
+log_level=INFO
+
+# defaults
+# meas_period=1
+# log_level=WARNING
+# generate_image=1
+# cache_image=0
+# outputs=0
+# performance_month=None
+# use_mi=1
+# use_preferences=1
+# use_history=1
+# use_coachiness=1
+# plot_goal_line=1
+# display_window=6

.env.remote

@@ -1,9 +1,12 @@
 # required knowledgebase paths
-mpm=https://raw.githubusercontent.com/Display-Lab/knowledge-base/1.7/prioritization_algorithms/motivational_potential_model.csv
-preferences=https://raw.githubusercontent.com/Display-Lab/knowledge-base/1.7/preferences.json
-manifest=https://raw.githubusercontent.com/Display-Lab/knowledge-base/refs/tags/1.7/mpog_manifest.yaml
+mpm=https://raw.githubusercontent.com/Display-Lab/knowledge-base-sandbox/1.0/prioritization_algorithms/motivational_potential_model.csv
+default_preferences=https://raw.githubusercontent.com/Display-Lab/knowledge-base-sandbox/1.0/preferences.json
+manifest=https://raw.githubusercontent.com/Display-Lab/knowledge-base-sandbox/refs/tags/1.0/manifest.yaml
+
+log_level=INFO
 
 # defaults
+# meas_period=1
 # log_level=WARNING
 # generate_image=1
 # cache_image=0

.gitignore

@@ -16,7 +16,7 @@ outputs/*
 *.DS_Store
 
 
-*.csv
+
 **/bin/__pycache__/
 python/.vscode/settings.json
 **/dist/

README.md

@@ -27,7 +27,7 @@ cd scaffold
 **Using `venv` and `pip`**
 
 ```zsh
-python --version # make sure you have python 3.11
+python --version # make sure you have python 3.12
 python -m venv .venv
 ```
 
@@ -53,7 +53,7 @@ pip install uvicorn # not installed by default (needed for running locally)
 **Alternative: Using [Poetry](https://python-poetry.org/) (for developers)**
 
 ```zsh
-poetry env use 3.11 # optional, but makes sure you have python 3.11 available
+poetry env use 3.12 # optional, but makes sure you have python 3.12 available
 poetry install # creates a virtual environment and install dependencies
 poetry shell # activates the enviroment
 ```
@@ -64,7 +64,7 @@ Clone the knowledge base repository in a separate location
 
 ```zsh
 cd ..
-git clone https://github.com/Display-Lab/knowledge-base.git 
+git clone https://github.com/Display-Lab/knowledge-base-sandbox.git 
 ```
 
 #### Running SCAFFOLD
@@ -75,10 +75,10 @@ Change back to the root of scaffold
 cd scaffold
 ```
 
-Update the `.env.local` file and change `path/to/knowledge-base` to point to the local knowledge base that you just checked out. (Don't remove the `file://` for default_preferences and manifest.)
+Create a copy of the `.env.local` file and call it `.env.dev` and update it by changing `path/to/knowledge-base` to point to the local knowledge base that you just checked out. (Don't remove the `file://` for default_preferences and manifest.)
 
 ```properties
-# .env.local
+# .env.dev
 default_preferences=file:///Users/bob/knowledge-base/preferences.json 
 mpm=/Users/bob/knowledge-base/prioritization_algorithms/motivational_potential_model.csv
 manifest=file:///Users/bob/knowledge-base/mpog_local_manifest.yaml
@@ -89,7 +89,7 @@ manifest=file:///Users/bob/knowledge-base/mpog_local_manifest.yaml
 There are two different ways to run SCAFFOLD API:
 1. Run SCAFFOLD API using uvicorn 
 ```zsh
-ENV_PATH=.env.local uvicorn scaffold.api:app
+ENV_PATH=.env.dev uvicorn scaffold.api:app
 # Expect to see a server start message like this "INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)"
 ```
 
@@ -118,24 +118,43 @@ ENV_PATH=/user/.../dev.env pipeline batch '/path/to/input/folder/' --max-files 5
 Use --max-files if you need to limit the number of files to process.
 
 ##### Run SCAFFOLD CLI with CSV inputs
-First install the python app. Then update the `.env.local` file and add links to history and preferences csv files along with other parameters mentioned earlier (manifest, default_preferences and mpm). 
+First install the python app. Then create the `.env.dev` file as mentioned above. 
 
 ```properties
-# .env.local
+# .env.dev
 default_preferences=file:///Users/bob/knowledge-base/preferences.json 
 mpm=/Users/bob/knowledge-base/prioritization_algorithms/motivational_potential_model.csv
 manifest=file:///Users/bob/knowledge-base/mpog_local_manifest.yaml
-preferences=/Users/bob/data/preferences.csv
-history=/Users/bob/data/history.csv
 ...
 ```
 Then use the following command to run the pipeline passing performance data csv file
 
 ```zsh
-ENV_PATH=/user/.../dev.env pipeline batch_csv '/path/to/performance/data/file.csv' --performance-month {performance month i.e. 2024-05-01} --max-files 500
+ENV_PATH=/user/.../dev.env python -m scaffold.cli batch-csv '/path/to/performance/data/folder' --performance-month {performance month i.e. 2025-05-01} --max-files 500
 ```
-Use --performance-month to set the performance month for batch_csv command and optional --max-files to limit the cases to process for development .
 
+Alternatively, you can use pip to install the pipeline command and use it to run the pipeline. Use the following command in the root of repository to install SCAFFOLF
+
+```zsh
+pip install .
+```
+
+Then you can use the following command to run the pipeline
+```zsh
+ENV_PATH=/user/.../dev.env pipeline batch-csv '/path/to/performance/data/folder' --performance-month {performance month i.e. 2025-05-01} --max-files 500
+```
+
+Alternatively, if you have poetry installed, you can run 
+```zsh
+poetry install
+```
+
+and then you shpuld be able to use the folloiwng command to run the pipeline:
+
+```zsh
+ENV_PATH=/user/.../dev.env pipeline batch_csv '/path/to/performance/data/folder' --performance-month {performance month i.e. 2025-05-01} --max-files 500
+```
+Use --performance-month to set the performance month for batch_csv command and optional --max-files to limit the cases to process for development.
 
 ## Environment variables
 
@@ -145,7 +164,7 @@ Local file path or URL (see .env.remote for github URL formats). All are require
 
 #### mpm: Path to the mpm csv file
 
-#### preferences: Path to the preferences json file
+#### default_preferences: Path to the default preferences json file
 
 #### manifest: Path to the manifest file that includes differend pieces of the base graph including (causal pathways, message templates, measures and comparators). See [manifest configuration](#manifest-configuration) for more detail
 

data ingestion model/README.md

@@ -0,0 +1,132 @@
+# SCAFFOLD [Data Ingestion Model](https://docs.google.com/spreadsheets/d/1qDjS2-a7F1El53jUx0fippL3m28pQilLcYNAv4pQkxI/edit?gid=1258033503#gid=1258033503)
+We adopted several [FHIR standard](https://hl7.org/fhir/index.html)'s resources to model data ingestion for SCAFFOLD. The input data include:
+- Provider information 
+- Performance data
+- Comparator data
+- Message history
+- Preference
+
+The data was structured using FHIR resources to the extent possible. Since no suitable resources exist for history and preferences, those were represented using our own format.
+
+## Data Structure
+### Provider information (`PractitionerRole`)
+The [`PractitionerRole`](https://build.fhir.org/practitionerrole.html) resource is used to represent message recipients(individuals or organizations), their relationships and their roles. The input data include a table (PractitionerRole.csv) with the following columns:
+- **[PractitionerRole.identifier](https://build.fhir.org/practitionerrole-definitions.html#PractitionerRole.identifier)**: Unique identifier for each row in the Practitiner table. This identifier links performance data, history and preferences to each recipient. In those datasets, PractitionerRole.identifier is refered to as subject.
+- **[PractitionerRole.practitioner](https://build.fhir.org/practitionerrole-definitions.html#PractitionerRole.practitioner)**: Contains the practitioner identifier. If this column has a value, the row represents an individual practitioner otherwise it is aggregate data for a group for example a hospital.
+- **[PractitionerRole.organization](https://build.fhir.org/practitionerrole-definitions.html#PractitionerRole.organization)**: Contains the identifier of the institution where the recipient serves. This field, together with `PractitionerRole.code` is used to identify the comparator data associated with each recipient. 
+
+- **[PractitionerRole.code](https://build.fhir.org/practitionerrole-definitions.html#PractitionerRole.code)**: Contains the role of the recipient in the institution. Example values for this field could be `Resident`, `Attending` or `CRNA`.
+- **type**: Indicates whether the performance data belong to an individual provider or to a group of providers. Accordingly, a `PractitionerRole` may represent either a single provider or a group. This field is not part of the FHIR `PractitionerRole` resource; in our model, it is introduced to classify `PractitionerRole` as either individual or group, allowing us to distinguish between the two types of performance data. Example values include `Practitioner` and `Organization`.
+  
+### Performance data (`MeasureReport`)
+Performance data are modeled using the [`MeasureReport`](https://build.fhir.org/measurereport.html) resource, which represents the results of a measure evaluation. In SCAFFOLD, each row of performance data is modled as a measure report. Accordingly, the input data include a table (PerformanceMeasureReport.csv) with the following columns:
+- **[identifier](https://build.fhir.org/measurereport-definitions.html#MeasureReport.identifier)**: Uniquely identifies a specific performance data record.
+- **[measure](https://build.fhir.org/measurereport-definitions.html#MeasureReport.measure)**: 	
+A reference to the measure with which the performance record is associated.
+- **[subject](https://hl7.org/fhir/measurereport-definitions.html#MeasureReport.subject)**: Contains the recipient's unique identifier.
+- **[period.start](https://hl7.org/fhir/datatypes-definitions.html#Period.start)**: The start date of the period for which the performance record was collected.
+- **[period.end](https://hl7.org/fhir/datatypes-definitions.html#Period.end)**: The end date of the period for which the performance record was collected.
+- **[measureScore](https://hl7.org/fhir/measurereport-definitions.html#MeasureReport.group.measureScore_x_).rate**: The calculated success rate for the performance record.
+- **[measureScore](https://hl7.org/fhir/measurereport-definitions.html#MeasureReport.group.measureScore_x_).denominator**: The total number of cases on which the performance record is based.
+- **[measureScore](https://hl7.org/fhir/measurereport-definitions.html#MeasureReport.group.measureScore_x_).range**: Used for categorical values.
+
+### Comparator data (`MeasureReport`):
+Comparator data , which represent aggregated performance for a selected group of recipients, are modeled using the [`MeasureReport`](https://build.fhir.org/measurereport.html) resource. In SCAFFOLD, each row of comparator data is modeled as a measure report. Accordingly, the input data include a table (ComparatorMeasureReport.csv) with the following columns:
+- **[identifier](https://build.fhir.org/measurereport-definitions.html#MeasureReport.identifier)**: Uniquely identifies a specific comparator data record.
+- **[measure](https://build.fhir.org/measurereport-definitions.html#MeasureReport.measure)**: A reference to the measure with which the comparator record is associated.
+- **[group.subject](https://hl7.org/fhir/measurereport-definitions.html#MeasureReport.subject)**: The identifier of the organization for which the comparator record is calculated. This column is equivalent to the `PractitionerRole.organization` column in the Provider Information data set.
+- **[period.start](https://hl7.org/fhir/datatypes-definitions.html#Period.start)**: The start date of the period for which the comparator record was collected.
+- **[period.end](https://hl7.org/fhir/datatypes-definitions.html#Period.end)**: The end date of the period for which the comparator record was collected.
+- **[measureScore](https://hl7.org/fhir/measurereport-definitions.html#MeasureReport.group.measureScore_x_).rate**: The average calculated success rate for the performance records of the selected group of providers.
+- **[measureScore](https://hl7.org/fhir/measurereport-definitions.html#MeasureReport.group.measureScore_x_).denominator**: The total number of cases on which the comparator record is based.
+- **[group.code](https://hl7.org/fhir/measurereport-definitions.html#MeasureReport.group.code)**: Specifies the type of comparator represented in each comparator record. Example values for this field include `peer average`, `Peer Top 10%` or `Goal Value`. 
+- **[PractitionerRole.code](https://build.fhir.org/practitionerrole-definitions.html#PractitionerRole.code)**: Indicates the role of the providers for whom the comparator record is calculated. Example values for this field include `Resident`, `Attending` or `CRNA`.
+
+### Message history
+Message history captures previously generated messages over defined time periods. SCAFFOLD's input data include a table (MessageHistory.csv) with the following columns:
+- **subject**: The provider (practitioner) identifier, to whom the message history record belongs.
+- **period.start**: The start date of the period for which the message was created.
+- **period.end**: The end date of the period for which the message was created.	
+- **history.json**: A JSON dictionary summarizing the generated message, with the following keys:
+    - **message_template**: The identifier of the message template used to generate the message.
+    - **message_template_name**: The name of the message template used to generate the message.
+    - **message_generated_datetime**: The date and time the message was generated.
+    - **measure**: The measure associated with the generated message.
+    - **acceptable_by**: The causal pathway associated with the generated message.
+
+### Preference
+Preferences captures providers' choices, priorities, and settings for messages that are generated for them. SCAFFOLD's input data include a table (Preferences.csv) with the following columns:
+- **subject**: The provider (practitioner) identifier, to whom the preferences record belongs.
+- **preferences.json**: A JSON dictionary with preferences detail. Here is an example of preferences JSON which SCAFFOLD can currently use 
+    ```json
+    {
+        "Utilities": {
+            "Message_Format": {  
+                "Social gain": "0.04", 
+                "Social stayed better":"-0.08",
+                "Worsening": "-0.1",  
+                "Improving": "-0.11", 
+                "Social loss": "0.69", 
+                "Social stayed worse": "-0.54", 
+                "Social better": "-1.23",
+                "Social worse": "0.54", 
+                "Social approach": "1.0",
+                "Goal gain": "0.07", 
+                "Goal approach": "1.1"
+            }, 
+            "Display_Format": {
+                "Bar chart": 1, 
+                "Line chart": 0, 
+                "Text-only": 0, 
+                "System-generated": 0
+            }
+        }
+    }
+    ```
+## Data Generator 
+First, create a folder for new data (i.e. `new_data`).
+If the data is going to be generated for individual recipients, create a config.json inside the new data folder containing
+```json
+{
+    "ComparatorMergeColumns":["group.subject", "PractitionerRole.code"]
+}
+```
+
+for hospital level data use
+```json
+{
+    "ComparatorMergeColumns":["PractitionerRole.code"]
+}
+```
+
+Now you can run the scripts sequentially to generate data.
+
+For example to generate hospital level data for 100 hospitals run the following commands:
+
+```zsh
+python data\ ingestion\ model/sandbox\ generator/PractitionerRole_hospital_level.py --num_orgs 100 --path new_data
+
+python data\ ingestion\ model/sandbox\ generator/PerformanceMeasureReport.py --path new_data   
+
+python data\ ingestion\ model/sandbox\ generator/ComparatorMeasureReport.py --path new_data
+
+python data\ ingestion\ model/sandbox\ generator/Preference.py --path new_data 
+
+ENV_PATH=/Path/to/your/environment/file/dev.env python data\ ingestion\ model/sandbox\ generator/MessageHistory.py --path new_data
+```
+
+This will start by creatig the list of hospitals in PractitionerRole.csv file. Then will generate performance data in PerformanceMeasureReports.csv. Next step will create the comparator data in ComparatorMeasureReport.csv. Then the preferences will be added to preferences.csv. Finall step will use SCAFFOLD to generate the history of messages generated by pipeline for the months before the performance month.
+
+## Example Data
+Sandbox hospital level example data is generated for 100 hospitals and included at 'sandbox examples' folder. This folder includes 
+- PractitionerRole.csv, which contains hospital definitions
+- PerformanceMeaasureReport, which contains performance data for each hospital on 12 defined measures in sandbox knowledge base for 12 month.
+- config.json, which is required to find the right comparator for each recipient
+- ComparatorMeasureReport.csv, which contains the comparator data for based on the entire network for each measure, for each month.
+- Preferences.csv, which includes preferences for a small subgroup of recipients.
+- MessageHistory.csv, which includes history of generated messages for 11 month before the performance month.
+
+# Run SCAFFOLD
+To run SCAFFOLD on sandbox data you need to prepare the environment and install SCAFFOLD. For more detail, follow the `Quick start` section of the [main SCAFFOLD documentation page](../README.md). Skip `Run SCAFFOLD API` and `Run SCAFFOLD CLI with JSON inputs` sections and continue with `Run SCAFFOLD CLI with CSV inputs`.
+
+