script added to convert tabular data to json. Example input json files added to sandbox. Minor updates for scaffold to work with new json input files. (#493)

Author: faridseifi

data ingestion model/README.md

@@ -11,15 +11,15 @@ The data was structured using FHIR resources to the extent possible. Since no su
 ## Performance Data Structure
 ### Provider information (`PractitionerRole`)
 The [`PractitionerRole`](https://build.fhir.org/practitionerrole.html) resource is used to represent message recipients(individuals or organizations), their relationships and their roles. The input data include a table (PractitionerRole.csv) with the following columns:
-- **[PractitionerRole.identifier](https://build.fhir.org/practitionerrole-definitions.html#PractitionerRole.identifier)**: Unique identifier for each row in the Practitiner table. This identifier links performance data, history and preferences to each recipient. In those datasets, PractitionerRole.identifier is refered to as subject.
+- **[PractitionerRole.identifier](https://build.fhir.org/practitionerrole-definitions.html#PractitionerRole.identifier)**: Unique identifier for each row in the Practitiner table. This identifier links performance data, history and preferences to each recipient. In those datasets, PractitionerRole.identifier is referred to as subject.
 - **[PractitionerRole.practitioner](https://build.fhir.org/practitionerrole-definitions.html#PractitionerRole.practitioner)**: Contains the practitioner identifier. If this column has a value, the row represents an individual practitioner otherwise it is aggregate data for a group for example a hospital.
 - **[PractitionerRole.organization](https://build.fhir.org/practitionerrole-definitions.html#PractitionerRole.organization)**: Contains the identifier of the institution where the recipient serves. This field, together with `PractitionerRole.code` is used to identify the comparator data associated with each recipient. 
 
 - **[PractitionerRole.code](https://build.fhir.org/practitionerrole-definitions.html#PractitionerRole.code)**: Contains the role of the recipient in the institution. Example values for this field could be `Resident`, `Attending` or `CRNA`.
 - **type**: Indicates whether the performance data belong to an individual provider or to a group of providers. Accordingly, a `PractitionerRole` may represent either a single provider or a group. This field is not part of the FHIR `PractitionerRole` resource; in our model, it is introduced to classify `PractitionerRole` as either individual or group, allowing us to distinguish between the two types of performance data. Example values include `Practitioner` and `Organization`.
 
 ### Performance data (`MeasureReport`)
-Performance data are modeled using the [`MeasureReport`](https://build.fhir.org/measurereport.html) resource, which represents the results of a measure evaluation. In SCAFFOLD, each row of performance data is modled as a measure report. Accordingly, the input data include a table (PerformanceMeasureReport.csv) with the following columns:
+Performance data are modeled using the [`MeasureReport`](https://build.fhir.org/measurereport.html) resource, which represents the results of a measure evaluation. In SCAFFOLD, each row of performance data is modeled as a measure report. Accordingly, the input data include a table (PerformanceMeasureReport.csv) with the following columns:
 - **[identifier](https://build.fhir.org/measurereport-definitions.html#MeasureReport.identifier)**: Uniquely identifies a specific performance data record.
 - **[measure](https://build.fhir.org/measurereport-definitions.html#MeasureReport.measure)**: 	
 A reference to the measure with which the performance record is associated.
@@ -86,6 +86,8 @@ Preferences captures providers' choices, priorities, and settings for messages t
     }
     ```
 ## Data Generator 
+
+### Generate Tabular data
 First, create a folder for new data (i.e. `new_data`).
 If the data is going to be generated for individual recipients, create a config.json inside the new data folder containing
 ```json
@@ -117,25 +119,100 @@ python data\ ingestion\ model/sandbox\ generator/Preference.py --path new_data
 ENV_PATH=/Path/to/your/environment/file/dev.env python data\ ingestion\ model/sandbox\ generator/MessageHistory.py --path new_data
 ```
 
-This will start by creatig the list of hospitals in PractitionerRole.csv file. Then will generate performance data in PerformanceMeasureReports.csv. Next step will create the comparator data in ComparatorMeasureReport.csv. Then the preferences will be added to preferences.csv. Finall step will use SCAFFOLD to generate the history of messages generated by pipeline for the months before the performance month.
+This will start by creating the list of hospitals in PractitionerRole.csv file. Then will generate performance data in PerformanceMeasureReports.csv. Next step will create the comparator data in ComparatorMeasureReport.csv. Then the preferences will be added to preferences.csv. Final step will use SCAFFOLD to generate the history of messages generated by pipeline for the months before the performance month.
 
 This process will start by creating a list of hospitals in the `PractitionerRole.csv` file. Next, performance data will be generated in `PerformanceMeasureReports.csv`. The following step will create comparator data in `ComparatorMeasureReport.csv`. Preferences will then be added to `preferences.csv`. Finally, SCAFFOLD will be used to generate the history of messages produced by the pipeline for the months preceding the performance month and store it in `MessageHistory.csv`.
 
+### Convert Data To JSON-LD Inputs
+You can use the script in TabularToJson.py to convert data in tabular format to json-ld input files. Use the following to run this script on a path where the tabular data with all the required files that follow the SCAFFOLD data ingestion model exists to generate json-ld inputs files
+
+```
+python data\ ingestion\ model/sandbox\ generator/TabularToJSON.py --path new_data --performance_month 2025-01-01
+```
+
 ## Example Data
-Sandbox hospital-level example data is generated for 100 hospitals and included in the sandbox examples folder. This folder includes:
+Sandbox hospital-level example data is generated for 100 hospitals and included in the sandbox examples folder. This folder includes both tabular data and json-ld input files for same hospitals. 
+
+
+### Tabular data
+Tabular input data includes:
 
-### Performance Data
+#### Performance Data
 - PractitionerRole.csv, which contains hospital definitions
 - PerformanceMeaasureReport, which contains performance data for each hospital on 12 defined measures in sandbox knowledge base for 12 month.
 - config.json, which is required to find the right comparator for each recipient
 - ComparatorMeasureReport.csv, which contains the comparator data based on the entire network for each measure, for each month.
 
-### Prioritization Data
+#### Prioritization Data
 - Preferences.csv, which includes preferences for a small subgroup of recipients.
 - MessageHistory.csv, which includes history of generated messages for 11 month before the performance month.
 
+### JSON-LD Input files
+The `JSON Messages` folder contains json-ld input files for the same data. Each file is created using the following template. See examples for more detail.
+
+```json
+{
+    "@context": {
+        "dcterms": "http://purl.org/dc/terms/",
+        "schema": "http://schema.org/",
+        "scaffold": "http://displaylab.com/scaffold#",
+        "psdo": "http://purl.obolibrary.org/obo/",
+        "slowmo": "http://example.com/slowmo#",
+        "message_template": {"@id": "psdo:PSDO_0000002"},
+        "measure": {"@id": "psdo:PSDO_0000102"},
+        "performance_summary_document": {"@id": "psdo:PSDO_0000098"},
+        "performance_month": {"@id": "scaffold:performance_month"},
+        "History": {"@id": "scaffold:History"},
+        "Preferences": {"@id": "scaffold:Preferences"},
+        "performance_measure_report": {"@id": "psdo:PSDO_0000107"},
+        "comparator_measure_report": {"@id": "scaffold:comparator_measure_report"},
+        "PractitionerRole": {"@id": "scaffold:PractitionerRole"},
+        "subject": {"@id": "scaffold:subject"},
+    },
+    "message_instance_id": "",
+    "performance_month": "2025-01-01",
+    "@type": "psdo:performance_summary_document",
+    "subject": "",
+    "PractitionerRole": [
+        [
+            "PractitionerRole.identifier",
+            "PractitionerRole.practitioner",
+            "PractitionerRole.organization",
+            "PractitionerRole.code",
+        ],
+    ],
+    "performance_measure_report": [
+        [
+            "identifier",
+            "measure",
+            "subject",
+            "period.start",
+            "period.end",
+            "measureScore.rate",
+            "measureScore.denominator",
+            "measureScore.range",
+        ],
+    ],
+    "comparator_measure_report": [
+        [
+            "identifier",
+            "measure",
+            "period.start",
+            "period.end",
+            "measureScore.rate",
+            "measureScore.denominator",
+            "group.subject",
+            "group.code",
+            "PractitionerRole.code",
+        ],
+    ],
+    "History": [],
+    "Preferences": {},
+}
+```
+
 ## Run SCAFFOLD
-To run SCAFFOLD on sandbox data you need to prepare the environment and install SCAFFOLD. For more detail, follow the `Quick start` section of the [main SCAFFOLD documentation page](../README.md). Skip `Run SCAFFOLD API` and `Run SCAFFOLD CLI with JSON inputs` sections and continue with `Run SCAFFOLD CLI with CSV inputs`.
+To run SCAFFOLD on sandbox data you need to prepare the environment and install SCAFFOLD. For more detail, follow the `Quick start` section of the [main SCAFFOLD documentation page](../README.md). You can process JSON input files using `Run SCAFFOLD API` or `Run SCAFFOLD CLI with JSON inputs` sections. Use `Run SCAFFOLD CLI with CSV inputs` section to process tabular data.
 
 ## Expected Output
 Here is an example of the output from SCAFFOLD after processing the sandbox example data:
@@ -206,4 +283,4 @@ Successful: 100, Failed: 0
 | Quiet-Rating-01        | 361         | 9.7           | 2.25             | 12        | 12.0        | 2.84           | 3.3                       |
 | Transfer-01            | 387         | 10.4          | 2.22             | 13        | 13.0        | 2.92           | 3.4                       |
 
-SCAFFOLD also creates a `messages` folder, which contains a summary of the generated candidates (`candidates.csv`) and a detailed JSON file for each generated message. Each JSON file includes information about the selected message, all created candidates with their scoreing details, and any generated images.
+SCAFFOLD also creates a `messages` folder, which contains a summary of the generated candidates (`candidates.csv`) and a detailed JSON file for each generated message. Each JSON file includes information about the selected message, all created candidates with their scoring details, and any generated images.