Learnings from using the Synthea data generator for use with ETL-Synthea

[anthonysena]

I faced a few challenges that I was able to overcome using Synthea which I'll contribute back to the README. For now, I'm adding them to this issue in case others are facing similar issues.

Synthea Setup & Usage

• After cloning the Synthea repo (synthetichealth/synthea@47d09bf), you need to edit src/main/resources/synthea.properties to update the exporter settings to make sure that CSVs are produced as these are the input into the ETL-Synthea process. During my work, I had to change the following settings:

exporter.fhir.export = false
exporter.fhir.transaction_bundle = false
exporter.hospital.fhir.export = false
exporter.practitioner.fhir.export = false
exporter.csv.export = true

I don't think there is any harm in leaving the fhir settings to true except that your producing extra data that you won't use with ETL-Synthea.

• Make sure you have the JAVA_HOME environmental variable set. I did not have this set and when running ./gradlew build check test I ran into some Java problems.
• Once the steps above are done, to produce a 1,000 person data set you can run run_synthea.bat -p 1000 which then creates the CSVs in the /output/csv folder in the Synthea directory. Note this location as it is needed for the ETL-Synthea processing.
• I faced an issue with using the CSVs from the latest Synthea builder as noted in Sync ETL-Synthea with latest master #43 and hope to have these incorporated soon.

Running ETL-Synthea

• Download the vocabulary files from Athena: http://athena.ohdsi.org/vocabulary/list. You'll need to make sure these files are unzipped and available as CSVs.
• You'll need to establish an empty database with 2 schemas: cdm will hold the OMOP CDM v5.3.1 tables and native will hold the native Synthea data.
• ETL-Synthea has functions to create your CDM & Vocabulary tables and load the Vocabulary tables in the cdm schema mentioned above. Alternatively, you can establish your CDM and Vocabulary tables using the v5.3.1 scripts on https://github.com/OHDSI/CommonDataModel.
• Once my CDM & Vocabulary tables were set and the vocabulary was loaded, I opted to create indicies on the vocabulary tables only using the script on https://github.com/OHDSI/CommonDataModel. The idea was that since the ETL process will use these tables, it would be best if they had some indices to improve performance.
• I used the R package approach as described in the README: https://github.com/OHDSI/ETL-Synthea#step-by-step-example-r-package.

Post-processing Synthea

• After the ETL ran successfully, I applied the indices using the v5.3.1 script on https://github.com/OHDSI/CommonDataModel.
• I also attempted to apply the foreign key constraints using the v5.3.1 script on https://github.com/OHDSI/CommonDataModel but ran into a problem noted in Cannot apply foreign keys to CDM after ETL #42.

[ablack3]

I ran into problems using the ETLSyntheaBuilder::LoadVocabFromCsv() function to load the vocab table. I was able to use the script here

However I ran into a "could not stat file" error when uploading the concept_relationship table which is the largest vocab table. I found a solution at https://stackoverflow.com/questions/53523051/error-could-not-stat-file-xx-csv-unknown-error

I think this is a windows issue. I'm using Windows 10 with postgres 12. The solution was
COPY cdm_synthea10.CONCEPT_RELATIONSHIP FROM PROGRAM 'cmd /c "type C:\CDMV5VOCAB\CONCEPT_RELATIONSHIP.csv"' WITH DELIMITER E'\t' CSV HEADER QUOTE E'\b' ;

The ETL seemed to run fine after that!

You can also generate the csv files from synthea using
java -jar synthea-with-dependencies.jar --exporter.csv.export true