Skip to content

Experiment: Test Combining #46

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
korourke5 wants to merge 8 commits into from
Closed

Experiment: Test Combining #46

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
dc8875e
Update parameters.json to add blood test parameters
korourke5 Sep 8, 2025
80177d8
Edit parameters.json and prepare.py to include
korourke5 Sep 8, 2025
e04997e
Added the scenarios created and updated readme
korourke5 Sep 8, 2025
69d2ec0
Merge remote-tracking branch 'origin/experiment-
korourke5 Sep 12, 2025
4192820
Updated prepare.py and parameters.json to include
korourke5 Sep 12, 2025
4af3868
Added results in summary, updated the S3 bin
korourke5 Oct 23, 2025
bda348f
Update README.md with results.
korourke5 Oct 23, 2025
6135dda
Delete crcsim/experiment/summary/~$summarized.xlsx
korourke5 Oct 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions crcsim/experiment/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

This branch is a template for `crcsim` experiments. It can be used as a starting point for new experiments.

## Experiment Goal

Test the combination of FIT and blood tests in parallel and serial.

## Experiment Results

FIT + blood in parallel leads to less CRC and more diagnostic tests (ie, positive screenings) than either test alone. FIT + blood in serial leads to more CRC and fewer diagnostic tests than either test alone.

## Experiment Workflow

`crcsim` experiments are managed as branches. We never merge experiment branches into main. Instead, each experiment branch has all relevant files in the `crcsim/experiment` directory, and is kept as a separate branch for historical record.
Expand Down Expand Up @@ -66,7 +74,7 @@ The subdirectories and files in `scenarios/` must be uploaded to AWS S3 for the

To upload the files to S3, run
```
aws s3 cp ./scenarios s3://crcsim-exp-template/scenarios --recursive
aws s3 cp ./scenarios s3://crcsim-exp-test-combining/scenarios --recursive
```
*(Another note: this manual step is necessary because `boto3` does not include functionality to upload a directory to S3 recursively. Future experiments could improve this workflow by writing a function to upload the directory recursively in `prepare.py`. Or submit a patch to resolve https://github.com/boto/boto3/issues/358)*

Expand Down Expand Up @@ -142,4 +150,4 @@ AWS Batch automatically sends log streams from jobs to AWS CloudWatch. Some logg

### ECR

Pushing the Docker image to ECR is the only step of the architecture setup that was NOT completed via the AWS Console. The script `deploy_to_aws.sh` contains all commands necessary to build the `crcsim` Docker image and upload it to ECR.
Pushing the Docker image to ECR is the only step of the architecture setup that was NOT completed via the AWS Console. The script `deploy_to_aws.sh` contains all commands necessary to build the `crcsim` Docker image and upload it to ECR.
53 changes: 51 additions & 2 deletions crcsim/experiment/parameters.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,9 +100,57 @@
"cost_perforation": 6487,
"compliance_rate_given_prev_compliant": [ 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0 ],
"compliance_rate_given_not_prev_compliant": [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ]
},
"Blood": {
"proportion": 0.0,
"routine_start": 45,
"routine_end": 75,
"routine_freq": 3,
"specificity": 0.90,
"sensitivity_polyp1": 0,
"sensitivity_polyp2": 0,
"sensitivity_polyp3": 0.13,
"sensitivity_cancer": 0.83,
"cost": 895,
"proportion_perforation": 0,
"cost_perforation": 0,
"compliance_rate_given_prev_compliant": [ 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0 ],
"compliance_rate_given_not_prev_compliant": [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ]
},
"FIT_Blood_parallel": {
"sensitivity_polyp1": 0.07000000000000006,
"sensitivity_polyp2": 0.07000000000000006,
"sensitivity_polyp3": 0.3214,
"sensitivity_cancer": 0.9558,
"specificity": 0.873,
"cost": 922,
"proportion": 0.0,
"routine_start": 45,
"routine_end": 75,
"routine_freq": 1,
"proportion_perforation": 0,
"cost_perforation": 0,
"compliance_rate_given_prev_compliant": [ 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0 ],
"compliance_rate_given_not_prev_compliant": [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ]
},
"FIT_Blood_serial": {
"sensitivity_polyp1": 0.0,
"sensitivity_polyp2": 0.0,
"sensitivity_polyp3": 0.0286,
"sensitivity_cancer": 0.6142,
"specificity": 0.997,
"cost": 922,
"proportion": 0.0,
"routine_start": 45,
"routine_end": 75,
"routine_freq": 1,
"proportion_perforation": 0,
"cost_perforation": 0,
"compliance_rate_given_prev_compliant": [ 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0 ],
"compliance_rate_given_not_prev_compliant": [ 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 ]
}
},
"routine_tests": [ "FIT", "Colonoscopy" ],
"routine_tests": [ "FIT", "Colonoscopy", "Blood", "FIT_Blood_parallel", "FIT_Blood_serial" ],
"diagnostic_test": "Colonoscopy",
"surveillance_test": "Colonoscopy",
"polypectomy_proportion_lethal": 0.00002,
Expand All @@ -122,4 +170,5 @@
"death_rate_white_male_rates": [ 0.005273, 0.000389, 0.000273, 0.000199, 0.000147, 0.000143, 0.000126, 0.000112, 0.000100, 0.000091, 0.000090, 0.000105, 0.000145, 0.000215, 0.000309, 0.000412, 0.000520, 0.000642, 0.000778, 0.000921, 0.001068, 0.001209, 0.001330, 0.001424, 0.001497, 0.001561, 0.001624, 0.001682, 0.001737, 0.001792, 0.001847, 0.001900, 0.001952, 0.002003, 0.002053, 0.002111, 0.002174, 0.002233, 0.002285, 0.002340, 0.002413, 0.002516, 0.002649, 0.002811, 0.002999, 0.003203, 0.003433, 0.003709, 0.004047, 0.004445, 0.004874, 0.005331, 0.005844, 0.006408, 0.007003, 0.007607, 0.008219, 0.008857, 0.009542, 0.010285, 0.011098, 0.011952, 0.012814, 0.013657, 0.014502, 0.015384, 0.016444, 0.017624, 0.018968, 0.020586, 0.022109, 0.024359, 0.026347, 0.028810, 0.031309, 0.034486, 0.038026, 0.042286, 0.046547, 0.051534, 0.057008, 0.062923, 0.069911, 0.078099, 0.086754, 0.096549, 0.106472, 0.119677, 0.134128, 0.149846, 0.166829, 0.185047, 0.204441, 0.224919, 0.246354, 0.268590, 0.291442, 0.314700, 0.338142, 0.361537, 1.000000 ],

"us_age_distribution_rates": [ 0.01237, 0.01236, 0.01234, 0.01231, 0.01208, 0.01214, 0.01213, 0.01209, 0.01211, 0.01225, 0.01225, 0.01225, 0.01264, 0.01270, 0.01260, 0.01257, 0.01262, 0.01257, 0.01257, 0.01293, 0.01318, 0.01304, 0.01310, 0.01319, 0.01332, 0.01362, 0.01389, 0.01409, 0.01443, 0.01466, 0.01460, 0.01401, 0.01366, 0.01340, 0.01342, 0.01348, 0.01304, 0.01320, 0.01317, 0.01301, 0.01323, 0.01241, 0.01214, 0.01200, 0.01164, 0.01197, 0.01159, 0.01174, 0.01224, 0.01290, 0.01303, 0.01232, 0.01204, 0.01203, 0.01223, 0.01294, 0.01315, 0.01311, 0.01306, 0.01319, 0.01326, 0.01278, 0.01266, 0.01248, 0.01198, 0.01183, 0.01129, 0.01081, 0.01033, 0.00995, 0.00965, 0.00930, 0.00912, 0.00941, 0.00689, 0.00671, 0.00645, 0.00649, 0.00557, 0.00499, 0.00462, 0.00424, 0.00395, 0.00352, 0.00324, 0.00298, 0.00255, 0.00234, 0.00211, 0.00188, 0.00168, 0.00140, 0.00120, 0.00099, 0.00079, 0.00064, 0.00049, 0.00036, 0.00027, 0.00019, 0.00028 ]
}

}
41 changes: 41 additions & 0 deletions crcsim/experiment/prepare.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,12 +128,53 @@ def create_scenarios() -> List:
# transform test proportions for the FIT scenarios.
.transform(transform_routine_test_proportion("Colonoscopy", 1.0))
.transform(transform_routine_test_proportion("FIT", 0.0))
.transform(transform_routine_test_proportion("Blood", 0.0))
.transform(transform_routine_test_proportion("FIT_Blood_parallel", 0.0))
.transform(transform_routine_test_proportion("FIT_Blood_serial", 0.0))
.transform(transform_lesion_risk_alpha(1.19))
)

scenarios.append(
Scenario(name=f"FIT_{scenario}", params=get_default_params())
.transform(transform_initial_compliance(screening_rate))
.transform(transform_routine_test_proportion("Colonoscopy", 0.0))
.transform(transform_routine_test_proportion("FIT", 1.0))
.transform(transform_routine_test_proportion("Blood", 0.0))
.transform(transform_routine_test_proportion("FIT_Blood_parallel", 0.0))
.transform(transform_routine_test_proportion("FIT_Blood_serial", 0.0))
.transform(transform_lesion_risk_alpha(1.19))
)

scenarios.append(
Scenario(name=f"Blood_{scenario}", params=get_default_params())
.transform(transform_initial_compliance(screening_rate))
.transform(transform_routine_test_proportion("Colonoscopy", 0.0))
.transform(transform_routine_test_proportion("FIT", 0.0))
.transform(transform_routine_test_proportion("Blood", 1.0))
.transform(transform_routine_test_proportion("FIT_Blood_parallel", 0.0))
.transform(transform_routine_test_proportion("FIT_Blood_serial", 0.0))
.transform(transform_lesion_risk_alpha(1.19))
)

scenarios.append(
Scenario(name=f"FIT_Blood_parallel_{scenario}", params=get_default_params())
.transform(transform_initial_compliance(screening_rate))
.transform(transform_routine_test_proportion("Colonoscopy", 0.0))
.transform(transform_routine_test_proportion("FIT", 0.0))
.transform(transform_routine_test_proportion("Blood", 0.0))
.transform(transform_routine_test_proportion("FIT_Blood_parallel", 1.0))
.transform(transform_routine_test_proportion("FIT_Blood_serial", 0.0))
.transform(transform_lesion_risk_alpha(1.19))
)

scenarios.append(
Scenario(name=f"FIT_Blood_serial_{scenario}", params=get_default_params())
.transform(transform_initial_compliance(screening_rate))
.transform(transform_routine_test_proportion("Colonoscopy", 0.0))
.transform(transform_routine_test_proportion("FIT", 0.0))
.transform(transform_routine_test_proportion("Blood", 0.0))
.transform(transform_routine_test_proportion("FIT_Blood_parallel", 0.0))
.transform(transform_routine_test_proportion("FIT_Blood_serial", 1.0))
.transform(transform_lesion_risk_alpha(1.19))
)

Expand Down
4 changes: 2 additions & 2 deletions crcsim/experiment/run_iteration.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ if [ ! -d "$output_dir" ]; then
mkdir $output_dir
fi

aws s3 cp "s3://crcsim-exp-template/scenarios/$scenario/params.json" "./params.json"
aws s3 cp "s3://crcsim-exp-test-combining/scenarios/$scenario/params.json" "./params.json"

crc-simulate \
--npeople=$npeople \
Expand All @@ -23,4 +23,4 @@ crc-simulate \
crc-analyze \
--params-file=./params.json &&

aws s3 cp ./results.csv "s3://crcsim-exp-demog-specific-survival/scenarios/$scenario/results_$iteration.csv"
aws s3 cp ./results.csv "s3://crcsim-exp-test-combining/scenarios/$scenario/results_$iteration.csv"
Loading