add a tutlial for oregon health insurance experiment #81

okiner-3 · 2025-10-14T07:56:35Z

Add a tutorial with Oregon Health Insurance Experiment.

TomeHirata · 2025-10-16T08:09:05Z

@okiner-3 Thank you so much for the PR! Sorry if it was not clear in the ticket description, but we want to focus on the local distributional treatment effects (LDTE) and LPTE since incomplete compliance was observed in the experiment. In the Oregon dataset, the treatment column represents the treatment assignment, the numhh_list column is the strata, and the ohp_all_ever_inperson column indicates the actual treatment received. We can keep covariates and outcomes as they are. Could you please take a look at https://cyberagentailab.github.io/python-dte-adjustment/api/local.html and revise the content? It is valuable to compare ITT and LDTE, so we can keep the current analysis as it is, but let's also include the LDTE/LPTE results. Let me know if you need further clarification.

okiner-3 · 2025-10-19T06:08:54Z

@TomeHirata
I understood. I will take care of it!

… to Non-Compliance

Copilot

Pull Request Overview

This PR adds a comprehensive tutorial demonstrating the use of Local Distribution Treatment Effects (LDTE) analysis with the Oregon Health Insurance Experiment dataset. The tutorial showcases how to handle non-compliance scenarios using instrumental variable approaches when not all participants assigned to treatment actually enrolled.

Key changes:

New comprehensive tutorial file analyzing emergency department costs and visits using local distribution treatment effects
Tutorial demonstrates both simple and ML-adjusted estimators for handling non-compliance
Includes stratified analysis by household registration patterns to examine treatment effect heterogeneity

Reviewed Changes

Copilot reviewed 2 out of 11 changed files in this pull request and generated 4 comments.

File	Description
docs/source/tutorials/oregon.rst	Comprehensive tutorial implementing LDTE analysis for the Oregon Health Insurance Experiment with non-compliance handling
docs/source/tutorials.rst	Added reference to the new Oregon tutorial in the documentation index

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

docs/source/tutorials/oregon.rst

TomeHirata · 2025-10-21T11:31:40Z

docs/source/tutorials/oregon.rst

+   :width: 800px
+   :align: center
+
+**LDTE Interpretation**: The positive LDTE values indicate that Medicaid assignment increases the cumulative probability of individuals having emergency department costs at or below each threshold among compliers (those who enroll when selected). This suggests that while Medicaid increases overall ED utilization, it may also help contain costs for some individuals who actually enroll.


Are the LDTE values really positive?

TomeHirata · 2025-10-21T11:32:27Z

docs/source/tutorials/oregon.rst

+
+**LDTE Interpretation**: The positive LDTE values indicate that Medicaid assignment increases the cumulative probability of individuals having emergency department costs at or below each threshold among compliers (those who enroll when selected). This suggests that while Medicaid increases overall ED utilization, it may also help contain costs for some individuals who actually enroll.
+
+**Statistical Significance**: Both simple and ML-adjusted local estimators show similar patterns, providing robust evidence that Medicaid assignment has significant distributional effects on emergency department costs for compliers. The confidence intervals indicate that these effects are statistically significant across most cost levels.


The confidence intervals indicate that these effects are statistically significant across most cost levels.

Is this correct?

TomeHirata · 2025-10-21T11:33:42Z

docs/source/tutorials/oregon.rst

+
+The Local Probability Treatment Effects analysis produces the following visualization:
+
+.. image:: ../_static/oregon_lpte_costs_comparison.png


Maybe we can increase the size of each bin. Currently there are too many bins in the graph.

TomeHirata · 2025-10-21T11:34:51Z

docs/source/tutorials/oregon.rst

+
+The side-by-side bar charts show probability treatment effects across different emergency department cost intervals, revealing how Medicaid enrollment affects healthcare utilization patterns:
+
+**Cost Distribution Effects**: The LPTE analysis shows how Medicaid assignment changes the probability of compliers incurring emergency department costs in specific ranges. Positive bars indicate cost intervals where Medicaid assignment increases the likelihood of incurring costs in that range, while negative bars show intervals where it decreases the probability.


This is a correct explanation about how to understand LPTE, but what's the insight specifically for this data?

TomeHirata · 2025-10-21T11:35:21Z

docs/source/tutorials/oregon.rst

+
+**Cost Distribution Effects**: The LPTE analysis shows how Medicaid assignment changes the probability of compliers incurring emergency department costs in specific ranges. Positive bars indicate cost intervals where Medicaid assignment increases the likelihood of incurring costs in that range, while negative bars show intervals where it decreases the probability.
+
+**Healthcare Utilization Patterns**: Both simple and ML-adjusted local estimators reveal consistent patterns in how Medicaid assignment affects emergency department utilization across different cost categories for compliers. The analysis shows that Medicaid assignment has heterogeneous effects, increasing utilization in some cost ranges while potentially reducing it in others.


ditto, can we make it a bit more detailed?

docs/source/tutorials/oregon.rst

TomeHirata · 2025-10-21T11:46:25Z

docs/source/tutorials/oregon.rst

+
+The emergency department visits analysis reveals complementary patterns to the cost analysis:
+
+**Visit Frequency Effects**: Medicaid assignment shows distinct effects on the probability of different visit frequencies for compliers. The LPTE analysis reveals which visit count categories are most affected by Medicaid assignment among those who actually enroll.


Can we make it a bit more detailed?

TomeHirata · 2025-10-21T11:51:51Z

docs/source/tutorials/oregon.rst

+
+**Policy Targeting Implications**: Understanding which household types respond most strongly to Medicaid assignment can inform more targeted policy interventions and help identify populations that would benefit most from expanded coverage when they actually enroll.
+
+**Methodological Consistency**: Both simple and ML-adjusted local estimators show similar patterns within each stratum, providing confidence in the robustness of the stratified findings across different analytical approaches while accounting for non-compliance.


Shall we also mention the difference in ci length?

TomeHirata · 2025-10-21T11:51:57Z

docs/source/tutorials/oregon.rst

+
+**Heterogeneity Analysis**: The stratified analysis by household registration type reveals important local treatment effect heterogeneity, showing that different populations respond differently to Medicaid assignment when they actually enroll.
+
+**Methodological Robustness**: Comparing simple and ML-adjusted local estimators provides confidence in our findings and demonstrates the robustness of the local distributional treatment effect methodology for handling non-compliance scenarios.


TomeHirata · 2025-10-21T11:53:14Z

docs/source/tutorials/oregon.rst

+    plt.tight_layout()
+    plt.show()
+
+.. image:: ../_static/oregon_ldte_strata.png


It seems the ml adjusted LPTE has something wrong, can we investigate the cause?

TomeHirata · 2025-10-21T11:56:38Z

docs/source/tutorials/oregon.rst

+**Methodological Consistency**: Both simple and ML-adjusted local estimators show similar patterns within each stratum, providing confidence in the robustness of the stratified findings across different analytical approaches while accounting for non-compliance.
+
+Conclusion
+~~~~~~~~~~


Let me repeat the same comment, can we ground the insights on the Oregon analysis results? The current summary feels too generic.

TomeHirata · 2025-12-02T07:16:40Z

Hi, @okiner-3. How's the progress so far? Feel free to let me know if you have any questions!

okiner-3 · 2025-12-21T12:11:15Z

@TomeHirata
I'm sorry it took so long.
The general and abstract considerations have been grounded in the current data and results.
Additionally, duplicate sections and code that were accidentally included have been removed.

TomeHirata · 2025-12-30T01:39:27Z

docs/source/tutorials/oregon.rst

+
+**2. Covariate Adjustment Effects and Confidence Intervals**
+
+The confidence intervals remain wide for both estimators, though ML adjustment shows slightly more consistent patterns in the moderate cost range. The limited precision suggests: (1) substantial heterogeneity in treatment effects within cost bins, (2) limited predictive power of covariates for specific cost levels, or (3) relatively small sample sizes within individual bins.


Did we observe no precision gain?

TomeHirata · 2025-12-30T03:37:57Z

docs/source/tutorials/oregon.rst

+- **"Signed self up" stratum**: Confidence intervals remain wide but manageable for both estimators, showing similar patterns to the overall population.
+- **"Signed self up + others" stratum**:
+
+  - Extreme estimation instability, particularly for ML-adjusted estimator


The ML estimation is a bit odd, have you tried using different ml models or fold number?

TomeHirata · 2025-12-30T03:56:12Z

docs/source/tutorials/oregon.rst

+
+Stratified analysis uncovers dramatic treatment effect heterogeneity: single-person households ("signed self up") show moderate effects (LDTE ≈ -0.18 to -0.20), while multi-person households ("signed self up + others") exhibit 3-4x larger effects (LDTE ≈ -0.55). This suggests household structure is a critical moderator—insurance enables care-seeking for multiple family members when households include dependents.
+
+**4. Limited Efficiency Gains from ML Adjustment**


We should figure out the way to increase the efficiency gain instead of listing its difficulty here.

add a tutlial for oregon health insurance experiment

26b5af5

okiner-3 self-assigned this Oct 14, 2025

okiner-3 requested a review from TomeHirata October 14, 2025 08:00

replace functions from predict_{dte, pte} to predict_{ldte, lpte} due…

39b2acd

… to Non-Compliance

TomeHirata requested a review from Copilot October 21, 2025 11:16

Copilot AI reviewed Oct 21, 2025

View reviewed changes

docs/source/tutorials/oregon.rst Show resolved Hide resolved

docs/source/tutorials/oregon.rst Outdated Show resolved Hide resolved

docs/source/tutorials/oregon.rst Outdated Show resolved Hide resolved

docs/source/tutorials/oregon.rst Outdated Show resolved Hide resolved