Updates E2E Testing Documentation for local development by ndegwamartin · Pull Request #1542 · google/fhir-data-pipes

ndegwamartin · 2025-12-18T22:32:46Z

Description of what I changed

Resolves #1534

E2E test

TESTED:

Please replace this with a description of how you tested your PR beyond the
automated e2e/unit tests.

Checklist: I completed these to help reviewers :)

I have read and will follow the
review process.
I am familiar with Google Style Guides for the language I have coded in.

No? Please take some time and review
Java and
Python style guides.
My IDE is configured to follow the Google
code styles.

No? Unsure? ->
configure your IDE.
I have added tests to cover my changes. (If you refactored existing
code that was well tested you do not have to add tests)
I ran mvn clean package right before creating this pull request and
added all formatting changes to my commit.
If I made any Python code changes, I ran black ., pylint . and
pyright . right before creating this pull request and added all
formatting changes to my commit.
All new and existing tests passed.
My pull request is based on the latest changes of the master branch.

No? Unsure? -> execute command git pull --rebase upstream master

codecov-commenter · 2025-12-18T22:47:23Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 46.61%. Comparing base (8434457) to head (588c32d).

Additional details and impacted files

@@            Coverage Diff            @@
##             master    #1542   +/-   ##
=========================================
  Coverage     46.61%   46.61%           
  Complexity      678      678           
=========================================
  Files            90       90           
  Lines          5886     5886           
  Branches        834      834           
=========================================
  Hits           2744     2744           
  Misses         2827     2827           
  Partials        315      315

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

bashir2

Thanks for creating this useful doc @ndegwamartin .

doc/run-cloud-build-locally.md

bashir2 · 2025-12-19T19:08:19Z

doc/debug-cloud-build-locally.md

+  -e OUTPUT_PARQUET_VIEW_PATH=/workspace/e2e-tests/FHIR_SEARCH_HAPI/VIEWS_TIMESTAMP_1 \
+  -e SINK_PATH=http://localhost:9001/fhir \
+  -e SINK_USERNAME=hapi -e SINK_PASSWORD=hapi \
+  -v $(pwd):/workspace \


Consider adding a note about output directories under this path that might be overwritten.

I think this would mainly be the path under the OUTPUT_PARQUET_VIEW_PATH env, correct? I've added a note.

I think in this case, it is PARQUET_PATH too, see here.

bashir2 · 2025-12-19T19:09:47Z

doc/run-cloud-build-locally.md

+    ${_REPOSITORY}/e2e-tests:${_TAG}
+```
+
+or just the script directly:


Again, since this is a local environment, I would recommend to people to use the non-docker version as running a local script is easier to debug. That said, I would also make a note that in the Cloud Build env this is done through docker (and keep the sample docker run).

This is correct, the direct script run is documented as an alternative immediately after this.

Updated the documentation to reflect the suggestion above.

As a side note, my original understanding was that we needed to fully replicate the previously documented cloud build workflow run via:

cloud-build-local --dryrun=false .

Should we also consider renaming the file(and instructions title) from run-cloud-build-locally.md ?

This document that you have created is actually better than cloud-build-local --dryrun=false . because even if we could use cloud-build-local we still needed to replicate the steps locally to be able to debug (which is what this doc is). Please feel free if you want to use a better name for this, e.g., debug-cloud-build-locally.md.

doc/run-cloud-build-locally.md

doc/debug-cloud-build-locally.md

doc/run-cloud-build-locally.md

bashir2 · 2025-12-23T17:52:40Z

doc/run-cloud-build-locally.md

+### 4. Bring up controller and Spark containers
+
+```bash
+PIPELINE_CONFIG=/workspace/docker/config DWH_ROOT=/workspace/e2e-tests/controller-spark/dwh docker compose -f docker/compose-controller-spark-sql-single.yaml up --force-recreate -d


Don't you need a volume mapping for /workspace?
nit: Also please break the long line (didn't our default formatter catch this?).

The bind mount is defined in the compose config file here -

fhir-data-pipes/docker/compose-controller-spark-sql-single.yaml

Lines 60 to 62 in 547612d

volumes:

- ${PIPELINE_CONFIG}:/app/config:ro

- ${DWH_ROOT}:/dwh

Don't think our default formatter(Spotless) can reformat these commands as appears in the .md files. We probably need to break these down manually.

doc/run-cloud-build-locally.md

doc/debug-cloud-build-locally.md

doc/run-cloud-build-locally.md

bashir2 · 2025-12-23T18:18:38Z

doc/run-cloud-build-locally.md

+    ${_REPOSITORY}/e2e-tests:${_TAG}
+```
+
+or just the script directly:


This document that you have created is actually better than cloud-build-local --dryrun=false . because even if we could use cloud-build-local we still needed to replicate the steps locally to be able to debug (which is what this doc is). Please feel free if you want to use a better name for this, e.g., debug-cloud-build-locally.md.

bashir2 · 2025-12-23T18:21:06Z

doc/debug-cloud-build-locally.md

+  -e OUTPUT_PARQUET_VIEW_PATH=/workspace/e2e-tests/FHIR_SEARCH_HAPI/VIEWS_TIMESTAMP_1 \
+  -e SINK_PATH=http://localhost:9001/fhir \
+  -e SINK_USERNAME=hapi -e SINK_PASSWORD=hapi \
+  -v $(pwd):/workspace \


I think in this case, it is PARQUET_PATH too, see here.

ndegwamartin requested a review from bashir2 December 18, 2025 22:32

ndegwamartin force-pushed the e2e-local-tests branch from 571b635 to e84d99d Compare December 18, 2025 22:34

Updates E2E Testing Documentation for local development

5dcd8eb

ndegwamartin force-pushed the e2e-local-tests branch from e84d99d to 5dcd8eb Compare December 18, 2025 22:39

Merge branch 'master' into e2e-local-tests

7c07bc3

bashir2 reviewed Dec 19, 2025

View reviewed changes

ndegwamartin added 2 commits December 22, 2025 23:03

Address PR Feedback 1

4ef1510

Adds additional instructions to run locally instructions

fbf61cf

bashir2 reviewed Dec 23, 2025

View reviewed changes

ndegwamartin and others added 3 commits December 30, 2025 23:30

Addresses PR Feedback

3636b81

Rename the cloud build documentation file

a98e520

Merge branch 'master' into e2e-local-tests

588c32d

	volumes:
	- ${PIPELINE_CONFIG}:/app/config:ro
	- ${DWH_ROOT}:/dwh

Comments

Conversation

ndegwamartin commented Dec 18, 2025

Description of what I changed

E2E test

Checklist: I completed these to help reviewers :)

Uh oh!

codecov-commenter commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bashir2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov-commenter commented Dec 18, 2025 •

edited

Loading