Improve partition support for Bulk Patch operation #7406

jamesagnew · 2025-11-25T10:46:46Z

This PR adds an explicit partitionId parameter to the bulk patch operation, and improves general support for running this batch job in a partitioned environment.

…date

* bump to 8.7, add version enums * changelog folder for 8.8 * increase memory to elastic search container --------- Co-authored-by: jdar <[email protected]>

* [7332] update mdm resource persistence id * [7332] add tests and changelog * [7332] update changelog * [7332] update test * [7332] update test * [7332] update changelog

* increase memory to elastic search container (#7326) Co-authored-by: jdar <[email protected]> * 7282 - Added the ability for an Interceptor to alter the HTTP Response Code when an Exception is thrown (#7303) * 7282 - Added the ability for an Interceptor to alter the HTTP Response Code when an Exception is thrown. * Enhanced the solution based on code review feedback. * Removed a class that was being used to store the Http response Code and used the existing HttpStatus class instead. * Enhanced the Unit Test. * Enhanced the Javadoc for the Pointcut.SERVER_OUTGOING_FAILURE_OPERATIONOUTCOME enum. * Changed the Interceptor Hook implementation based on Code Review feedback. * Removed an unused variable declaration. * Removed an unused variable declaration. * Ran mvn spotless:apply to apply proper formatting. * More code review feedback. * Use ResponseDetails for exception outcome modification. * Remove bit about "valid status" -- we allow anything * changelog * Replace the response object --------- Co-authored-by: Michael Buckley <[email protected]> * batch2 jobs that are cancelled will not be returned when requesting a… (#7321) (#7327) * Improve URL processing for JpaPersistedResourceValidationSupport (#7330) * Improve URL processing for JpaPersistedResourceValidationSupport * Add changelog * allow fetching subsequent pages when registering an allowed spec of r… (#7329) * allow fetching subsequent pages when registering an allowed spec of resourcetype search * 7833: add missing relationship between TermConcept and TermConceptProperty (#7346) * 7316 do not expand chain refparams (#7350) * Make sure refparams with chains are not expanded via MDM * 7316: code review feedback * [7269] bulk export history fails with client-assigned string IDs (#7300) * fixed 7269 bulk export history failure with client-assigned string IDs * fixed 7269 bulk export history failure with client-assigned string IDs * Update hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/changelog/8_6_0/7296-bulk-export-fails-with-string-resource-ids.yaml Co-authored-by: jdar8 <[email protected]> * changed string ids to IResourcePersistentId * changed string ids to IResourcePersistentId * batch2 jobs that are cancelled will not be returned when requesting a… (#7321) * Bump to 8 7 (#7322) * bump to 8.7, add version enums * changelog folder for 8.8 * increase memory to elastic search container --------- Co-authored-by: jdar <[email protected]> * Update hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/dao/index/IdHelperService.java Co-authored-by: Michael Buckley <[email protected]> * fixed with spotless * Revert "Bump to 8 7 (#7322)" This reverts commit 40f1128. * Revert "changed string ids to IResourcePersistentId" This reverts commit 06af86a. * Revert "batch2 jobs that are cancelled will not be returned when requesting a… (#7321)" This reverts commit be08680. * changed string ids to IResourcePersistentId * fixed with spotless * Bumped version number * adding missing change to fix build --------- Co-authored-by: Rob Caruso <[email protected]> Co-authored-by: jdar8 <[email protected]> Co-authored-by: TipzCM <[email protected]> Co-authored-by: jdar <[email protected]> Co-authored-by: Michael Buckley <[email protected]> * [7332] fix mdm-query-links (#7356) * 7324 unable to access collection bundles when performing search requests (#7325) * Add failing tests * Fix * Changelog * Fix comment * Fix test * Try fixing CodeQL autobuild error * Implement revision suggestions * Add missed test case parameter * PatientIdPartitionIntercepto should allow Encounter.subject parameter (#7348) * Work on megascale enhancements part 6 * Add changelog * Test fix * Update hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/changelog/8_6_0/7348-allow-patientidcompartmentmode-search-encounter-subject.yaml Co-authored-by: Nathan Doef <[email protected]> * Test fixes --------- Co-authored-by: Nathan Doef <[email protected]> * Bump to spring62 (#7351) * Bump spring framework version * Replace Base64Utils with Base64.getEncoder() * Bump spring boot test deps to match spring version 6.2 * Bump junit for spring tests --------- Co-authored-by: Gary <[email protected]> * 7217 with rel 8 6 (#7337) * Automated Migration Testing (HAPI-FHIR) - updated test migration scripts for 8_6_0 (#7357) * Remove dead autowire (#7365) * Don't reuse TransactionDetails for partitioned TX execution (#7363) * Fix tags on megascale * Add test * Test fix * Version bump --------- Co-authored-by: jdar <[email protected]> Co-authored-by: Kevin Dougan <[email protected]> Co-authored-by: Michael Buckley <[email protected]> Co-authored-by: TipzCM <[email protected]> Co-authored-by: James Agnew <[email protected]> Co-authored-by: another-luis <[email protected]> Co-authored-by: Rob Caruso <[email protected]> Co-authored-by: Rob Caruso <[email protected]> Co-authored-by: Yaling Pei <[email protected]> Co-authored-by: jmarchionatto <[email protected]> Co-authored-by: Nathan Doef <[email protected]> Co-authored-by: tadgh <[email protected]> Co-authored-by: Gary <[email protected]> Co-authored-by: volodymyr-korzh <[email protected]> Co-authored-by: Gary Graham <[email protected]>

* Ensure streaming queries don't impose limit clauses on the sql when called in "synchronous" mode. * changelog

Co-authored-by: Elie Maamari <[email protected]>

…7392) * Fix logic in Icd10CmLoader to append 7th character extensions to the leaf-level concepts in the hierarchy (rather than to just the immediate child concepts). * Fix logic in Icd10CmLoader to append 7th character extensions to the leaf-level concepts in the hierarchy (rather than to just the immediate child concepts). * Fix logic in Icd10CmLoader to append 7th character extensions to the leaf-level concepts in the hierarchy (rather than to just the immediate child concepts). --------- Co-authored-by: Ian Marshall <[email protected]>

…#7389) * Inferno test parsing failure when 401 rejection does not contain OperationOutcome - failing test * Inferno test parsing failure when 401 rejection does not contain OperationOutcome - implementation * Inferno test parsing failure when 401 rejection does not contain OperationOutcome - changelog

* backport complete * Be more resilient to unexpected resource version IDs in database (#7216) * Handle mismatched versions * Be more resilient to unexpected resource versions in DB * Test fix * Test fix * Work on tests * Test fixes * Test fixes * Add changelog * Spotless * Address review comments * HAPI bump * Fix API breakages * api breakages * version set * changelog and versioning --------- Co-authored-by: James Agnew <[email protected]> Co-authored-by: Gary Graham <[email protected]> Co-authored-by: jdar <[email protected]>

) * STORAGE_TRANSACTION_PROCESSING-doesn't-pass-ServletRequestDetails * Add changelog --------- Co-authored-by: James Agnew <[email protected]>

…7388) * Fix Encounter $everything not working in REQUEST_TENANT partitioning * Add test * changelog * Remove unused HttpServletRequest parameter * Remove unnecessary overriding

…date

* bump to 8.7, add version enums * changelog folder for 8.8 * increase memory to elastic search container --------- Co-authored-by: jdar <[email protected]>

* [7332] update mdm resource persistence id * [7332] add tests and changelog * [7332] update changelog * [7332] update test * [7332] update test * [7332] update changelog

…b.com:hapifhir/hapi-fhir into ja_20251106_megascale_bulk_terminology_update

robogary · 2025-11-25T10:48:06Z

Formatting check succeeded!

michaelabuckley · 2025-11-25T22:34:46Z

hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/cache/ResourceVersionSvcDaoImpl.java

 	}

+	@Override
+	public ResourceVersionMap getVersionMap(String theResourceName, SearchParameterMap theSearchParamMap) {


question: why add this method? Is there a reason the callers can't provide the parititon scope?

michaelabuckley · 2025-11-25T22:35:48Z

hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/batch2/Batch2DaoSvcImpl.java

 				.withRequest(theRequestDetails)
-				.search(() -> builder.createQueryStream(
+				.withRequestPartitionId(theRequestPartitionId)
+				.search(partition -> builder.createQueryStream(


nice change! Passing partition through makes shard search much more practical.

michaelabuckley · 2025-11-25T22:39:53Z

...in/java/ca/uhn/fhir/rest/server/interceptor/partition/RequestTenantPartitionInterceptor.java

 					return requestDetails.getRequestPartitionId();
 				}
-				return RequestPartitionId.defaultPartition(myPartitionSettings);
+				return RequestPartitionId.allPartitions();


comment: this is a big change. Is this safe?

michaelabuckley · 2025-11-25T22:40:59Z

...fhir-jpaserver-test-r4/src/test/java/ca/uhn/fhir/jpa/dao/r4/FhirResourceDaoR4CreateTest.java

-			final String matchUrl = "identifier=http://tempuri.org|1";
-			bundleBuilder.addTransactionCreateEntry(myTask1, "urn:uuid:59cda086-4763-4ef0-8e36-8c90058686ea")
-				.conditional(matchUrl);
+			Callable<Bundle> bundleSupplier = () -> {


trivial: Supplier.get() is clearer than Callable.call().

michaelabuckley · 2025-11-25T22:42:29Z

hapi-fhir-jpaserver-test-r4/src/test/java/ca/uhn/fhir/jpa/dao/r4/PartitioningSqlR4Test.java

+		assertThat(toUnqualifiedVersionlessIdValues(result)).containsExactly("Patient/A");
+
+		myCaptureQueriesListener.logSelectQueriesForCurrentThread();
+		String searchSql = myCaptureQueriesListener.getSelectQueries().get(0).getSql(true, true);


suggestion: add a positive assertion to make sure you got the right query.

michaelabuckley · 2025-11-25T22:49:06Z

hapi-fhir-storage/src/main/java/ca/uhn/fhir/jpa/partition/BaseRequestPartitionHelperSvc.java

 			return RequestPartitionId.allPartitions();
 		}

+		RequestPartitionId existingPartitionId =


question: why do we need this? Maybe add a comment explaining.

michaelabuckley · 2025-11-25T22:50:45Z

...-fhir-storage-batch2/src/main/java/ca/uhn/fhir/batch2/jobs/step/GenerateRangeChunksStep.java

+				RequestPartitionId requestPartitionId = partitionedUrl.getRequestPartitionId();
+				requestPartitionId = getIfNull(requestPartitionId, RequestPartitionId.allPartitions());
+
+				List<RequestPartitionId> partitionChunks =


comment: So sharding is finally a concept in Hapi. This will make things easier.

michaelabuckley · 2025-11-25T22:51:58Z

...-fhir-storage-batch2/src/main/java/ca/uhn/fhir/batch2/jobs/step/GenerateRangeChunksStep.java

+				List<RequestPartitionId> partitionChunks =
+						myJobPartitionProvider.splitPartitionByShards(requestPartitionId);
+
+				for (RequestPartitionId nextPartitionId : partitionChunks) {


suggestion: this might be cleaner if we had a service to "expand" a list of PartitionedUrls instead of doing it inline here. Move that into IJobPartitionProvider. This code won't make any sense to Hapi devs.

michaelabuckley · 2025-11-25T22:54:05Z

.../java/ca/uhn/fhir/batch2/jobs/bulkmodify/framework/base/BaseBulkModifyOrRewriteProvider.java

 			throws IOException {
 		ServletRequestUtil.validatePreferAsyncHeader(theRequestDetails, getOperationName());
+
+		theJobParameters.setRequestPartitionId(


security question: do we want to allways allow explicit partition ids, or just in UNNAMED mode? This feels like a security hole in REQEUST_TENANT.

jamesagnew and others added 30 commits November 7, 2025 18:37

Work on bulk update

fdcbbcb

Add bulk patch partitioning

74c9144

Merge branch 'rel_8_6' into ja_20251106_megascale_bulk_terminology_up…

e6e9efd

…date

Bump to 8 7 (#7322)

27d6054

* bump to 8.7, add version enums * changelog folder for 8.8 * increase memory to elastic search container --------- Co-authored-by: jdar <[email protected]>

Check for null before removing 'max' parameter (#7242)

1572faf

[7332] Fix $mdm-query-links in partition mode (#7333)

2b87b9f

* [7332] update mdm resource persistence id * [7332] add tests and changelog * [7332] update changelog * [7332] update test * [7332] update test * [7332] update changelog

Ensure streaming queries don't have limit clauses (#7382)

0c64846

* Ensure streaming queries don't impose limit clauses on the sql when called in "synchronous" mode. * changelog

update search docs for accuracy (#7383)

f99cfb1

Co-authored-by: Elie Maamari <[email protected]>

Add debug logging to migrator. (#7381)

edbec01

STORAGE_TRANSACTION_PROCESSING-doesn't-pass-ServletRequestDetails (#7340

f60ea28

) * STORAGE_TRANSACTION_PROCESSING-doesn't-pass-ServletRequestDetails * Add changelog --------- Co-authored-by: James Agnew <[email protected]>

7217 with rel 8 6 (#7337)

f9e2e82

Fix Encounter $everything not working in REQUEST_TENANT partitioning (#…

52966b0

…7388) * Fix Encounter $everything not working in REQUEST_TENANT partitioning * Add test * changelog * Remove unused HttpServletRequest parameter * Remove unnecessary overriding

bump hapi

153d3b0

bump hapi

f2341f8

versioning

05e0408

Merge branch 'rel_8_6' into ja_20251106_megascale_bulk_terminology_up…

9769f9b

…date

Cleanup

a3bc1ba

Work on tests

63c1f4e

Work on test fixes, about to add TX api for megascale tx service

93dc059

Get patch working on megascale

c645678

Bump to 8 7 (#7322)

4fd9d53

* bump to 8.7, add version enums * changelog folder for 8.8 * increase memory to elastic search container --------- Co-authored-by: jdar <[email protected]>

[7332] Fix $mdm-query-links in partition mode (#7333)

79a6172

* [7332] update mdm resource persistence id * [7332] add tests and changelog * [7332] update changelog * [7332] update test * [7332] update test * [7332] update changelog

[7332] Fix $mdm-query-links in partition mode (#7333)

5fedb64

* [7332] update mdm resource persistence id * [7332] add tests and changelog * [7332] update changelog * [7332] update test * [7332] update test * [7332] update changelog

Test fix

dc5d34a

Test fixes

28b3f31

Spotless

345474c

jamesagnew added 7 commits November 24, 2025 14:59

Build fixes

5b1e4ab

Merge branch 'ja_20251106_megascale_bulk_terminology_update' of githu…

dce3d3f

…b.com:hapifhir/hapi-fhir into ja_20251106_megascale_bulk_terminology_update

Add troubleshooting logging

8dedff3

Add troubleshooting logs

7026814

Add troubleshooting

4c11ac7

Test fixes

a1c562e

Merge branch 'master' into ja_20251106_megascale_bulk_terminology_update

938d0e4

jamesagnew requested review from AD1306, fil512, michaelabuckley and tadgh as code owners November 25, 2025 10:46

jamesagnew added 2 commits November 25, 2025 05:49

Add changelog

fac62f8

Fix build

d499e39

michaelabuckley approved these changes Nov 25, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Improve partition support for Bulk Patch operation #7406

Improve partition support for Bulk Patch operation #7406

Uh oh!

jamesagnew commented Nov 25, 2025

Uh oh!

robogary commented Nov 25, 2025

Uh oh!

michaelabuckley Nov 25, 2025

Uh oh!

michaelabuckley Nov 25, 2025

Uh oh!

michaelabuckley Nov 25, 2025

Uh oh!

michaelabuckley Nov 25, 2025

Uh oh!

michaelabuckley Nov 25, 2025

Uh oh!

michaelabuckley Nov 25, 2025

Uh oh!

michaelabuckley Nov 25, 2025

Uh oh!

michaelabuckley Nov 25, 2025

Uh oh!

michaelabuckley Nov 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

Improve partition support for Bulk Patch operation #7406

Are you sure you want to change the base?

Improve partition support for Bulk Patch operation #7406

Uh oh!

Conversation

jamesagnew commented Nov 25, 2025

Uh oh!

robogary commented Nov 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants