fix(sdk): DSPX-4590 default per-segment sizes when a writer omits them by dmihalcik-virtru · Pull Request #3979 · opentdf/platform

dmihalcik-virtru · 2026-09-04T13:26:14Z

Jira: https://virtru.atlassian.net/browse/DSPX-4590

Part 2 of a 3-PR stack: #3933 (base) <- this PR <- #3981 (zip64 conformance, stacked on top).

Based on #3933 (fix(sdk): map ReadAt plaintext offsets from cumulative
segment sizes) rather than main, since both PRs rewrote Reader.ReadAt
for independent reasons and needed to be reconciled rather than merged
separately.

Relationship to #3933

#3933 replaced ReadAt's uniform-stride segment lookup
(offset / DefaultSegmentSize) with a cumulative walk over each segment's
actual declared size, so non-uniform segments (which sdk/experimental/tdf
can emit) map to the correct plaintext offsets.

This PR fixes finding 7 of the DSPX-4590 investigation on top of that:
manifest.schema.json makes per-segment segmentSize/encryptedSegmentSize
optional (a writer may omit them whenever they equal the manifest-level
default), and web-sdk does exactly that. go-sdk's reader didn't default
them back, so every web-sdk container over one segment failed to decrypt.

IntegrityInformation.resolveSegmentSizes now substitutes the manifest
defaults, wired into LoadTDF's payloadSize computation, WriteTo, and
ReadAt (using #3933's cumulative-walk structure, calling
resolveSegmentSizes per segment instead of trusting seg.Size/
seg.EncryptedSize directly).

The empty/omitted per-segment size ambiguity

JSON can't distinguish an omitted key from an explicit 0. go-sdk's own
CreateTDF writes segmentSize: 0 (no omitempty) for the sole segment of
an empty-payload TDF, so an early version of this fix that treated any 0
as "omitted" mis-resolved that case. Checking go-sdk, java-sdk and web-sdk's
actual manifest-writing source found:

go-sdk and java-sdk always set Size/EncryptedSize together (never
independently zero).
web-sdk (lib/tdf3/src/tdf.ts) decides whether to omit each field with
two independent equals-the-default comparisons, not one joint check.

resolveSegmentSizes now resolves EncryptedSize first -- its zero value
is never ambiguous, since ciphertext can never legitimately be zero bytes --
and disambiguates a zero Size by comparing the resolved EncryptedSize
against DefaultEncryptedSegSize, rather than assuming the two fields are
only ever omitted together. This needs no assumption about the cipher's
per-segment overhead (nonce/tag size stays out of manifest.go entirely):
the overhead is constant across every segment in one manifest, so if the
resolved EncryptedSize equals its default, the plaintext size must too,
regardless of what that overhead number actually is.

GMAC failure classification

calculateSignature's too-short-ciphertext-for-GMAC path returned a bare,
unclassified errors.New(...), unlike every other integrity failure in this
file. It now returns a new ErrGMACSignatureFailed, wrapped in ErrTampered
like ErrSegSizeMismatch/ErrSegSigValidation.

A note on #3933 standalone

Without this fix, #3933's cumulative-walk ReadAt uses seg.Size directly,
so an omitted (0) per-segment size stalls the plaintext cursor and desyncs
the ciphertext offset for every segment after it. Reading a web-sdk
multi-segment file then fails with a misleading tamper detected: failed integrity check on segment hash instead of main's current (also broken,
but at least consistent) fail to create gmac signature. #3933 should not
be merged or relied on standalone for real multi-segment interop until
this lands on top of it.

Testing

cd sdk && go test ./... -race
make fmt, make lint (0 new issues)
cd sdk && go test -run TestREADMECodeBlocks
Verified against opentdf/tests' DSPX-4592-java-underflow branch
(test_tdfs.py::test_chunky_roundtrip, a 5 MiB round-trip that forces a
full-default-sized segment): with platform-ref/otdfctl-ref pointed at
this branch and XT_FORCE_SUPPORTS=chunky, js-encrypt -> go-decrypt
passes. The one remaining failure in that run, js-encrypt -> java-decrypt,
is java-sdk's own pre-existing GMAC-on-empty-segment bug (DSPX-4589),
unrelated to this change.

Supersedes #3967, which is left open, unmodified, for reference.

coderabbitai · 2026-09-04T13:26:27Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Team

Run ID: fdc85c12-a80b-414a-ac9a-bdf53d4f6c72

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-09-04T13:33:13Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	239.201789ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	135.989915ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	432.55876ms
Throughput	231.18 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	1m0.707806352s
Average Latency	605.878326ms
Throughput	82.36 requests/second

github-actions · 2026-09-04T13:41:29Z

X-Test Failure Report

bats-test-results
✅ go@main-pull-3979
✅ go@pull-3979-pull-3979
✅ go@main-v0.9.0
✅ go@main-main
✅ java@main-main
✅ go@pull-3979-v0.9.0
✅ js@main-pull-3979
✅ java@main-pull-3979
✅ js@main-main
✅ java@main-v0.9.0
✅ js@main-v0.9.0
cukes-report
server-logs-go@pull-3979-main
opentdf~~platform~~T5C5FA.dockerbuild
govulncheck-failure-8
govulncheck-failure-3
govulncheck-failure-1

github-actions · 2026-09-04T15:47:46Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	170.151469ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	99.831069ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	347.126643ms
Throughput	288.08 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	45.602323608s
Average Latency	455.077545ms
Throughput	109.64 requests/second

github-actions · 2026-09-04T15:58:29Z

X-Test Results

✅ go@pull-3979-pull-3979
✅ go@main-pull-3979
✅ go@pull-3979-main
✅ go@main-main
bats-test-results
✅ go@pull-3979-v0.9.0
✅ go@main-v0.9.0
✅ java@main-main
✅ java@main-pull-3979
✅ js@main-main
✅ js@main-pull-3979
✅ java@main-v0.9.0
cukes-report
✅ js@main-v0.9.0
opentdf~~platform~~DYBFL6.dockerbuild
govulncheck-failure-8
govulncheck-failure-3
govulncheck-failure-1

github-actions · 2026-09-04T16:34:21Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	168.793481ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	99.084786ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	316.130097ms
Throughput	316.33 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	41.678502269s
Average Latency	416.035912ms
Throughput	119.97 requests/second

segmentSize and encryptedSegmentSize are optional per-segment overrides: manifest.schema.json requires only the integrityInformation defaults, and web-sdk omits the per-segment keys whenever they equal the default, so every web-sdk container over one segment failed to decrypt in go-sdk. Fall back to the manifest defaults in the payload-size computation, WriteTo and ReadAt. Rebased onto #3933 (map ReadAt plaintext offsets from cumulative segment sizes), which rewrote ReadAt's segment lookup from a uniform DefaultSegmentSize stride to a walk over each segment's actual plaintext/ciphertext size -- necessary for the non-uniform segments sdk/experimental/tdf can emit. Reconciling the two surfaced a further bug: resolveSegmentSizes treated a per-segment size of 0 as "omitted, use the default" independently for Size and EncryptedSize, but JSON can't distinguish an omitted key from an explicit 0, and go-sdk's own CreateTDF already writes segmentSize: 0 for the sole segment of an empty-payload TDF (no omitempty on the field). That made an empty TDF round-trip to the wrong payloadSize. resolveSegmentSizes now resolves EncryptedSize first -- its zero value is never ambiguous, since ciphertext can never legitimately be zero bytes -- and disambiguates a zero Size by comparing the resolved EncryptedSize against DefaultEncryptedSegSize rather than assuming Size and EncryptedSize are only ever omitted together. Checking go-sdk, java-sdk and web-sdk's actual manifest-writing source confirmed go-sdk and java-sdk always set both fields together (so a joint-zero assumption happened to hold for them), but web-sdk's lib/tdf3/src/tdf.ts decides whether to omit segmentSize and encryptedSegmentSize with two independent equals-the-default comparisons, not one joint check -- so a joint-zero- only version would have mis-resolved a segment where only one of the two happened to be omitted. The corrected comparison needs no assumption about the cipher's per-segment overhead (nonce/tag size stays out of manifest.go entirely): the overhead is constant across every segment in one manifest, so if the resolved EncryptedSize equals its default, the plaintext size must too, regardless of what that overhead number actually is. Also gives calculateSignature's too-short-ciphertext-for-GMAC error (previously a bare, unclassified error) a proper ErrTampered-wrapped sentinel, consistent with the rest of this file's integrity failures. Verified against opentdf/tests' DSPX-4592-java-underflow branch (adds test_tdfs.py::test_chunky_roundtrip, a 5 MiB round-trip that forces a full-default-sized segment): with platform-ref and otdfctl-ref both pointed at this branch and XT_FORCE_SUPPORTS=chunky, js-encrypt -> go-decrypt passes (js omits per-segment sizes on the full-sized segment; go now defaults them back). The one remaining failure in that run, js-encrypt -> java-decrypt, is java-sdk's own pre-existing GMAC-on-empty- segment bug (DSPX-4589), unrelated to this change. Note on #3933 standalone: without this fix, #3933's cumulative-walk ReadAt uses seg.Size directly, so an omitted (0) per-segment size stalls the plaintext cursor and desyncs the ciphertext offset for every segment after it. Reading a web-sdk multi-segment file then fails with a misleading "tamper detected: failed integrity check on segment hash" instead of main's current (also broken, but at least consistent) "fail to create gmac signature". #3933 should not be merged or relied on standalone for real multi-segment interop until this lands on top of it. The zip64/ZIP64-conformance findings originally bundled with this change (findings 1-6 of the DSPX-4590 investigation) now live in a separate PR stacked on top of this one, since they are independent of the segment- size defaulting fixed here. Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

github-actions · 2026-09-04T16:38:25Z

X-Test Failure Report

cukes-report
✅ js@main-pull-3979
opentdf~~platform~~O4IV7P.dockerbuild
govulncheck-failure-8
govulncheck-failure-3
govulncheck-failure-1

github-actions · 2026-09-04T16:39:30Z

⚠️ Govulncheck found vulnerabilities ⚠️

The following modules have known vulnerabilities:

otdfctl
service
tests-bdd

See the workflow run for details.

github-actions · 2026-09-04T16:41:12Z

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	175.029963ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric	Value
Approved Decision Requests	1000
Denied Decision Requests	0
Total Time	96.16689ms

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	350.532564ms
Throughput	285.28 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	45.899478871s
Average Latency	458.294541ms
Throughput	108.93 requests/second

github-actions · 2026-09-04T16:52:50Z

X-Test Results

✅ go@main-main
✅ go@pull-3979-main
✅ go@pull-3979-pull-3979
bats-test-results
✅ go@pull-3979-v0.9.0
✅ java@main-main
✅ go@main-v0.9.0
✅ go@main-pull-3979
✅ js@main-main
✅ js@main-pull-3979
✅ js@main-v0.9.0
✅ java@main-pull-3979
cukes-report
✅ java@main-v0.9.0
opentdf~~platform~~RHFLCT.dockerbuild
govulncheck-failure-8
govulncheck-failure-3
govulncheck-failure-1

github-actions Bot added the comp:sdk A software development kit, including library, for client applications and inter-service communicati label Sep 4, 2026

github-actions Bot added the size/m label Sep 4, 2026

dmihalcik-virtru force-pushed the DSPX-4590-zip64-conformance-v2 branch from 7be0388 to 407835f Compare September 4, 2026 15:42

dmihalcik-virtru mentioned this pull request Sep 4, 2026

feat(xtest): 2.1 GiB ZIP64 boundary coverage + chunky segment-defaulting (DSPX-4592) opentdf/tests#586

Open

dmihalcik-virtru force-pushed the DSPX-4590-zip64-conformance-v2 branch from 407835f to fad425f Compare September 4, 2026 16:28

dmihalcik-virtru changed the title ~~fix(sdk): DSPX-4590 zip64 conformance and per-segment size defaults (on #3933)~~ fix(sdk): DSPX-4590 default per-segment sizes when a writer omits them Sep 4, 2026

dmihalcik-virtru mentioned this pull request Sep 4, 2026

fix(sdk): DSPX-4590 zip64 conformance #3981

Draft

dmihalcik-virtru force-pushed the DSPX-4590-zip64-conformance-v2 branch from fad425f to 128c23a Compare September 4, 2026 16:36

dmihalcik-virtru marked this pull request as ready for review September 4, 2026 17:44

dmihalcik-virtru requested review from a team as code owners September 4, 2026 17:44

Uh oh!

Conversation

dmihalcik-virtru commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relationship to #3933

The empty/omitted per-segment size ambiguity

GMAC failure classification

A note on #3933 standalone

Testing

Uh oh!

coderabbitai Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

github-actions Bot commented Sep 4, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 4, 2026

X-Test Failure Report

Uh oh!

github-actions Bot commented Sep 4, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 4, 2026

X-Test Results

Uh oh!

github-actions Bot commented Sep 4, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 4, 2026

X-Test Failure Report

Uh oh!

github-actions Bot commented Sep 4, 2026

⚠️ Govulncheck found vulnerabilities ⚠️

Uh oh!

github-actions Bot commented Sep 4, 2026

Benchmark authorization.GetDecisions Results:

Benchmark authorization.v2.GetMultiResourceDecision Results:

Benchmark Statistics

Bulk Benchmark Results

TDF3 Benchmark Results:

Uh oh!

github-actions Bot commented Sep 4, 2026

X-Test Results

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

dmihalcik-virtru commented Sep 4, 2026 •

edited

Loading

coderabbitai Bot commented Sep 4, 2026 •

edited

Loading