Skip to content

Spooled result downloads return 404 when proxied through Trino Gateway #924

@nandaqmr

Description

@nandaqmr

Issue Body

Environment

Component Version
Trino Gateway 17
Trino Server 478
Trino JDBC Driver 478
Java 21
Kubernetes Yes (with Istio service mesh)

Trino Spooling Configuration

protocol.spooling.enabled=true
protocol.spooling.shared-secret-key=<redacted>
protocol.spooling.retrieval-mode=COORDINATOR_PROXY
protocol.spooling.inlining.enabled=false
protocol.spooling.initial-segment-size=64MB
protocol.spooling.max-segment-size=128MB

Trino Gateway Configuration

extraWhitelistPaths:
- '/v1/.*'
- '/v2/.*'
- '.*'
- '/v1/spooled/download/.*'

config:
serverConfig:
node.environment: test
http-server.http.port: 8080
http-server.http.enabled: true
http-server.process-forwarded: true

Problem Description

When clients connect through Trino Gateway to a Trino cluster with spooling enabled, requests to /v1/spooled/download/... return HTTP 404. The gateway does not proxy these requests to the backend Trino cluster.

Direct connection to Trino: Works (spooling works correctly) Connection through Trino Gateway: Fails with 404 on spooled download

Workaround: Setting SET SESSION protocol_spooling_enabled = false or using JDBC URL parameter protocol=SPOOLING_V1_DISABLED resolves the issue, but disables spooling entirely.

Expected Behavior

Trino Gateway should proxy /v1/spooled/download/... requests to the appropriate backend Trino cluster, similar to how it handles /v1/statement/... requests.

Actual Behavior

The gateway returns 404 for /v1/spooled/download/... requests without forwarding them to the backend.

Evidence

Trino Gateway logs show successful routing for /v1/statement requests but no routing logs for requests: /v1/spooled/download/

2026-02-17T11:27:34.029Z INFO io.trino.gateway.ha.handler.RoutingTargetHandler Rerouting [https://1x.20x.2x.2xx:443/v1/statement/executing/...]--> [https://backend-trino-cluster/v1/statement/executing/...]

Istio proxy logs confirm the request reaches the gateway but returns 404:

{
"path": "/v1/spooled/download/",
"response_code": "404",
"response_flags": "-",
"response_code_details": "via_upstream",
"upstream_host": "10.5x.1xx.1xx:8080",
"upstream_service_time": "1"
}

The 404 is returned by the gateway itself (1ms response time, via_upstream pointing to gateway port 8080), not by the backend Trino cluster.

Client Error

java.io.IOException: Could not open segment for streaming, got error 'Not Found' with code 404
at io.trino.jdbc.$internal.client.OkHttpSegmentLoader.load(OkHttpSegmentLoader.java:67)
at io.trino.jdbc.$internal.client.spooling.SpooledSegmentIterator.load(SpooledSegmentIterator.java:64)[wrapped] java.io.UncheckedIOException: java.io.IOException: Could not open segment for streaming, got error 'Not Found' with code 404    at io.trino.jdbc.$internal.client.spooling.SpooledSegmentIterator.load(SpooledSegmentIterator.java:70)    at io.trino.jdbc.$internal.client.spooling.SpooledSegmentIterator.computeNext(SpooledSegmentIterator.java:91)    at io.trino.jdbc.$internal.client.spooling.SpooledSegmentIterator.computeNext(SpooledSegmentIterator.java:33)    at io.trino.jdbc.$internal.guava.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)    at io.trino.jdbc.$internal.guava.collect.AbstractIterator.hasNext(AbstractIterator.java:136)    at io.trino.jdbc.$internal.client.spooling.SegmentsIterator.computeNext(SegmentsIterator.java:49)    at io.trino.jdbc.$internal.client.spooling.SegmentsIterator.computeNext(SegmentsIterator.java:28)    at io.trino.jdbc.$internal.guava.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)    at io.trino.jdbc.$internal.guava.collect.AbstractIterator.hasNext(AbstractIterator.java:136)    at io.trino.jdbc.$internal.client.CloseableLimitingIterator.computeNext(CloseableLimitingIterator.java:55)    at io.trino.jdbc.$internal.guava.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)    at io.trino.jdbc.$internal.guava.collect.AbstractIterator.hasNext(AbstractIterator.java:136)    at io.trino.jdbc.$internal.client.StatementClientV1$HeartbeatingResultRows$1.computeNext(StatementClientV1.java:727)    at io.trino.jdbc.$internal.client.StatementClientV1$HeartbeatingResultRows$1.computeNext(StatementClientV1.java:722)    at io.trino.jdbc.$internal.guava.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)    at io.trino.jdbc.$internal.guava.collect.AbstractIterator.hasNext(AbstractIterator.java:136)    at io.trino.jdbc.AsyncResultIterator.lambda$new$1(AsyncResultIterator.java:76)    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) [wrapped] java.sql.SQLException: Error fetching results    at io.trino.jdbc.AbstractTrinoResultSet.next(AbstractTrinoResultSet.java:230)    at io.trino.jdbc.TrinoResultSet.next(TrinoResultSet.java:33)

Steps to Reproduce

  1. Configure Trino cluster with protocol.spooling.enabled=true and protocol.spooling.retrieval-mode=COORDINATOR_PROXY
  2. Deploy Trino Gateway v17 pointing to this cluster
  3. Connect via JDBC through the gateway
  4. Execute any query that returns results (e.g., SHOW CATALOGS)
  5. Observe 404 error when client attempts to fetch spooled results

Additional Context

  • The extraWhitelistPaths configuration includes but this doesn't appear to enable proxying for these paths /v1/spooled/download/.*

Metadata

Metadata

Assignees

No one assigned

    Labels

    roadmapAny items we definitely plan to tackle that are not bugs.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions