CLI throws cert validation errors inconsistently with TLS ingress using private CA cert.

[Cerebus]

I have trino deployed in k8s with istio, and the gateway is keyed from a private CA, so I packed a trust store for the CLI.

However, the CLI throws validation errors on inconsistently. Queries submitted in the same session will work, then fail. E.g.:

> trino --truststore-path=$(pwd)/truststore --truststore-password=changeit --server=https://trino.lvh.me --catalog=tpch --schema=sf1
trino:sf1> show catalogs;
 Catalog
---------
 system
 tpcds
 tpch
(3 rows)

Query 20230808_060739_00031_84g3i, FINISHED, 2 nodes
Splits: 20 total, 20 done (100.00%)
0.27 [0 rows, 0B] [0 rows/s, 0B/s]

trino:sf1> show catalogs;
Error running command: Error starting query at https://trino.lvh.me/v1/statement returned an invalid response: JsonResponse{statusCode=503, headers={content-length=[218], content-type=[text/plain], date=[Wed, 09 Aug 2023 13:53:14 GMT]}, hasValue=false} [Error: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED]

Occasionally, the CLI complains at startup, but I think this is the same root issue. Included here b/c it gives me a traceback:

> trino --truststore-path=$(pwd)/truststore --truststore-password=changeit --server=https://trino.lvh.me --catalog=tpch --schema=sf1
Aug 09, 2023 9:00:09 AM com.google.common.cache.LocalCache$Segment$1 run
WARNING: Exception thrown during refresh
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error starting query at https://trino.lvh.me/v1/statement returned an invalid response: JsonResponse{statusCode=503, headers={content-length=[218], content-type=[text/plain], date=[Wed, 09 Aug 2023 14:00:08 GMT]}, hasValue=false} [Error: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED]
        at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:547)
        at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:113)
        at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:240)
        at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2317)
        at com.google.common.cache.LocalCache$Segment$1.run(LocalCache.java:2297)
        at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
        at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:761)
        at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:136)
        at com.google.common.cache.LocalCache$Segment.loadAsync(LocalCache.java:2292)
        at com.google.common.cache.LocalCache$Segment.refresh(LocalCache.java:2364)
        at com.google.common.cache.LocalCache.refresh(LocalCache.java:4138)
        at com.google.common.cache.LocalCache$LocalLoadingCache.refresh(LocalCache.java:4969)
        at io.trino.cli.TableNameCompleter.lambda$populateCache$0(TableNameCompleter.java:105)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1623)
Caused by: java.lang.RuntimeException: Error starting query at https://trino.lvh.me/v1/statement returned an invalid response: JsonResponse{statusCode=503, headers={content-length=[218], content-type=[text/plain], date=[Wed, 09 Aug 2023 14:00:08 GMT]}, hasValue=false} [Error: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED]
        at io.trino.client.StatementClientV1.requestFailedException(StatementClientV1.java:457)
        at io.trino.client.StatementClientV1.<init>(StatementClientV1.java:123)
        at io.trino.client.StatementClientFactory.newStatementClient(StatementClientFactory.java:27)
        at io.trino.cli.QueryRunner.startInternalQuery(QueryRunner.java:159)
        at io.trino.cli.QueryRunner.startInternalQuery(QueryRunner.java:150)
        at io.trino.cli.TableNameCompleter.queryMetadata(TableNameCompleter.java:86)
        at io.trino.cli.TableNameCompleter.listFunctions(TableNameCompleter.java:80)
        at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:169)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533)
        at com.google.common.cache.LocalCache$Segment.loadAsync(LocalCache.java:2291)
        ... 7 more

This seems very strange. Normally this kind of issue would make me think there's a load balancer problem, but that can't be it.
This is a dev deploy on a single-node k8s cluster, a single-pod gateway and a single-coordinator trino cluster:

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
trino-trino-coordinator   1/1     1            1           5d18h

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
public-ingressgateway   1/1     1            1           5d19h

Any ideas?

I saw this issue but currently my toy deployment is unauthN, so that's not at issue.

[Cerebus]

With --debug I get a different stack trace:

trino> show catalogs;
Error running command: Error starting query at https://trino.lvh.me/v1/statement returned an invalid response: JsonResponse{statusCode=503, headers={content-length=[218], content-type=[text/plain], date=[Wed, 09 Aug 2023 14:39:20 GMT]}, hasValue=false} [Error: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED]
java.lang.RuntimeException: Error starting query at https://trino.lvh.me/v1/statement returned an invalid response: JsonResponse{statusCode=503, headers={content-length=[218], content-type=[text/plain], date=[Wed, 09 Aug 2023 14:39:20 GMT]}, hasValue=false} [Error: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED]
        at io.trino.client.StatementClientV1.requestFailedException(StatementClientV1.java:457)
        at io.trino.client.StatementClientV1.<init>(StatementClientV1.java:123)
        at io.trino.client.StatementClientFactory.newStatementClient(StatementClientFactory.java:27)
        at io.trino.cli.QueryRunner.startInternalQuery(QueryRunner.java:159)
        at io.trino.cli.QueryRunner.startQuery(QueryRunner.java:145)
        at io.trino.cli.Console.process(Console.java:376)
        at io.trino.cli.Console.runConsole(Console.java:307)
        at io.trino.cli.Console.run(Console.java:193)
        at io.trino.cli.Console.call(Console.java:100)
        at io.trino.cli.Console.call(Console.java:73)
        at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
        at picocli.CommandLine.access$1500(CommandLine.java:148)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
        at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
        at picocli.CommandLine.execute(CommandLine.java:2170)
        at io.trino.cli.Trino.main(Trino.java:42)

[nineinchnick]

What Trino CLI and server versions are you using? Can you try using other tools, maybe openssl s_client -connect trino.lvh.me:443, to verify the cert to see if they would fail intermittently?

[Cerebus]

Server 422
CLI 412 (keep forgetting to update)

s_client is showing me the right certs.

[nineinchnick]

Can you run s_client in a loop for a bit?

[nineinchnick]

Also, can you test the latest CLI version?

[Cerebus]

When in doubt, redeploy. I wiped and redeployed, and now I can't reproduce. Been running TPC-H queries for the last little bit and no issues. 🤷

[Cerebus]

So, what I think happened, in case anyone else lands here in the future:

• I'd fiddled with the Gateway's Secret more than once (editing in-cluster) b/c I had to roll the cert, and while Istio was perfectly happy with self-signed, none of my browsers were happy about it, so I rolled up a cheap CA.
• istiod was reporting that it picked up the each change to Secret as I committed them to the cluster.
• I think an envoy thread didn't get the message.

Wiping out the trino deploy deleted the VirtualService, which caused the related envoy threads to die. So when the VS deployed again they all started with a clean slate, and all is right with the world.

What bugs me about this was I wasn't able to definitively show this with s_client.