Trino + Iceberg Connector + RestCatalog with S3 Filesystem

[chrigehr]

Hi Trino Community.

I'm having some problem with a Trino (409) + Iceberg Connector + RestCatalog + Files on S3/MinIO setup (local docker/docker-compose setup on my workstation). It is possible to create databases and tables with this setup, but an SELECT Operation on a table within this setup fails because the connector cannot load the class "org.apache.iceberg.aws.s3.S3FileIO". As fallback the connector tries to load a hadoop filesystem provider for "s3a", but this also fails. The interesting stuff seems to happen in the class "org.apache.iceberg.io.ResolvingFileIO".

Am i doing something wrong here? Are iceberg-aws or hadoop-aws libraries missing for some reason?

Searched issues and pull requests for this, but so far have only seen one possible connection to #16455
Update: #16213 seems also to be very similar, but in the ticket there are no concrete solutions if I understand it correctly.

I am grateful for every possible hint
Best regards,
Chris

[bitsondatadev]

@chrigehr, that version for Trino doesn't look correct, can you verify?

[chrigehr]

Thank you for the hint, mixed up Iceberg and Trino versions. I have corrected this.

[chrigehr]

In the above description I forgot to describe that I am doing the database creation, the table creation and the insert into the table via Spark. The read access via Trino and the Iceberg connector then fails as described.

I did another test today where the database creation, table creation and insert was done using trino itself (so essentially everything then took place via trino). To my surprise this worked as expected with no problems or errors at all.

Additionally I made the test with Trino 410 instead of 409, but the behaviour was the same.

[chrigehr]

Further details:

I used Spark 3.3.2 and Iceberg 1.1.0.

Trino Stacktrace:

trino-coordinator | 2023-03-10T15:48:10.331Z INFO Query-20230310_154810_00008_a7egi-336 org.apache.iceberg.CatalogUtil Loading custom FileIO implementation: org.apache.iceberg.aws.s3.S3FileIO
trino-coordinator | 2023-03-10T15:48:10.332Z WARN Query-20230310_154810_00008_a7egi-336 org.apache.iceberg.io.ResolvingFileIO Failed to load FileIO implementation: org.apache.iceberg.aws.s3.S3FileIO, falling back to org.apache.iceberg.hadoop.HadoopFileIO
trino-coordinator | java.lang.IllegalArgumentException: Cannot initialize FileIO, missing no-arg constructor: org.apache.iceberg.aws.s3.S3FileIO
trino-coordinator | at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:310)
trino-coordinator | at org.apache.iceberg.io.ResolvingFileIO.io(ResolvingFileIO.java:136)
trino-coordinator | at org.apache.iceberg.io.ResolvingFileIO.newInputFile(ResolvingFileIO.java:60)
trino-coordinator | at org.apache.iceberg.BaseSnapshot.cacheManifests(BaseSnapshot.java:148)
trino-coordinator | at org.apache.iceberg.BaseSnapshot.dataManifests(BaseSnapshot.java:174)
trino-coordinator | at org.apache.iceberg.DataTableScan.doPlanFiles(DataTableScan.java:82)
trino-coordinator | at org.apache.iceberg.BaseTableScan.planFiles(BaseTableScan.java:152)
trino-coordinator | at org.apache.iceberg.DataTableScan.planFiles(DataTableScan.java:27)
trino-coordinator | at io.trino.plugin.iceberg.TableStatisticsReader.makeTableStatistics(TableStatisticsReader.java:145)
trino-coordinator | at io.trino.plugin.iceberg.TableStatisticsReader.getTableStatistics(TableStatisticsReader.java:93)
trino-coordinator | at io.trino.plugin.iceberg.IcebergMetadata.lambda$getTableStatistics$66(IcebergMetadata.java:2421)
trino-coordinator | at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
trino-coordinator | at io.trino.plugin.iceberg.IcebergMetadata.getTableStatistics(IcebergMetadata.java:2399)
trino-coordinator | at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorMetadata.getTableStatistics(ClassLoaderSafeConnectorMetadata.java:319)
trino-coordinator | at io.trino.metadata.MetadataManager.getTableStatistics(MetadataManager.java:461)
trino-coordinator | at io.trino.cost.CachingTableStatsProvider.getTableStatistics(CachingTableStatsProvider.java:45)
trino-coordinator | at io.trino.cost.TableScanStatsRule.doCalculate(TableScanStatsRule.java:62)
trino-coordinator | at io.trino.cost.TableScanStatsRule.doCalculate(TableScanStatsRule.java:38)
trino-coordinator | at io.trino.cost.SimpleStatsRule.calculate(SimpleStatsRule.java:39)
trino-coordinator | at io.trino.cost.ComposableStatsCalculator.calculateStats(ComposableStatsCalculator.java:85)
trino-coordinator | at io.trino.cost.ComposableStatsCalculator.calculateStats(ComposableStatsCalculator.java:73)
trino-coordinator | at io.trino.cost.CachingStatsProvider.getStats(CachingStatsProvider.java:82)
trino-coordinator | at io.trino.sql.planner.optimizations.DeterminePartitionCount.lambda$getPartitionCountBasedOnOutputSize$1(DeterminePartitionCount.java:182)
trino-coordinator | at java.base/java.util.stream.ReferencePipeline$6$1.accept(ReferencePipeline.java:248)
trino-coordinator | at java.base/java.util.Collections$2.tryAdvance(Collections.java:4853)
trino-coordinator | at java.base/java.util.Collections$2.forEachRemaining(Collections.java:4861)
trino-coordinator | at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
trino-coordinator | at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
trino-coordinator | at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
trino-coordinator | at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
trino-coordinator | at java.base/java.util.stream.DoublePipeline.collect(DoublePipeline.java:541)
trino-coordinator | at java.base/java.util.stream.DoublePipeline.sum(DoublePipeline.java:450)
trino-coordinator | at io.trino.sql.planner.optimizations.DeterminePartitionCount.getSourceNodesOutputStats(DeterminePartitionCount.java:273)
trino-coordinator | at io.trino.sql.planner.optimizations.DeterminePartitionCount.getPartitionCountBasedOnOutputSize(DeterminePartitionCount.java:180)
trino-coordinator | at io.trino.sql.planner.optimizations.DeterminePartitionCount.determinePartitionCount(DeterminePartitionCount.java:146)
trino-coordinator | at io.trino.sql.planner.optimizations.DeterminePartitionCount.optimize(DeterminePartitionCount.java:118)
trino-coordinator | at io.trino.sql.planner.optimizations.StatsRecordingPlanOptimizer.optimize(StatsRecordingPlanOptimizer.java:54)
trino-coordinator | at io.trino.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:245)
trino-coordinator | at io.trino.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:220)
trino-coordinator | at io.trino.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:215)
trino-coordinator | at io.trino.execution.SqlQueryExecution.doPlanQuery(SqlQueryExecution.java:466)
trino-coordinator | at io.trino.execution.SqlQueryExecution.planQuery(SqlQueryExecution.java:447)
trino-coordinator | at io.trino.execution.SqlQueryExecution.start(SqlQueryExecution.java:388)
trino-coordinator | at io.trino.execution.SqlQueryManager.createQuery(SqlQueryManager.java:249)
trino-coordinator | at io.trino.dispatcher.LocalDispatchQuery.lambda$startExecution$7(LocalDispatchQuery.java:143)
trino-coordinator | at io.trino.$gen.Trino_410____20230310_152245_2.run(Unknown Source)
trino-coordinator | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
trino-coordinator | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
trino-coordinator | at java.base/java.lang.Thread.run(Thread.java:833)
trino-coordinator | Caused by: java.lang.NoSuchMethodException: Cannot find constructor for interface org.apache.iceberg.io.FileIO
trino-coordinator | Missing org.apache.iceberg.aws.s3.S3FileIO [java.lang.ClassNotFoundException: org.apache.iceberg.aws.s3.S3FileIO]
trino-coordinator | at org.apache.iceberg.common.DynConstructors.buildCheckedException(DynConstructors.java:250)
trino-coordinator | at org.apache.iceberg.common.DynConstructors.access$200(DynConstructors.java:32)
trino-coordinator | at org.apache.iceberg.common.DynConstructors$Builder.buildChecked(DynConstructors.java:220)
trino-coordinator | at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:307)
trino-coordinator | ... 48 more
trino-coordinator | Suppressed: java.lang.ClassNotFoundException: org.apache.iceberg.aws.s3.S3FileIO
trino-coordinator | at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
trino-coordinator | at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587)
trino-coordinator | at io.trino.server.PluginClassLoader.loadClass(PluginClassLoader.java:126)
trino-coordinator | at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
trino-coordinator | at java.base/java.lang.Class.forName0(Native Method)
trino-coordinator | at java.base/java.lang.Class.forName(Class.java:467)
trino-coordinator | at org.apache.iceberg.common.DynConstructors$Builder.impl(DynConstructors.java:149)
trino-coordinator | at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:306)
trino-coordinator | ... 48 more
trino-coordinator |

[chrigehr]

After some tests I could now narrow down the error to Trino and the Iceberg Connector in interaction with the RestCatalog. Spark has nothing to do with it. Furthermore, the same works with the JDBCCatalog without any problems.

I'm testing with Trino 410, Iceberg Connector+RESTCatalog, Minio. The following steps lead always to the described error:

1. Start Trino (iceberg_rest is the configured Iceberg Catalog using the Iceberg RESTCatalog)
2. CREATE SCHEMA iceberg_rest.trino_schema;
3. CREATE TABLE iceberg_rest.trino_schema.trino_table1 (id bigint, data varchar);
4. SELECT * FROM iceberg_rest.trino_schema.trino_table1; -- select working, throws no error
5. Restart Trino
6. SELECT * FROM iceberg_rest.trino_schema.trino_table1; -- not working, throws the error java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
7. CREATE TABLE iceberg_rest.trino_schema.trino_table2 (id bigint, data varchar);
8. SELECT * FROM iceberg_rest.trino_schema.trino_table1; -- select working again, throws no error

if I'm not mistaken this looks like a bug in the RestCatalog. Creating a table in the Iceberg catalog seems to reload something (e.g. a Java Class) or set some configuration on the catalog temporarily.

[harel-e]

@chrigehr - If you don't mind sharing, how did you configure Trino and Iceberg connector to use minio?

When I working with iceberg directly, I'm setting s3.endpoint property for the JdbcCatalog and also io-impl=org.apache.iceberg.aws.s3.S3FileIO, which works just fine.

Thank you in advance,
Harel