Cannot load delta table in dbeaver

[hurcy]

Hi,

I have delta tables, but I couldn't read it from dbeaver.
Do you know how to solve this issue?

2023-01-08 08:54:49.989 ERROR org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Error getting tables:
org.apache.kyuubi.KyuubiSQLException: Error operating GetTables: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.io.delta.hive.DeltaStorageHandler
	at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:297)
	at org.apache.spark.sql.hive.client.HiveClientImpl.convertHiveTableToCatalogTable(HiveClientImpl.scala:471)
	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getTablesByName$2(HiveClientImpl.scala:423)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at scala.collection.TraversableLike.map(TraversableLike.scala:286)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
	at scala.collection.AbstractTraversable.map(Traversable.scala:108)
	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getTablesByName$1(HiveClientImpl.scala:423)
	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:294)
	at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:225)
	at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:224)
	at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:274)
	at org.apache.spark.sql.hive.client.HiveClientImpl.getTablesByName(HiveClientImpl.scala:423)
	at org.apache.spark.sql.hive.HiveExternalCatalog.getRawTablesByNames(HiveExternalCatalog.scala:126)
	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$getTablesByName$1(HiveExternalCatalog.scala:733)
	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:101)
	at org.apache.spark.sql.hive.HiveExternalCatalog.getTablesByName(HiveExternalCatalog.scala:733)
	at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.getTablesByName(ExternalCatalogWithListener.scala:142)
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getTablesByName(SessionCatalog.scala:538)
	at org.apache.kyuubi.engine.spark.shim.CatalogShim_v2_4.$anonfun$getCatalogTablesOrViews$1(CatalogShim_v2_4.scala:73)
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
	at scala.collection.Iterator.foreach(Iterator.scala:943)
	at scala.collection.Iterator.foreach$(Iterator.scala:943)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
	at org.apache.kyuubi.engine.spark.shim.CatalogShim_v2_4.getCatalogTablesOrViews(CatalogShim_v2_4.scala:71)
	at org.apache.kyuubi.engine.spark.shim.CatalogShim_v3_0.getCatalogTablesOrViews(CatalogShim_v3_0.scala:164)
	at org.apache.kyuubi.engine.spark.operation.GetTables.runInternal(GetTables.scala:71)
	at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:164)
	at org.apache.kyuubi.session.AbstractSession.runOperation(AbstractSession.scala:99)
	at org.apache.kyuubi.engine.spark.session.SparkSessionImpl.runOperation(SparkSessionImpl.scala:78)
	at org.apache.kyuubi.session.AbstractSession.getTables(AbstractSession.scala:160)
	at org.apache.kyuubi.service.AbstractBackendService.getTables(AbstractBackendService.scala:94)
	at org.apache.kyuubi.service.TFrontendService.GetTables(TFrontendService.scala:316)
	at org.apache.kyuubi.shade.org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetTables.getResult(TCLIService.java:1637)
	at org.apache.kyuubi.shade.org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetTables.getResult(TCLIService.java:1622)
	at org.apache.kyuubi.shade.org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.kyuubi.shade.org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36)
	at org.apache.kyuubi.shade.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.io.delta.hive.DeltaStorageHandler
	at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:310)
	at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:292)
	... 50 more
Caused by: java.lang.ClassNotFoundException: io.delta.hive.DeltaStorageHandler
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:305)
	... 51 more

	at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) ~[kyuubi-common_2.12-1.6.1-incubating.jar:1.6.1-incubating]
	at org.apache.kyuubi.engine.spark.operation.SparkOperation$$anonfun$onError$1.applyOrElse(SparkOperation.scala:112) ~[?:?]
	at org.apache.kyuubi.engine.spark.operation.SparkOperation$$anonfun$onError$1.applyOrElse(SparkOperation.scala:96) ~[?:?]
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38) ~[scala-library-2.12.15.jar:?]
	at org.apache.kyuubi.engine.spark.operation.GetTables.runInternal(GetTables.scala:82) ~[?:?]
	at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:164) ~[kyuubi-common_2.12-1.6.1-incubating.jar:1.6.1-incubating]
	at org.apache.kyuubi.session.AbstractSession.runOperation(AbstractSession.scala:99) ~[kyuubi-common_2.12-1.6.1-incubating.jar:1.6.1-incubating]
	at org.apache.kyuubi.engine.spark.session.SparkSessionImpl.runOperation(SparkSessionImpl.scala:78) ~[?:?]
	at org.apache.kyuubi.session.AbstractSession.getTables(AbstractSession.scala:160) ~[kyuubi-common_2.12-1.6.1-incubating.jar:1.6.1-incubating]
	at org.apache.kyuubi.service.AbstractBackendService.getTables(AbstractBackendService.scala:94) ~[kyuubi-common_2.12-1.6.1-incubating.jar:1.6.1-incubating]
	at org.apache.kyuubi.service.TFrontendService.GetTables(TFrontendService.scala:316) ~[kyuubi-common_2.12-1.6.1-incubating.jar:1.6.1-incubating]
	at org.apache.kyuubi.shade.org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetTables.getResult(TCLIService.java:1637) ~[?:?]
	at org.apache.kyuubi.shade.org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetTables.getResult(TCLIService.java:1622) ~[?:?]
	at org.apache.kyuubi.shade.org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[?:?]
	at org.apache.kyuubi.shade.org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[?:?]
	at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) ~[kyuubi-common_2.12-1.6.1-incubating.jar:1.6.1-incubating]
	at org.apache.kyuubi.shade.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_352]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_352]
	at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_352]
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.io.delta.hive.DeltaStorageHandler
	at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:297) ~[?:?]
	at org.apache.spark.sql.hive.client.HiveClientImpl.convertHiveTableToCatalogTable(HiveClientImpl.scala:471) ~[?:?]
	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getTablesByName$2(HiveClientImpl.scala:423) ~[?:?]
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286) ~[scala-library-2.12.15.jar:?]
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) ~[scala-library-2.12.15.jar:?]
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) ~[scala-library-2.12.15.jar:?]
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) ~[scala-library-2.12.15.jar:?]
	at scala.collection.TraversableLike.map(TraversableLike.scala:286) ~[scala-library-2.12.15.jar:?]
	at scala.collection.TraversableLike.map$(TraversableLike.scala:279) ~[scala-library-2.12.15.jar:?]
	at scala.collection.AbstractTraversable.map(Traversable.scala:108) ~[scala-library-2.12.15.jar:?]
	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getTablesByName$1(HiveClientImpl.scala:423) ~[?:?]
	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:294) ~[?:?]
	at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:225) ~[?:?]
	at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:224) ~[?:?]
	at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:274) ~[?:?]
	at org.apache.spark.sql.hive.client.HiveClientImpl.getTablesByName(HiveClientImpl.scala:423) ~[?:?]
	at org.apache.spark.sql.hive.HiveExternalCatalog.getRawTablesByNames(HiveExternalCatalog.scala:126) ~[?:?]
	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$getTablesByName$1(HiveExternalCatalog.scala:733) ~[?:?]
	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:101) ~[?:?]
	at org.apache.spark.sql.hive.HiveExternalCatalog.getTablesByName(HiveExternalCatalog.scala:733) ~[?:?]
	at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.getTablesByName(ExternalCatalogWithListener.scala:142) ~[?:?]
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getTablesByName(SessionCatalog.scala:538) ~[?:?]
	at org.apache.kyuubi.engine.spark.shim.CatalogShim_v2_4.$anonfun$getCatalogTablesOrViews$1(CatalogShim_v2_4.scala:73) ~[?:?]
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293) ~[scala-library-2.12.15.jar:?]
	at scala.collection.Iterator.foreach(Iterator.scala:943) ~[scala-library-2.12.15.jar:?]
	at scala.collection.Iterator.foreach$(Iterator.scala:943) ~[scala-library-2.12.15.jar:?]
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) ~[scala-library-2.12.15.jar:?]
	at scala.collection.IterableLike.foreach(IterableLike.scala:74) ~[scala-library-2.12.15.jar:?]
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73) ~[scala-library-2.12.15.jar:?]
	at scala.collection.AbstractIterable.foreach(Iterable.scala:56) ~[scala-library-2.12.15.jar:?]
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293) ~[scala-library-2.12.15.jar:?]
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290) ~[scala-library-2.12.15.jar:?]
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108) ~[scala-library-2.12.15.jar:?]
	at org.apache.kyuubi.engine.spark.shim.CatalogShim_v2_4.getCatalogTablesOrViews(CatalogShim_v2_4.scala:71) ~[?:?]
	at org.apache.kyuubi.engine.spark.shim.CatalogShim_v3_0.getCatalogTablesOrViews(CatalogShim_v3_0.scala:164) ~[?:?]
	at org.apache.kyuubi.engine.spark.operation.GetTables.runInternal(GetTables.scala:71) ~[?:?]
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:Error in loading storage handler.io.delta.hive.DeltaStorageHandler
	at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:310) ~[?:?]
	at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:292) ~[?:?]
Caused by: java.lang.ClassNotFoundException: io.delta.hive.DeltaStorageHandler
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387) ~[?:1.8.0_352]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_352]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_352]
	at java.lang.Class.forName0(Native Method) ~[?:1.8.0_352]
	at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_352]
	at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:305) ~[?:?]

[pan3793]

Have you follow the delta docs to[1] set up the delta w/ your spark deployment? Kyuubi is a Gateway for computing engine(in your case, is Spark), and it relies on the computing engine to support delta lake.

Basically, you should put the delta jar into $SPARK_HOME/jars, and add configurations into $SPARK_HOME/conf/spark-defaults.conf

spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension
spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog

[1] https://docs.delta.io/latest/quick-start.html#set-up-apache-spark-with-delta-lake

[hurcy]

I put delta jar into $SPARK_HOME/jars and configurations too.
It was okey when I submit job in shell.