[pubsublite] incompatibility between google-cloud-pubsublite and pubsublite-spark-sql-streaming

[jhsenjaliya]

1. Specify the API at the beginning of the title. For example, "BigQuery: ...").
   General, Core, and Other are also allowed as types
2. OS type and version: Mac M1
3. Java version: 11
4. version(s):

Steps to reproduce

1. try to create pubsub reader (same as examples from repo) using google-cloud-pubsublite:1.12.10 ( any version after 1.10) and pubsublite-spark-sql-streaming: 1.0.0

Code example

// example from https://github.com/googleapis/java-pubsublite-spark/tree/main/samples

Stack trace

2023-05-30 11:15:05,024 [INFO  ] [84] org.apache.spark.sql.execution.streaming.MicroBatchExecution - Reading table [com.google.cloud.pubsublite.spark.PslTable@79708f8] from DataSourceV2 named 'pubsublite' [com.google.cloud.pubsublite.spark.PslTableProvider@64ddf4ea]
2023-05-30 11:15:07,496 [INFO  ] [84] org.apache.spark.sql.execution.streaming.MicroBatchExecution - Starting new streaming query.
2023-05-30 11:15:07,498 [INFO  ] [84] org.apache.spark.sql.execution.streaming.MicroBatchExecution - Stream started from {}
2023-05-30 11:15:09,308 [INFO  ] [84] org.apache.spark.sql.execution.streaming.CheckpointFileManager - Writing atomically to file:/private/var/folders/m9/fgqzm44d0111yh6kpgv51wpw0000gn/T/temporary-a794d805-5113-4f16-a758-36d8ce45a5e8/offsets/0 using temp file file:/private/var/folders/m9/fgqzm44d0111yh6kpgv51wpw0000gn/T/temporary-a794d805-5113-4f16-a758-36d8ce45a5e8/offsets/.0.40a4adab-d8ac-4e10-8af0-447fc3d8110c.tmp
2023-05-30 11:15:09,348 [INFO  ] [84] org.apache.spark.sql.execution.streaming.CheckpointFileManager - Renamed temp file file:/private/var/folders/m9/fgqzm44d0111yh6kpgv51wpw0000gn/T/temporary-a794d805-5113-4f16-a758-36d8ce45a5e8/offsets/.0.40a4adab-d8ac-4e10-8af0-447fc3d8110c.tmp to file:/private/var/folders/m9/fgqzm44d0111yh6kpgv51wpw0000gn/T/temporary-a794d805-5113-4f16-a758-36d8ce45a5e8/offsets/0
2023-05-30 11:15:09,349 [INFO  ] [84] org.apache.spark.sql.execution.streaming.MicroBatchExecution - Committed offsets for batch 0. Metadata OffsetSeqMetadata(0,1685470509297,Map(spark.sql.streaming.stateStore.providerClass -> org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider, spark.sql.streaming.join.stateFormatVersion -> 2, spark.sql.streaming.stateStore.compression.codec -> lz4, spark.sql.streaming.stateStore.rocksdb.formatVersion -> 5, spark.sql.streaming.statefulOperator.useStrictDistribution -> true, spark.sql.streaming.flatMapGroupsWithState.stateFormatVersion -> 2, spark.sql.streaming.multipleWatermarkPolicy -> min, spark.sql.streaming.aggregation.stateFormatVersion -> 2, spark.sql.shuffle.partitions -> 200))

Exception in thread "pubsub-lite-futures-1" java.lang.NoSuchMethodError: 'com.google.api.gax.rpc.ApiCallContext com.google.cloud.pubsublite.internal.wire.ServiceClients.getCallContext(com.google.cloud.pubsublite.internal.wire.PubsubContext, com.google.cloud.pubsublite.internal.wire.RoutingMetadata)'
	at com.google.cloud.pubsublite.spark.PslReadDataSourceOptions.lambda$null$1(PslReadDataSourceOptions.java:139)
	at com.google.cloud.pubsublite.internal.wire.SingleConnection.<init>(SingleConnection.java:71)
	at com.google.cloud.pubsublite.internal.wire.SingleConnection.<init>(SingleConnection.java:77)
	at com.google.cloud.pubsublite.internal.wire.ConnectedSubscriberImpl.<init>(ConnectedSubscriberImpl.java:54)
	at com.google.cloud.pubsublite.internal.wire.ConnectedSubscriberImpl.<init>(ConnectedSubscriberImpl.java:33)
	at com.google.cloud.pubsublite.internal.wire.ConnectedSubscriberImpl$Factory.New(ConnectedSubscriberImpl.java:46)
	at com.google.cloud.pubsublite.internal.wire.ConnectedSubscriberImpl$Factory.New(ConnectedSubscriberImpl.java:39)
	at com.google.cloud.pubsublite.internal.wire.RetryingConnectionImpl.reinitialize(RetryingConnectionImpl.java:106)
	at com.google.cloud.pubsublite.internal.wire.RetryingConnectionImpl.lambda$doStart$0(RetryingConnectionImpl.java:94)

External references such as API reference guides

• ?

Any additional information below

ref:
googleapis/java-pubsublite-spark#637
#1335

Thanks!