NoSuchElementException using jaeger v2 and cassandra

[pxpdupix]

i have set up a Jaeger 2.10.0 using Cassandra storage. I'm configuring Jaeger to auto create the schema.
I'm trying to run the spark-dependencies job against the Cassandra keyspace, but I'm hitting the error described below. I’d like to confirm if I'm doing something wrong as I have look for this error and it seems im the only one having it 😄
Jaeger.yaml

...
      jaeger_storage:
        backends:
          cassandra_primary:
            cassandra:
              schema:
                create: true
                keyspace: "jaeger_vnode"
                datacenter: "datacenter1"
                replication_factor: 1
                disable_compression: true
                trace_ttl: 24h
                dependencies_ttl: 24h
                compaction_window: 2h
 ...
       containers:
        - name: jaeger
          image: jaegertracing/jaeger:2.10.0

The resulting schema is

cqlsh> DESCRIBE TABLE jaeger_vnode.traces;

CREATE TABLE jaeger_vnode.traces (
    trace_id blob,
    span_id bigint,
    span_hash bigint,
    duration bigint,
    flags int,
    logs list<frozen<log>>,
    operation_name text,
    parent_id bigint,
    process frozen<process>,
    refs list<frozen<span_ref>>,
    start_time bigint,
    tags list<frozen<keyvalue>>,
    PRIMARY KEY (trace_id, span_id, span_hash)
) WITH CLUSTERING ORDER BY (span_id ASC, span_hash ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
    AND comment = ''
    AND compaction = {'class': 'TimeWindowCompactionStrategy', 'compaction_window_size': '120', 'compaction_window_unit': 'MINUTES'}
    AND compression = {'sstable_compression': 'LZ4WithDictsCompressor'}
    AND crc_check_chance = 1
    AND default_time_to_live = 86400
    AND gc_grace_seconds = 10800
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND speculative_retry = 'NONE'
    AND tombstone_gc = {'mode': 'timeout', 'propagation_delay_in_seconds': '3600'};

When I execute the spark-dependencies job i get this error

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in stage 0.0 failed 1 times, most recent failure: Lost task 5.0 in stage 0.0 (TID 5) (jaeger-spark-dependencies-test-pj68m executor driver): java.util.NoSuchElementException
	at com.datastax.oss.driver.internal.core.util.CountingIterator.next(CountingIterator.java:102)
	at com.datastax.spark.connector.rdd.reader.PrefetchingResultSetIterator.next(PrefetchingResultSetIterator.scala:51)
	at com.datastax.spark.connector.rdd.reader.PrefetchingResultSetIterator.next(PrefetchingResultSetIterator.scala:22)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:496)
	at com.datastax.spark.connector.util.CountingIterator.next(CountingIterator.scala:16)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:169)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
	at org.apache.spark.scheduler.Task.run(Task.scala:141)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:621)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:624)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2898)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2834)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2833)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2833)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1253)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1253)
	at scala.Option.foreach(Option.scala:407)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1253)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3102)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3036)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3025)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:995)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2393)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2414)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2433)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2458)
	at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1049)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:410)
	at org.apache.spark.rdd.RDD.collect(RDD.scala:1048)
	at org.apache.spark.api.java.JavaRDDLike.collect(JavaRDDLike.scala:362)
	at org.apache.spark.api.java.JavaRDDLike.collect$(JavaRDDLike.scala:361)
	at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
	at io.jaegertracing.spark.dependencies.DependenciesSparkHelper.derive(DependenciesSparkHelper.java:35)
	at io.jaegertracing.spark.dependencies.cassandra.CassandraDependenciesJob.run(CassandraDependenciesJob.java:175)
	at io.jaegertracing.spark.dependencies.cassandra.CassandraDependenciesJob.main(CassandraDependenciesJob.java:305)
Caused by: java.util.NoSuchElementException
	at com.datastax.oss.driver.internal.core.util.CountingIterator.next(CountingIterator.java:102)
	at com.datastax.spark.connector.rdd.reader.PrefetchingResultSetIterator.next(PrefetchingResultSetIterator.scala:51)
	at com.datastax.spark.connector.rdd.reader.PrefetchingResultSetIterator.next(PrefetchingResultSetIterator.scala:22)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:496)
	at com.datastax.spark.connector.util.CountingIterator.next(CountingIterator.scala:16)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:169)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
	at org.apache.spark.scheduler.Task.run(Task.scala:141)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:621)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:624)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

I was playing around and I tried replacing the .where(...) with a client-side filter:

.cassandraTable(keyspace, "traces", mapRowTo(CassandraSpan.class))
.filter(span -> span.getStartTime() > microsLower && span.getStartTime() < microsUpper)

…the job completes successfully

Thanks

[dosubot[bot]]

Hi @pxpdupix! I'm Dosu and I’m helping the jaeger team.

Your Jaeger auto-created Cassandra schema matches the expected structure and should be compatible with spark-dependencies in standard Cassandra. The NoSuchElementException you’re seeing is often caused by query mismatches or empty result sets. Since switching from a server-side .where(...) to a client-side .filter(...) resolves the error, it suggests the issue may be with how the spark-dependencies job constructs its CQL query—possibly referencing fields or using a filter that doesn’t match the schema or returns no rows in the selected time window.

A few things to check:

• Make sure there is data in the relevant time window in the traces table; an empty result can trigger this exception in Spark jobs.
• Compare the actual CQL query generated by spark-dependencies (the .where(...) clause) to your schema to ensure all referenced fields exist and match types.
• Using client-side filtering is a valid workaround if it works for you, though it may be less efficient for large datasets.

There are no known incompatibilities between Jaeger’s auto-created schema and spark-dependencies in standard Cassandra, but similar issues have been reported with Cassandra-like backends (like CosmosDB) where server-side filtering or ordering isn’t fully supported, and client-side filtering is required instead (example).

If you can share the exact .where(...) clause or more details about the spark-dependencies job’s configuration, I can help dig deeper. Otherwise, continuing with client-side filtering is a reasonable approach.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}