I don't know how this works, but does this suggest there are 2 running for download 19298?

| small-dwnld-0019298-260208012135463-6e3cec9c5e11aee5-driver                 ●                 2/2                  Running                                       0 192.168.196.74                 c8n3.gbif.org                  20h                    │
│ small-dwnld-0019298-260208012135463-8e41c69c61816894-driver                 ●                 2/2                  Running                                       0 192.168.254.48                 c8n12.gbif.org                 4h9m                   │
│ small-dwnld-0019298-260208012135463-bq47r                                   ●                 0/1                  Error                                         0 192.168.177.53                 c7n4                           20h                    │
│ small-dwnld-0019298-260208012135463-zkq9g                                   ●                 1/1                  Running                                       0 192.168.55.32                  c7n3.gbif.org                  4h9m

Both drivers are sitting with similar logs reporting an error, followed by a Lock release and then hanging:

[ERROR] [02/14/2026 21:32:47.623] [DownloadSystem0019298-260208012135463-akka.actor.default-dispatcher-8] [akka://DownloadSystem0019298-260208012135463/user/DownloadMaster0019298-260208012135463/downloadWorkerRouter] null
java.util.ConcurrentModificationException
	at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1221)
	at org.gbif.dwc.terms.TermFactory.addTerm(TermFactory.java:193)
	at org.gbif.dwc.terms.TermFactory.createUnknownTerm(TermFactory.java:304)
	at org.gbif.dwc.terms.TermFactory.findTerm(TermFactory.java:241)
	at org.gbif.search.es.SearchHitConverter.mapTerm(SearchHitConverter.java:340)
	at org.gbif.search.es.occurrence.SearchHitOccurrenceConverter.lambda$extractVerbatimFields$3(SearchHitOccurrenceConverter.java:138)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1850)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at org.gbif.search.es.occurrence.SearchHitOccurrenceConverter.extractVerbatimFields(SearchHitOccurrenceConverter.java:142)
	at org.gbif.search.es.occurrence.SearchHitOccurrenceConverter.apply(SearchHitOccurrenceConverter.java:116)
	at org.gbif.search.es.occurrence.SearchHitOccurrenceConverter.apply(SearchHitOccurrenceConverter.java:72)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at org.gbif.search.es.EsResponseParser.parseHits(EsResponseParser.java:169)
	at org.gbif.search.es.EsResponseParser.buildSearchResponse(EsResponseParser.java:66)
	at org.gbif.occurrence.download.file.common.SearchQueryProcessor.consume(SearchQueryProcessor.java:97)
	at org.gbif.occurrence.download.file.common.SearchQueryProcessor.processQuery(SearchQueryProcessor.java:82)
	at org.gbif.occurrence.download.file.simplecsv.SimpleCsvDownloadActor.doWork(SimpleCsvDownloadActor.java:95)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
	at scala.PartialFunction.applyOrElse(PartialFunction.scala:127)
	at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126)
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175)
	at akka.actor.Actor.aroundReceive(Actor.scala:537)
	at akka.actor.Actor.aroundReceive$(Actor.scala:535)
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
	at akka.actor.ActorCell.invoke(ActorCell.scala:548)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
	at akka.dispatch.Mailbox.run(Mailbox.scala:231)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

2026-02-15T13:33:40,882 INFO [DownloadSystem0019298-260208012135463-akka.actor.default-dispatcher-6] org.gbif.occurrence.download.file.simplecsv.SimpleCsvDownloadActor - Lock released, job detail: DownloadFileWork{from=572, to=858, baseDataFileName=/tmp//0019298-260208012135463/0019298_260208012135463, jobId=2, query={
  "bool" : {
    "filter" : [
      {
        "bool" : {

Small occurrence download hangs at the end of the spark job #457

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions