After System.exit, closing storage adapter intermittently enters endless loop

[davisdk]

We have a class that performs some graph operations in the main method and then uses System.exit. This usually works fine, but we have started seeing an exception repeated indefinitely in the logs for some runs (so the containing job is not terminating). It looks like StandardJanusGraph is trying to close the graph instance associated with the storage manager, but then repeatedly spits out the trace below.

In this particular case we are using the FDB adapter (https://github.com/JanusGraph/janusgraph-foundationdb). The isolation level we are using for the adapter allows 3 retries and then throws a PermanentBackendException. We are observing this in the logs, where it attempts three times and then throws the exception, but the process immediately starts over again.

We have tried debugging and tweaking some settings with no luck, so wanted to reach out to see if we can get any insight. We've also opened an issue in the adapter repo. Thank you!

[pool-37-thread-1] WARN org.janusgraph.diskstorage.log.kcvs.KCVSLog - Could not read messages for timestamp [2021-05-11T02:21:21.872Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
	at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56)
	at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:158)
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:725)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Permanent exception while executing backend operation messageReading@0:0
	at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:81)
	at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54)
	... 9 more
Caused by: java.util.concurrent.CompletionException: com.apple.foundationdb.FDBException: Broken promise
	at java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
	at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
	at com.apple.foundationdb.RangeQuery$AsyncRangeIterator.hasNext(RangeQuery.java:250)
	at org.janusgraph.diskstorage.foundationdb.FoundationDBRecordIterator.fetchNext(FoundationDBRecordIterator.java:64)
	at org.janusgraph.diskstorage.foundationdb.FoundationDBRecordAsyncIterator.fetchNext(FoundationDBRecordAsyncIterator.java:66)
	at org.janusgraph.diskstorage.foundationdb.FoundationDBRecordIterator.hasNext(FoundationDBRecordIterator.java:47)
	at org.janusgraph.diskstorage.util.StaticArrayEntryList.of(StaticArrayEntryList.java:390)
	at org.janusgraph.diskstorage.util.StaticArrayEntryList.ofStaticBuffer(StaticArrayEntryList.java:345)
	at org.janusgraph.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreAdapter.convert(OrderedKeyValueStoreAdapter.java:145)
	at org.janusgraph.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreAdapter.getSlice(OrderedKeyValueStoreAdapter.java:69)
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:791)
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:788)
	at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147)
	at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161)
	at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68)
	... 10 more
Caused by: com.apple.foundationdb.FDBException: Broken promise
	at com.apple.foundationdb.NativeFuture.Future_getError(Native Method)
	at com.apple.foundationdb.FutureResults.getIfDone_internal(FutureResults.java:38)
	at com.apple.foundationdb.FutureResults.getIfDone_internal(FutureResults.java:25)
	at com.apple.foundationdb.NativeFuture.marshalWhenDone(NativeFuture.java:63)
	... 3 more

[baiwfg2]

Hi @davisdk You're still using janusgraph-foundationdb ? Any advantages over other storage backends ?

[porunov]

@davisdk I believe this was fixed in #2769 .
Let us know if you still experience this issue in JanusGraph 0.6.0