JanusGraph Stalls On Data Ingestion Due To `Could not read messages for timestamp ... (this read will be retried)` #3778

ZachMurray · 2023-05-17T15:00:33Z

ZachMurray
May 17, 2023

How do I fix JanusGraph's java.lang.IllegalArgumentException: Timestamp overflow detected?
I just get streams of these Exception-Warnings that seem to be preventing me from completing data-upload in a timely manner.
I've simplified the problem to its base issue: g.V() and janusGraphManagement.getOrCreatePropertyKey().
If both of these lines exist within the test execution then I get this issue.
The issue persists regardless of:

which order I place them in
switching between g.V().count().next() and g.V().drop.iterate()
deleting the data (but this does reduce to only 1 time)
Changing Java versions
Rebuilding, restarting, clearing the computer, container, and IDE

Why does JanusGraph stop working and start printing these out as soon as I try adding indexes?

...
2023-05-17 09:54:02,197 [INFO] [o.j.d.l.k.KCVSLog.main] ::	 Loaded unidentified ReadMarker start time 2023-05-17T14:54:02.197877100Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@58fa5769
2023-05-17 09:54:02,267 [WARN] [o.j.g.t.StandardJanusGraphTx.main] ::	 Query requires iterating over all vertices [[]]. For better performance, use indexes
2023-05-17 09:54:02,303 [WARN] [o.j.d.l.k.KCVSLog.pool-5-thread-1] ::	 Could not read messages for timestamp [2023-05-17T14:54:02.197877100Z] (this read will be retried)
java.lang.IllegalArgumentException: Timestamp overflow detected: 2023-05-17T14:54:02.197877100Z
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog.getTimeSlice(KCVSLog.java:393) ~[janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog.access$1500(KCVSLog.java:115) ~[janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:743) [janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2023-05-17 09:54:07,317 [WARN] [o.j.d.l.k.KCVSLog.pool-5-thread-1] ::	 Could not read messages for timestamp [2023-05-17T14:54:02.197877100Z] (this read will be retried)
java.lang.IllegalArgumentException: Timestamp overflow detected: 2023-05-17T14:54:02.197877100Z
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog.getTimeSlice(KCVSLog.java:393) ~[janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog.access$1500(KCVSLog.java:115) ~[janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:743) [janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2023-05-17 09:54:12,321 [WARN] [o.j.d.l.k.KCVSLog.pool-5-thread-1] ::	 Could not read messages for timestamp [2023-05-17T14:54:02.197877100Z] (this read will be retried)
java.lang.IllegalArgumentException: Timestamp overflow detected: 2023-05-17T14:54:02.197877100Z
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog.getTimeSlice(KCVSLog.java:393) ~[janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog.access$1500(KCVSLog.java:115) ~[janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:743) [janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2023-05-17 09:54:17,323 [WARN] [o.j.d.l.k.KCVSLog.pool-5-thread-1] ::	 Could not read messages for timestamp [2023-05-17T14:54:02.197877100Z] (this read will be retried)
java.lang.IllegalArgumentException: Timestamp overflow detected: 2023-05-17T14:54:02.197877100Z
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog.getTimeSlice(KCVSLog.java:393) ~[janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog.access$1500(KCVSLog.java:115) ~[janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:743) [janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2023-05-17 09:54:22,335 [WARN] [o.j.d.l.k.KCVSLog.pool-5-thread-1] ::	 Could not read messages for timestamp [2023-05-17T14:54:02.197877100Z] (this read will be retried)
java.lang.IllegalArgumentException: Timestamp overflow detected: 2023-05-17T14:54:02.197877100Z
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog.getTimeSlice(KCVSLog.java:393) ~[janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog.access$1500(KCVSLog.java:115) ~[janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:743) [janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2023-05-17 09:54:27,350 [WARN] [o.j.d.l.k.KCVSLog.pool-5-thread-1] ::	 Could not read messages for timestamp [2023-05-17T14:54:02.197877100Z] (this read will be retried)
java.lang.IllegalArgumentException: Timestamp overflow detected: 2023-05-17T14:54:02.197877100Z
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog.getTimeSlice(KCVSLog.java:393) ~[janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog.access$1500(KCVSLog.java:115) ~[janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:743) [janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2023-05-17 09:54:32,359 [WARN] [o.j.d.l.k.KCVSLog.pool-5-thread-1] ::	 Could not read messages for timestamp [2023-05-17T14:54:02.197877100Z] (this read will be retried)
java.lang.IllegalArgumentException: Timestamp overflow detected: 2023-05-17T14:54:02.197877100Z
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog.getTimeSlice(KCVSLog.java:393) ~[janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog.access$1500(KCVSLog.java:115) ~[janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:743) [janusgraph-core-1.0.0-20230504-014643.988c094.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2023-05-17 09:54:36,521 [INFO] [Main.main] ::	 g.V().count().next():	2015811

Process finished with exit code 0

public class Test {
    private static final Logger logger = LogManager.getLogger(Main.class);

    public static void main(String[] args) throws InterruptedException, ExecutionException {
        JanusGraph janusGraph = JanusGraphFactory.build()
                .set("storage.backend", "cql")
                .set("storage.hostname", "localhost:29042")
//                .set(TIMESTAMP_PROVIDER.toStringWithoutRoot(), "NANO")
                .open();
        JanusGraphManagement janusGraphManagement = janusGraph.openManagement();
        GraphTraversalSource g = janusGraph.traversal();
        logger.info("g.V().count().next():\t" + g.V().count().next());
//        g.V().limit(10000).drop().iterate();
        PropertyKey propertyKey = janusGraphManagement.getOrCreatePropertyKey("_id");
//        GraphTraversalSource g = janusGraph.traversal();
//        logger.info("g.V().count().next():\t" + g.V().count().next());
        janusGraph.close();
    }
}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

JanusGraph Stalls On Data Ingestion Due To `Could not read messages for timestamp ... (this read will be retried)` #3778

Uh oh!

{{title}}

Uh oh!

Replies: 0 comments

Select a reply

Uh oh!

Uh oh!

JanusGraph Stalls On Data Ingestion Due To Could not read messages for timestamp ... (this read will be retried) #3778

Uh oh!

ZachMurray May 17, 2023

Replies: 0 comments

JanusGraph Stalls On Data Ingestion Due To `Could not read messages for timestamp ... (this read will be retried)` #3778

ZachMurray
May 17, 2023