Unable to Use Custom Long IDs in JanusGraph: Validity Constraints and Workarounds

[MateusMelloSouza]

I'm working with JanusGraph and encountering challenges when attempting to use custom Long-type IDs for vertices. Here's a detailed breakdown of my situation:

1. What I Want to Do:

   I aim to add at least 3 billion vertex nodes, each with a unique custom ID of type Long, into JanusGraph. My setup includes:

   • Backend Storage: Two Cassandra machines.
   • JanusGraph Machine: A single instance.

   The goal is to utilize custom Long IDs to efficiently manage and reference a large number of vertices within this environment.

2. What I've Done and the Problems Encountered:

   • Configuration Adjustments:

     • Enabled custom vertex IDs by setting graph.set-vertex-id=true in the JanusGraph configuration.
     • Attempted to expand the ID bit range by tweaking settings like cluster.max-partitions and ids.authority.conflict-avoidance-bitwidth.

   • Adding Vertices with Gremlin:

     • Successfully added vertices with smaller Long IDs:

       g.addV().property(T.id, 2048).property("name", "domain").property("value", "www.google.com").next()
       g.addV().property(T.id, 123456).property("name", "domain").property("value", "www.google.com").next()

     • Encountered errors when adding certain Long IDs:

       g.addV().property(T.id, 5678).property("name", "domain").property("value", "www.google.com").next()

       Error:

       GremlinServerError: 500: Not a valid vertex id: 5678
       

   • Handling IDs Beyond Long Range:

     • Attempted to add a vertex with an ID exceeding the Long range:

       g.addV("Person").property(T.id, 9223372036854775808)

       Error:

       java.lang.IllegalArgumentException
       

     • Tried using graph.getIDManager().toVertexId() with a BigInteger:

       g.addV("Person").property(T.id, graph.getIDManager().toVertexId(9223372036854775808))

       Error:

       No signature of method: org.janusgraph.graphdb.idmanagement.IDManager.toVertexId() is applicable for argument types: (BigInteger) values: [9223372036854775808]

     • Used toVertexId with a long suffix:

       g.addV("Person").property(T.id, graph.getIDManager().toVertexId(9223372036854775808L))

       Error:

       groovysh_evaluate: 3: Number of value 9223372036854775808 does not fit in the range of long, but long was enforced. @ line 3, column 66.
          graph.getIDManager().toVertexId(9223372036854775808L)
                                        ^
       
       1 error
       

     • Tried a slightly smaller Long:

       g.addV("Person").property(T.id, graph.getIDManager().toVertexId(9223372036854775805L))
       

       Error:

       Vertex id is too large: 9223372036854775805

3. How Can I Overcome These Issues?

   • Using Long IDs:
     • Validity Constraints: Given the inconsistencies in accepting certain Long IDs (e.g., 5678 is rejected while 2048 and 123456 are accepted), what defines a valid Long ID in JanusGraph? Are there hidden constraints or required configurations that I'm missing?
     • ID Range Limitations: Is JanusGraph inherently limited in the range of Long IDs it can accept for custom vertex IDs? If so, what is the maximum permissible value?
   • Alternative ID Types:
     • String IDs: If Long IDs are not feasible for my scale, what are the potential drawbacks of switching to String-based IDs? How might this impact performance or storage?
     • Managing Larger or Random IDs: What are the best practices for managing random or larger IDs within JanusGraph if opting for alternative ID types? Are there recommended strategies to ensure uniqueness and efficiency?
   • Configuration and Strategies:
     • Optimizing JanusGraph Settings: Beyond the configurations I've already attempted (cluster.max-partitions and ids.authority.conflict-avoidance-bitwidth), are there other settings or strategies that can help accommodate the required number of unique Long IDs?
     • Backend Storage Considerations: Could adjustments or optimizations on the Cassandra backend contribute to resolving these ID-related issues?
     • Alternative ID Management Approaches: Are there recommended patterns or external systems for managing and generating unique IDs that integrate well with JanusGraph?

Any insights, recommendations, or best practices to resolve these ID-related challenges in JanusGraph would be greatly appreciated!

[porunov]

Please, check the following limitations: https://docs.janusgraph.org/advanced-topics/technical-limitations/#size-limitation

When you are creating your Long vertex ID you should follow the following rules:

1. Always use graph.getIDManager().toVertexID(123L) to convert your vertex ID into the internal vertex id type. (see documentation).
2. You vertex ID should be a positive value not exceeding (2^60)/2. In other words, you can create up to 576 quadrillion vertices and up to 1.1 quintillion edges.

You was trying to use value 9223372036854775805 which is larger then 576460752303423488. Try to use smaller values in range 0 to 576460752303423488 for vertex ids.

I believe we should clarify this in documentation.

[MateusMelloSouza]

Thank you very much for your prompt and fast response.

I'm dealing with a very specific case. The ID I mentioned, 9223372036854775805, is a unique identifier that I've generated from other databases and is used within a larger system for uniquely identifying records. My intention in using it as a custom ID was to gain performance benefits in path traversal queries, such as:

g.V(9223372036854775805)
.until(.hasId(622337203685477580))
.repeat(.both().simplePath())
.limit(1)
.valueMap(true)
.path()
.profile()

Given this scenario, does JanusGraph offer any workaround to support larger custom IDs like this? Is there a way to adjust the configurations to accommodate IDs beyond the current limit? Or do you have any tips on how I might handle this situation?

[porunov]

JanusGraph supports String ids up to 512 characters in length: https://docs.janusgraph.org/advanced-topics/custom-vertex-id/#custom-string-id
In this case you can convert your generated Long id into the string by calling String.valueOf(9223372036854775805L).
Alternatively, you could also serialize your Long into Base64 String which also decreases the length of the generated string.
For example: Base64.getEncoder().withoutPadding().encodeToString(Longs.toByteArray(9223372036854775805L)).
To convert that string back to Long you would use a base64 decoder like: Longs.fromByteArray(Base64.getDecoder().decode(yourBase64String)).