How can I make a real-time graph management(i.e., CRUD) with HadoopGraph?

[mrleemember]

Hello, guys.

I have testing the janusgraph with hadoop on yarn(also, with hbase) for OLAP analysis.
I can only found a document to load data from a file belows,
hdfs.copyFromLocal('data/tinkerpop-modern.kryo', 'tinkerpop-modern.kryo')
But, is there no method to manage the graph real time(e.g., use g.V().addV(...))?

I used a SparkGraphComputer with hadoop graph configulation like,

gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
gremlin.hadoop.graphReader=org.janusgraph.hadoop.formats.hbase.HBaseInputFormat
gremlin.hadoop.graphWriter=org.apache.hadoop.mapreduce.lib.output.NullOutputFormat

Also, I have referenced the source code of HadoopGraph.java and there is no support for add vertex.
@OverRide
public Vertex addVertex(final Object... keyValues) {
throw Exceptions.vertexAdditionsNotSupported();
}

On Janusgraph with spark on yarn, please, how can I mange the graph in realtime?
I need the analysis framework for our streaming data.

[cdegroc]

Hi @mrleemember,

If you want to execute many traversals in parallel using Spark, a simple approach could be to just open the graph in each executor and execute traversals from there, just as you would do for OLTP.

Hope that helps.

[mrleemember]

Thanks bro.
But, is there a way to build a graph with spark, not traversals?
I need to build a graph continuosly with streaming data.
Building a graph with a data loaded from file is not enough for my case.

I tried to add vertext on gremlin console with a graph function, addVertex() from HadoopGraph, but, it is not supported.
Is there any recommandation for this?

[cdegroc]

But, is there a way to build a graph with spark, not traversals?

Not that I'm aware of. There is no HBaseOutputFormat for instance. But other devs might chime in.
Though I am not sure why you'd want to avoid traversals. To the best of my knowledge, traversals are executed locally within Spark executors so it's likely the traversal execution will just end up calling addVertex under the hood.

I tried to add vertext on gremlin console with a graph function, addVertex() from HadoopGraph, but, it is not supported.

Indeed, HadoopGraph does not support addVertex(). Other graph implementations (such as StandardJanusGraph) do. So, if you really want to go down that path, you can probably change your configuration to open the graph using a different class (e.g. JanusGraphFactory). I'm not sure how that will run with Hadoop/Spark though.

[mrleemember]

Thanks again.

I tried JanusGraphFactory first(gremlin.graph=org.janusgraph.core.JanusGraphFactory), but it's failed.

gremlin> graph = GraphFactory.open('conf/hadoop-graph/read-hbase-spark.properties')
GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]

Also, tried to use HadoopGraph and traversal(is this what did you mean?) but, also failed.

gremlin> graph = GraphFactory.open('conf/hadoop-graph/read-hbase-spark.properties')
==>hadoopgraph[hbaseinputformat->nulloutputformat]
gremlin> g = graph.traversal()
==>graphtraversalsource[hadoopgraph[hbaseinputformat->nulloutputformat], standard]
gremlin> g.addV("test").property("name","testname")
Graph does not support adding vertices

really appreciate your help.

[vtslab]

If you want to open a graph from a sprak executor, you have to use janusgraph-hbase-es.properties as a template, not read-hbase.properties.

[vtslab]

If you want to use janusgraph-hbase for traversals from a spark executor you have to make the connection as:
graph = GraphFactory.open('conf/janusgraph-hbase-es.properties')