Speed up MapReduceIndexManagement with Spark/Hadoop

[VertigoStr]

Hello!

I have a cluster of ApacheSpark and ApacheHadoop. I would like to speed up the process of indexing data using Spark and Hadoop.

Data indexing takes up to 14 days on our data (20 billions of vertex and 50 billions of edges) by running the following code:

mgmt = graph.openManagement(); 
mr = new org.janusgraph.hadoop.MapReduceIndexManagement(graph);
mr.updateIndex(mgmt.getGraphIndex("device_id_hash_indx"), SchemaAction.REINDEX).get()
mgmt.commit();

While studying the source code of MapReduceIndexManagement I noticed that it is possible to pass configuration for connection to hadoop:

public class MapReduceIndexManagement {

public ScanJobFuture updateIndex(Index index, SchemaAction updateAction) throws BackendException {
        return updateIndex(index, updateAction, new Configuration());
}

Help me, is it possible:

1. To use hadoop/apache spark to speed up indexing?
2. How to specify in janusgraph configurations data about apache spark for use in MapReduceIndexManagement?

Our janusgraph config for graph:

gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cql
storage.hostname=some_ip
storage.port=some_port
storage.cql.keyspace=janusgraph
storage.cql.local-datacenter=DC1
cache.db-cache = false
storage.username=some_user_name
storage.password=some_password

ids.block-size=2000000
ids.renew-percentage=0.3
storage.batch-loading=true

index.search.backend=elasticsearch
index.search.elasticsearch.create.ext.number_of_shards=15
index.search.elasticsearch.create.ext.number_of_replicas=3
index.search.elasticsearch.create.ext.shard.check_on_startup=true

index.search.elasticsearch.http.auth.basic.password=some_password
index.search.elasticsearch.http.auth.basic.username=some_user_name

index.search.hostname=some_ip
index.search.port=som_port


spark.master=spark://some_path
spark.executor.memory=10g
spark.executor.extraClassPath=/etc/janusgraph/janusgraph-1.1.0/lib/*
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator=org.janusgraph.hadoop.serialize.JanusGraphKryoRegistrator


gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
gremlin.hadoop.graphReader=org.janusgraph.hadoop.formats.cql.CqlInputFormat
gremlin.hadoop.graphWriter=org.apache.hadoop.mapreduce.lib.output.NullOutputFormat

gremlin.hadoop.jarsInDistributedCache=true
gremlin.hadoop.inputLocation=none
gremlin.hadoop.outputLocation=output
gremlin.spark.persistContext=true

janusgraphmr.ioformat.conf.storage.backend=cql
janusgraphmr.ioformat.conf.storage.cql.local-datacenter=DC2-nb
janusgraphmr.ioformat.conf.storage.cassandra.local-datacenter=DC2-nb
# This specifies the hostname & port for Cassandra data store.
janusgraphmr.ioformat.conf.storage.hostname=some_ip
janusgraphmr.ioformat.conf.storage.cassandra.username=some_username
janusgraphmr.ioformat.conf.storage.cassandra.password=some_password
janusgraphmr.ioformat.conf.storage.port=some_port
# This specifies the keyspace where data is stored.
janusgraphmr.ioformat.conf.storage.cql.keyspace=janusgraph
# This defines the indexing backend configuration used while writing data to JanusGraph.
janusgraphmr.ioformat.conf.index.search.backend=elasticsearch
janusgraphmr.ioformat.conf.index.search.hostname=some_ip
janusgraphmr.ioformat.conf.index.search.elasticsearch.http.auth.basic.password=some_password
janusgraphmr.ioformat.conf.index.search.elasticsearch.http.auth.basic.username=some_username

cassandra.input.partitioner.class=org.apache.cassandra.dht.Murmur3Partitioner
cassandra.input.widerows=true