JanusGraph with Google bigtable as storage backend

[kaaquist]

Hi all,

I'm having some problems with my setup, where I'm creating too many connections to BigTable admin endpoint and therefore I run into some Quota limit in writing to BigTable.

My setup is using spark to do some deduplicating and aggregation befre creating partitioning on data and then create a GraphWriter for each of the partitions. This unfortunately leads to a quota limit restriction of 100 ops pr minute to admin endpoint.
So I thought that I need to do some sort of connection pool, therefor I thought ,JanusGraph as a cluster, where I would be able to take advantage of connection pools, but i can't seem to understand what the cluster ip and port etc. should be set too in this here setup.

And is this here even the right approach or am I missing something? 😄
config settings:

    conf.setProperty("clusterConfiguration.hosts", "<bigtable.googleapis.com -> not sure>")
    conf.setProperty("clusterConfiguration.port", "<not sure, maybe 443>")
    conf.setProperty("clusterConfiguration.serializer.className", "org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0")
    conf.setProperty("clusterConfiguration.serializer.config.ioRegistries", "[org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry]")
    conf.setProperty("storage.backend", "hbase")
    conf.setProperty("storage.hbase.ext.hbase.client.connection.impl", "com.google.cloud.bigtable.hbase2_x.BigtableConnection")

    conf.setProperty("storage.hbase.ext.google.bigtable.project.id", projectId)
    conf.setProperty("storage.hbase.ext.google.bigtable.instance.id", instanceId)
    conf.setProperty("graph.timestamps", "MILLI")
    conf.setProperty("storage.transactions", "false")
    conf.setProperty("storage.batch-loading", "true")
    conf.setProperty("graph.set-vertex-id", "true")
    conf.setProperty("schema.default", "none")

I hope it make sense and thanks for any thoughts.

BR. Kasper

[miguruiz]

I am having a similar issue, +1