Unable to remove a composite index from bigtable with JanusGraphManagement (Single instance janus)

[roi-tevel]

Hello,
I am trying to remove a composite index with JanusGraphManagement (with bigtable as storage backend) there is no errors, but the job doesn't seem to be removing anything:
9195551 [Thread-11] INFO org.janusgraph.graphdb.olap.job.IndexRemoveJob - Index byEntityType metrics: success-tx: 1 doc-updates: 0 succeeded: 0
9195644 [Thread-9] INFO org.janusgraph.graphdb.olap.job.IndexRemoveJob - Index byEntityType metrics: success-tx: 2 doc-updates: 0 succeeded: 0
9195645 [Thread-9] INFO org.janusgraph.graphdb.database.management.ManagementSystem - Index update job successful for [byEntityType]

after that the index remains in DISABLED status and doesn't seem to be removed from bigtable.
what could be the problem?
Thanks,
Roi

[vtslab]

Maybe checking this could help: https://docs.janusgraph.org/operations/recovery/#janusgraph-instance-failure

[roi-tevel]

Forgot to mention, we have a single instance, not a cluster

gremlin> mgmt.getOpenInstances() //all open instances
==>0a1c03311-janusgraph-7c76f5c4d8-b54621(current)

Or you mean that we can also close this single active instance without loosing data and it should solve the problem?

[vtslab]

No, closing the current instance will not help. I was guessing in the wrong direction. On second read of the ref docs I find:

Index removal deletes everything associated with the index except its schema definition and its DISABLED state. This schema stub for the index remains even after deletion, though its storage footprint is negligible and fixed.

[roi-tevel]

The problem is that nothing gets removed from bigtable, now I tried with another index and found these errors in the log:

6898168 [grpc-default-executor-150] ERROR com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation  - Could not complete RPC. Failure #0, got: Status{code=FAILED_PRECONDITION, description=Error while reading table 'projects/integration-618527513/instances/curago0/tables/janusgraph' : Read returned 284MiB from row '$=\350Q6\211\240findin\347' which exceeds the limit of 256MiB. Make sure you are setting an appropriate request filter to retrieve only recent versions and only the columns you want. If columns are accumulating more versions than you need to read, you can also create a garbage collection policy: https://cloud.google.com/bigtable/docs/configuring-garbage-collection#versions, cause=null} on channel {cbt:99,grpc:ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=227, target=bigtable.googleapis.com:443}}}.
Trailers: Metadata(grpc-server-stats-bin=AAD+46OwFwAAAA,x-goog-ext-425905942-bin=Cg5ldXJvcGUtd2VzdDItYhIGY3VyYWdv,bigtable-channel-id={cbt:99,grpc:ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=227, target=bigtable.googleapis.com:443}}})
6898171 [data-puller-0] ERROR org.janusgraph.diskstorage.keycolumnvalue.scan.MultiThreadsRowsCollector  - Could not load data from storage
java.io.UncheckedIOException: com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.IOExceptionWithStatus: Error in response stream
	at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55)
	at com.google.common.collect.Iterators$5.computeNext(Iterators.java:635)
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
	at org.janusgraph.diskstorage.hbase.HBaseKeyColumnValueStore$RowIterator.hasNext(HBaseKeyColumnValueStore.java:330)
	at org.janusgraph.diskstorage.util.MetricInstrumentedStore.runWithMetrics(MetricInstrumentedStore.java:272)
	at org.janusgraph.diskstorage.util.MetricInstrumentedIterator.hasNext(MetricInstrumentedIterator.java:73)
	at org.janusgraph.diskstorage.keycolumnvalue.scan.MultiThreadsRowsCollector$DataPuller.run(MultiThreadsRowsCollector.java:221)
Caused by: com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.IOExceptionWithStatus: Error in response stream
	at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.scanner.ResultQueueEntry$ExceptionResultQueueEntry.getResponseOrThrow(ResultQueueEntry.java:100)
	at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.scanner.ResponseQueueReader.getNextMergedRow(ResponseQueueReader.java:100)
	at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.scanner.ResponseQueueReader.getNextMergedRow(ResponseQueueReader.java:106)
	at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.scanner.ResumingStreamingResultScanner.next(ResumingStreamingResultScanner.java:78)
	at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.scanner.ResumingStreamingResultScanner.next(ResumingStreamingResultScanner.java:33)
	at com.google.cloud.bigtable.hbase.adapters.read.BigtableResultScannerAdapter$1.next(BigtableResultScannerAdapter.java:63)
	at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:53)
	... 7 more
Caused by: com.google.bigtable.repackaged.io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Error while reading table 'projects/integration-618527513/instances/curago0/tables/janusgraph' : Read returned 284MiB from row '$=\350Q6\211\240findin\347' which exceeds the limit of 256MiB. Make sure you are setting an appropriate request filter to retrieve only recent versions and only the columns you want. If columns are accumulating more versions than you need to read, you can also create a garbage collection policy: https://cloud.google.com/bigtable/docs/configuring-garbage-collection#versions
	at com.google.bigtable.repackaged.io.grpc.Status.asRuntimeException(Status.java:524)
	at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.onError(AbstractRetryingOperation.java:215)
	at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation.onClose(AbstractRetryingOperation.java:178)
	at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.scanner.RetryingReadRowsOperation.onClose(RetryingReadRowsOperation.java:231)
	at com.google.bigtable.repackaged.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at com.google.bigtable.repackaged.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at com.google.bigtable.repackaged.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.RefreshingOAuth2CredentialsInterceptor$UnAuthResponseListener.onClose(RefreshingOAuth2CredentialsInterceptor.java:82)
	at com.google.bigtable.repackaged.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at com.google.bigtable.repackaged.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at com.google.bigtable.repackaged.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.Watchdog$WatchedCall$1.onClose(Watchdog.java:188)
	at com.google.bigtable.repackaged.com.google.cloud.bigtable.grpc.io.ChannelPool$InstrumentedChannel$2.onClose(ChannelPool.java:213)
	at com.google.bigtable.repackaged.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at com.google.bigtable.repackaged.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at com.google.bigtable.repackaged.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.google.bigtable.repackaged.io.grpc.census.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:701)
	at com.google.bigtable.repackaged.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at com.google.bigtable.repackaged.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at com.google.bigtable.repackaged.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.google.bigtable.repackaged.io.grpc.census.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:398)
	at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:617)
	at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
	at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:803)
	at com.google.bigtable.repackaged.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:782)
	at com.google.bigtable.repackaged.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at com.google.bigtable.repackaged.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
6898193 [Thread-13] INFO  org.janusgraph.graphdb.olap.job.IndexRemoveJob  - Index byEntityTypeAndTask metrics: success-tx: 1 doc-updates: 0 succeeded: 0
6898283 [Thread-11] INFO  org.janusgraph.graphdb.olap.job.IndexRemoveJob  - Index byEntityTypeAndTask metrics: success-tx: 2 doc-updates: 0 succeeded: 0
6898284 [Thread-11] INFO  org.janusgraph.graphdb.database.management.ManagementSystem  - Index update job successful for [byEntityTypeAndTask]

And though removal of the index clearly failed (nothing got removed from big table due to these errors) Janus doesn't report any error.
The problem is also that Janus is saving too much data in the same row (since many vertices share the same entity_type+latest property values which are being indexed here) and so violates the limit of 256MiB row read size in bigtable

[vtslab]

Thanks for reporting this detailed behavior. Could you please file an issue for JanusGraph 1) failing silently on failing to remove a composite index 2) not respecting the maximum BigTable row size on updating a value of a Composite index.
After that you could ask yourself what could be the use of an index that returns 256MB worth of vertex identifiers. Using such an index entry in a graph traversal would probably take hours to complete. You probably want to add the property key of this index to another index so that you can lookup vertices that meet predicates on both properties, see the 'byNameAndAgeComposite' example at https://docs.janusgraph.org/schema/index-management/index-performance/#composite-index .

[roi-tevel]

Thanks, will open an issue.