PermanentLockingException while commit

[pinal-shah]

Hello,

Issue 1:
vertex A, which has edge associated with vertex B.
If I search vertex A by filtering with __guid property, i am not able to get that vertex. But if i search it through vertex B (g.V('B').in().has('__guid','guid of A')) , i am able to get the vertex. This scenario is specific to only this vertex A, for other vertices it is normal.
Note: Other properties works fine. I can get the vertex by searching with other property.

Issue 2:
If i try to update the property '__guid' of vertex A, I am getting PermanantLockException while commit()

Janusgraph version: v0.6.1
Env: hbasev2.4.6, solr v8.4.1

Config:
cache.db-cache=true
cache.db-cache-clean-wait=20
cache.db-cache-size=0.5
cache.tx-cache-size=15000
cache.tx-dirty-size=120

Observation:

Vertex Index Name              | Type        | Unique    | Backing        | Key:           Status |

__guid                         | Composite   | true      | internalindex  | __guid:       ENABLED |

gremlin> g.V().has('__guid','95524ff5-79cf-4268-b687-dc3b6f5e8fa1')
---- vertex A not found

gremlin> g.V().has('__guid','75bfe71b-6834-4d9c-89e1-b3548b23277e')
==>v[87843032]
----- vertex B found

gremlin> g.V().has('__guid','75bfe71b-6834-4d9c-89e1-b3548b23277e').in().has('__guid','95524ff5-79cf-4268-b687-dc3b6f5e8fa1')
==>v[4457615456]
----- vertex A found

gremlin> g.V(4457615456)
==>v[4457615456]

gremlin> g.V(4457615456).valueMap('__guid')
==>[__guid:[95524ff5-79cf-4268-b687-dc3b6f5e8fa1]]
------ vertex A __guid, which is exactly same when tried in the 1st step

gremlin> g.V(4457615456).property('__guid', '95524ff5-79cf-4268-b687-dc3b6f5e8fa1')
==>v[4457615456]
------ trying to reset the __guid value of vertex A

gremlin> g.V().has('__guid','95524ff5-79cf-4268-b687-dc3b6f5e8fa1')
==>v[4457615456]
------ After resetting , I can get the vertex through this property, which is exactly same as 1st step

gremlin> g.tx().commit()
org.janusgraph.core.JanusGraphException: Could not commit transaction due to exception during persistence
	at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1536)
	at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:322)
	at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104)
	at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.commit(JanusGraphBlueprintsGraph.java:300)
	at org.apache.tinkerpop.gremlin.structure.Transaction$commit.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:119)
	at groovysh_evaluate.run(groovysh_evaluate:3)
	at groovysh_evaluate$run.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at groovysh_evaluate$run.call(Unknown Source)
	at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:82)
	at org.codehaus.groovy.tools.shell.Evaluator$evaluate.call(Unknown Source)
	at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:201)
	at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.super$3$execute(GremlinGroovysh.groovy)
	at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:144)
	at org.apache.tinkerpop.gremlin.console.GremlinGroovysh.execute(GremlinGroovysh.groovy:83)
	at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:120)
	at org.codehaus.groovy.tools.shell.Shell$leftShift$2.call(Unknown Source)
	at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:93)
	at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:144)
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:164)
	at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:138)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:160)
	at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:57)
	at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1217)
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:144)
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:164)
	at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:97)
	at java_lang_Runnable$run.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:119)
	at org.apache.tinkerpop.gremlin.console.Console.<init>(Console.groovy:170)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:80)
	at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:59)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:237)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:265)
	at org.apache.tinkerpop.gremlin.console.Console.main(Console.groovy:524)
Caused by: org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
	at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54)
	at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:96)
	at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:153)
	at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:210)
	at org.janusgraph.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:136)
	at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:790)
	at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1525)
	... 72 more
Caused by: org.janusgraph.diskstorage.locking.PermanentLockingException: Expected value mismatch for KeyColumn [k=0x0xF8B439B50A89A039353532346666352D373963662D343236382D623638372D6463336236663565386661B1, c=0x0x00]: expected=[0x104D4720E0] vs actual=[] (store=graphindex)
	at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.checkSingleExpectedValueUnsafe(ExpectedValueCheckingTransaction.java:279)
	at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.access$000(ExpectedValueCheckingTransaction.java:51)
	at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction$1.call(ExpectedValueCheckingTransaction.java:222)
	at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction$1.call(ExpectedValueCheckingTransaction.java:219)
	at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:66)
	at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.checkSingleExpectedValue(ExpectedValueCheckingTransaction.java:219)
	at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.checkAllExpectedValues(ExpectedValueCheckingTransaction.java:193)
	at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.prepareForMutations(ExpectedValueCheckingTransaction.java:158)
	at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:77)
	at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:99)
	at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:96)
	at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:66)
	at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:52)
	... 78 more
------ when tried committing 

Even while dropping the vertex, encountered same exception.
Note: There are no other open instances

Can anyone help, what an cause this issue? Also how can i reset it?

[pinal-shah]

One thing i missed,
there were many other open instances, but i force closed instances other than current.

[pinal-shah]

@porunov Can you please help shade lights on this?