Tx Correctness Tests behave inconsistently

[plaflamme]

The tx correctness tests have been behaving inconsistently. The following errors have been observed:

• tx already committed
• "epoch regression"

The first error is much more prominent on circle-ci builds so it seems to be related to timing (circle-ci machines are slow).

In order to investigate this, it would be important to have the full trace of the various calls to the cockroach cluster and the particular History that was being run.

Something like:

*** FAILED ***
History: I1(A) R1(A) C2 I2(A) R2(A) C2
Trace:
[eBKeqzjJ]         ->Increment      (tx=HmNMBGh8KEMgqpT7ZoUU isolation=SERIALIZABLE pri=1 state=PENDING epoch=0 ts=1433987262.190587300,0 orig=1433987262.190587300,0 max=1433987262.440587500,0) 
[eBKeqzjJ]     4ms <-Increment      (tx=HmNMBGh8KEMgqpT7ZoUU isolation=SERIALIZABLE pri=1 state=PENDING epoch=0 ts=1433987262.190587300,0 orig=1433987262.190587300,0 max=1433987262.440587500,0)  ok
[S9XnswZz]         ->Get            (tx=HmNMBGh8KEMgqpT7ZoUU isolation=SERIALIZABLE pri= state= epoch= ts= orig= max=) 
[LyBvok95]     1ms <-Get            (tx=HmNMBGh8KEMgqpT7ZoUU isolation=SERIALIZABLE pri=1 state=PENDING epoch=0 ts=1433987262.190587300,0 orig=1433987262.190587300,0 max=1433987262.440
[ekqRlhUU]         ->EndTransaction (tx=HmNMBGh8KEMgqpT7ZoUU isolation=SERIALIZABLE pri=1 state=PENDING epoch=0 ts=1433987262.190587300,0 orig=1433987262.190587300,0 max=1433987262.440587500,0) 
[ekqRlhUU]   503ms <-EndTransaction (tx=HmNMBGh8KEMgqpT7ZoUU isolation=SERIALIZABLE pri=1 state=COMMITTED epoch=0 ts=1433987262.190587300,0 orig=1433987262.190587300,0 max=1433987262.440587500,0)  ok
[9jfbK0If]         ->Increment      (tx=QiATpEiQXFWb9BUhRwQM isolation=SERIALIZABLE pri=2 state=PENDING epoch=0 ts=1433987262.192366800,0 orig=1433987262.192366800,0 max=1433987262.442366800,0) 
[9jfbK0If]   504ms <-Increment      (tx=QiATpEiQXFWb9BUhRwQM isolation=SERIALIZABLE pri=2 state=PENDING epoch=0 ts=1433987262.192366800,0 orig=1433987262.192366800,0 max=1433987262.442366800,0)  ok
[TLUSG5mD]         ->Get            (tx=ve2XQJUalUk4xBmrBfE3 isolation=SERIALIZABLE pri=2 state=PENDING epoch=1 ts=1433987204.487029000,1 orig=1433987204.487029000,1 max=1433987204.735508000,0) 
[TLUSG5mD]  1010ms <-Get            (tx=ve2XQJUalUk4xBmrBfE3 isolation=SERIALIZABLE pri=2 state=PENDING epoch=1 ts=1433987204.487029000,1 orig=1433987204.487029000,1 max=1433987204.735508000,0)  ok
[uQBlziyJ]         ->EndTransaction (tx=QiATpEiQXFWb9BUhRwQM isolation=SERIALIZABLE pri=2 state=PENDING epoch=0 ts=1433987262.192366800,0 orig=1433987262.192366800,0 max=1433987262.442366800,0) 
[uQBlziyJ]   507ms <-EndTransaction (tx=QiATpEiQXFWb9BUhRwQM isolation=SERIALIZABLE pri=2 state=COMMITTED epoch=0 ts=1433987262.192366800,0 orig=1433987262.192366800,0 max=1433987262.442366800,0)  ok