Problem: non-consolidated storage architecture (#665)

Solution: wrote up a proposal

Author: tomtau

architecture-docs/adr-001.md

@@ -0,0 +1,92 @@
+# ADR 001: Storage Consolidation
+
+## Changelog
+* 10-12-2019: Initial Draft
+* 11-12-2019: Extra comments
+
+## Context
+Currently (not counting Tendermint's internal storage or wallets), two processes maintain their internal storage:
+
+* chain-abci: stores the node state, Merkle trie of staking states, transaction metadata (whether spent or not),
+validator tracking, etc.
+* tx-validation enclave (TVE): sealed transaction data (of valid obfuscated transactions that have outputs)
+
+The reason for having two processes is that SGX SDK compilation is different and needs Intel SGX SDK tooling
+(and the subsequent process execution requires Intel SGX PSW tooling, such as AESM service),
+so for the development convenience, the transaction validation code that needs to execute in an enclave
+is isolated. (For example, one can build and run chain-abci on any platform (e.g. macOS), 
+and run the enclave parts inside a docker container or on a remote Linux host.)
+The inter-process communication is over a simple REQ-REP 0MQ socket.
+
+*Problem 1: These two storage locations need to be "in sync"*:
+
+when an obfuscated transaction arrives that spends some transaction outputs, chain-abci will do a basic validation and check if they are unspent and forward it to TVE (assuming its storage contains
+sealed transaction data of respective outputs). There is currently a naive check that TVE
+stores the latest app hash provided by chain-abci; and upon a startup, chain-abci cross-checks if TVE is in sync with it. This leads to various errors and annoyances that are usually resolved by removing all storage and syncing from scratch (in certain cases, there may be a better mechanism, but wasn't implemented).
+
+*Problem 2: Transaction querying*:
+
+As wallet / client-* may be lightweight client and not have access to TEE directly, it will connect to one remotely.
+For this purpose, there is transaction query enclave (TQE). See [this document](https://github.com/crypto-com/chain-docs/blob/master/plan.md#transaction-query-enclave-tqe-optional--for-client-infrastructure) for more details.
+
+There are two flows (over an attested secure channel):
+
+1. retrieving transactions: client submits transaction identifiers signed by its view key, and TQE replies with matching transaction data. For this workflow, TQE contacts TVE over REQ-REP 0MQ socket to retrieve data.
+
+2. submitting new transactions: client submits a new transaction, TQE forwards it to TVE that checks it (so that it doesn't obfuscate random / invalid data) and if it's valid, it encrypts it with the obfuscation key (currently compile-time mock, but planned to be periodically regenerated
+by another type of enclave) and returns the obfuscated transaction to TQE that forwards it to the client.
+
+In the first flow, TQE only talks to the TVE's application wrapper that handles the persistence -- it can unseal
+the transaction data, because the key policy is MRSIGNER. 
+
+In the second flow, TVE holds the obfuscation key inside the enclave memory, so the payload goes to TVE.
+Currently, TVE cannot check everything, e.g. staked state or if a transaction output was spent or not
+-- in the future, it may internally have app hash components and at least require some lightweight proofs
+for these things.
+
+For the first flow, it's unnecessary for TQE to talk to TVE. For the second flow, it'll be desirable
+to do a more complete verification (currently there are a few hacks and workarounds).
+
+## Decision
+This will be a bit big change, so it can be done in several steps:
+
+1. separate out the storage functionality from chain-abci into chain-storage crate
+
+2. TBD: separate storage process? moving enclave app wrappers to chain-abci?
+
+3. change enclave-protocol for chain-abci/storage to add sealed transaction data, and TVE to return Ok(sealed transaction data) back to chain-abci/storage
+
+4. give TQE some way to talk to chain-abci/storage (TBD: another 0MQ socket or abci_query?)
+
+5. change the first flow of TQE to use that instead of talking to TVE
+
+6. TBD: change the second flow to go through chain-abci (it'd still need to forward to TVE)
+
+7. remove storage from TVE (as it should be all handled by chain-abci/storage)
+
+## Status
+
+Proposed
+
+## Consequences
+
+### Positive
+
+* Only one place to store transaction data -- no need to keep storage of two processes in sync
+* Decoupling state machine logic (chain-abci) from the storage
+* As the second TQE flow may demand a full validation / more storage needs, it should be easier directly talking to abci.
+
+### Negative
+* Perhaps higher latency in some flows
+* If storage runs as a service (a la Libra architecture, see below, where it is a GRPC process), it'll be an extra complexity in deployment.
+* If enclave app wrappers are moved to chain-abci, development setup may be trickier on non-Linux platforms.
+
+### Neutral
+* One more crate
+* Coupling TQE process to chain-abci or storage process
+* Storage space shared between chain-abci and "sub-abci" enclave applications: probably an extra column in RocksDB-like storage
+
+## References
+
+* moving app wrapers to chain-abci: https://github.com/crypto-com/chain/pull/665#discussion_r356377869
+* https://github.com/libra/libra/tree/master/storage/storage-service