Merge branch 'master' into feature/nodejs-ci

Author: Artemkaaas

cli/README.md

@@ -32,7 +32,7 @@ by beginning the line with a `#`.
 
 ### Getting help
 The most simple way is just start cli by `indy-cli` command and put `help` command. Also you can look to
-[Indy CLI Design](../doc/cli-design.md) doc that contains the list of commands and architecture overview.
+[Indy CLI Design](../doc/design/001-cli) doc that contains the list of commands and architecture overview.
 
 ### Old python-based CLI migration
 It is possible to import did's stored in the wallet of deprecated python-based CLI tool.

doc/design/008-state-proof-pluggable-parsing/README.md

@@ -0,0 +1,133 @@
+## Legend
+There are some types of requests to Nodes in the Pool allowing to use StateProof optimization in Client-Node communication.
+Instead of sending requests to all nodes in the Pool, a client can send a request to a single Node and expect StateProof signed by BLS multi-signature.
+
+BLS multi-signature (BLS MS) guaranties that there was a consensus of Nodes which signed some State identified by State RootHash.
+StateProof (SP) is small amount of data which allows to verify particular values against RootHash.
+Combination of BLS MS and SP allows clients to be sure about response of single node is a part of State signed by enough Nodes. 
+
+## Goals
+Libindy allows to extend building of supported requests via plugged interface and send them.
+It is nice to have a way to support BLS MS and SP verification for these plugged transactions.
+
+Implementation of math for SP verification is a bit complicate for plugin logic.
+Therefore libindy should perform all math calculation inside.
+A plugin should provide handler to parse custom reply to fixed data structure.
+
+## API
+The signature of the handler is described below together with custom `free` call to deallocate result data.
+
+```rust
+extern fn CustomTransactionParser(reply_from_node: *const c_char, parsed_sp: *mut *const c_char) -> ErrorCode;
+extern fn CustomFree(data: *mut c_char) -> ErrorCode;
+``` 
+
+Libindy API will contain call to register handler for specific transaction type:
+```rust
+extern fn indy_register_transaction_parser_for_sp(command_handle: i32,
+                                                  pool_handle: i32,
+                                                  txn_type: *const c_char,
+                                                  parser: CustomTransactionParser,
+                                                  free: CustomFree,
+                                                  cb: extern fn(command_handle_: i32, err: ErrorCode)) -> ErrorCode;
+```
+
+### Parsed Data structure
+
+A plugin should parse `reply_from_node` and return back to libindy parsed data as JSON string.
+Actually this data is array of entities each of them is describe SP Trie and set of key-value pairs to verify against this trie.
+It can be represented as `Vec<ParsedSP>` serialized to JSON.
+
+
+```rust
+/**
+ Single item to verification:
+ - SP Trie with RootHash
+ - BLS MS
+ - set of key-value to verify
+*/
+struct ParsedSP {
+    /// encoded SP Trie transferred from Node to Client
+    proof_nodes: String,
+    /// RootHash of the Trie, start point for verification. Should be same with appropriate filed in BLS MS data
+    root_hash: String,
+    /// entities to verification against current SP Trie
+    kvs_to_verify: KeyValuesInSP,
+    /// BLS MS data for verification
+    multi_signature: serde_json::Value,
+}
+
+/**
+ Variants of representation for items to verify against SP Trie
+ Right now 2 options are specified:
+ - simple array of key-value pair
+ - whole subtrie
+*/
+enum KeyValuesInSP {
+    Simple(KeyValueSimpleData),
+    SubTrie(KeyValuesSubTrieData),
+}
+
+/**
+ Simple variant of `KeyValuesInSP`.
+
+ All required data already present in parent SP Trie (built from `proof_nodes`).
+ `kvs` can be verified directly in parent trie
+*/
+struct KeyValueSimpleData {
+    kvs: Vec<(String /* b64-encoded key */, Option<String /* val */>)>
+}
+
+/**
+ Subtrie variant of `KeyValuesInSP`.
+
+ In this case Client (libindy) should construct subtrie and append it into trie based on `proof_nodes`.
+ After this preparation each kv pair can be checked.
+*/
+struct KeyValuesSubTrieData {
+    /// base64-encoded common prefix of each pair in `kvs`. Should be used to correct merging initial trie and subtrie
+    sub_trie_prefix: Option<String>,
+    kvs: Vec<(String /* b64-encoded key_suffix */, Option<String /* val */>)>,
+}
+```
+
+Expected libindy and plugin workflow is the following:
+1. Libindy receive reply from a Node, perform initial processing and pass raw reply to plugin.
+1. Plugin parse reply from the Node and specify one or more SP Trie with metadata and items for verification.
+1. Each SP Trie described by plugin as `ParsedSP`:
+    1. Set of encoded nodes of the SP Trie, received from Node - `proof_nodes`. May be fetched from response "as is".
+    1. RootHash of this Trie. May be fetched from the response "as is" also.
+    1. BLS MS data. Again may be fetched from the response "as is".
+    1. Key-value items to verification. Here plugin should define correct keys (path in the trie) and corresponded values.
+1. Plugin return serialized as JSON array of `ParsedSP`
+1. For each `ParsedSP` libindy:
+    1. build base trie from `proof_nodes`
+    1. if items to verify is `SubTrie`, construct this subtrie from (key-suffix, value) pairs and merge it with trie from clause above
+    1. iterate other key-value pairs and verify that trie (with signed `root_hash`) contains `value` at specified `key`
+    1. verify multi-signature
+1. If any verification is failed, libindy ignore particular SP + BLS MS and try to request same data from another node,
+or collect consensus of same replies from enough count of Nodes.
+
+
+Below is JSON structure for `Simple` case.
+```json
+[
+ {
+   "proof_nodes": "string with serialized SP tree",
+   "root_hash": "string with root hash",
+   "kvs_to_verify": {
+     "type": "simple",
+     "kvs": [["key1", "value1"], ["key2", "value2"]]
+   },
+   "multi_signature": "JSON object from Node`s reply as is"
+ }
+]
+```
+
+### Simple and SubTrie verification
+
+Some use cases require verification multiply pairs of key-value in one Trie.
+Moreover there is possible situation when client would like to verify whole subtrie.
+In this case, the amount of data transferred from Node to Client can be significantly reduced.
+Instead of including all nodes for SP verification to `proof_nodes`, Node can include only prefix path down to subtrie.
+The entire subtrie to verification can be restored on Client side from key-value pairs and combined with prefix part.

libindy/src/api/wallet.rs

@@ -150,7 +150,7 @@ pub extern fn indy_register_wallet_storage(command_handle: i32,
 ///   {
 ///       "storage": <object>  List of supported keys are defined by wallet type.
 ///   }
-/// credentials: Wallet credentials json (if NULL, then default config will be used).
+/// credentials_json: Wallet credentials json (if NULL, then default config will be used).
 ///   {
 ///       "key": string,
 ///       "rekey": Optional<string>,
@@ -170,29 +170,29 @@ pub extern fn indy_create_wallet(command_handle: i32,
                                  name: *const c_char,
                                  storage_type: *const c_char,
                                  config: *const c_char,
-                                 credentials: *const c_char,
+                                 credentials_json: *const c_char,
                                  cb: Option<extern fn(xcommand_handle: i32,
                                                       err: ErrorCode)>) -> ErrorCode {
-    trace!("indy_create_wallet: >>> pool_name: {:?}, name: {:?}, storage_type: {:?}, config: {:?}, credentials: {:?}",
-           pool_name, name, storage_type, config, credentials);
+    trace!("indy_create_wallet: >>> pool_name: {:?}, name: {:?}, storage_type: {:?}, config: {:?}, credentials_json: {:?}",
+           pool_name, name, storage_type, config, credentials_json);
 
     check_useful_c_str!(pool_name, ErrorCode::CommonInvalidParam2);
     check_useful_c_str!(name, ErrorCode::CommonInvalidParam3);
     check_useful_opt_c_str!(storage_type, ErrorCode::CommonInvalidParam4);
     check_useful_opt_c_str!(config, ErrorCode::CommonInvalidParam5);
-    check_useful_c_str!(credentials, ErrorCode::CommonInvalidParam6);
+    check_useful_c_str!(credentials_json, ErrorCode::CommonInvalidParam6);
     check_useful_c_callback!(cb, ErrorCode::CommonInvalidParam7);
 
-    trace!("indy_create_wallet: entities >>> pool_name: {:?}, name: {:?}, storage_type: {:?}, config: {:?}, credentials: {:?}",
-           pool_name, name, storage_type, config, credentials);
+    trace!("indy_create_wallet: entities >>> pool_name: {:?}, name: {:?}, storage_type: {:?}, config: {:?}, credentials_json: {:?}",
+           pool_name, name, storage_type, config, credentials_json);
 
     let result = CommandExecutor::instance()
         .send(Command::Wallet(WalletCommand::Create(
             pool_name,
             name,
             storage_type,
             config,
-            credentials,
+            credentials_json,
             Box::new(move |result| {
                 let err = result_to_err_code!(result);
                 trace!("indy_create_wallet:");
@@ -218,7 +218,7 @@ pub extern fn indy_create_wallet(command_handle: i32,
 ///   {
 ///       "storage": Optional<object>  List of supported keys are defined by wallet type.
 ///   }
-/// credentials: Wallet credentials json.
+/// credentials_json: Wallet credentials json.
 ///   {
 ///       "key": string,
 ///       "rekey": Optional<string>,
@@ -236,24 +236,24 @@ pub extern fn indy_create_wallet(command_handle: i32,
 pub extern fn indy_open_wallet(command_handle: i32,
                                name: *const c_char,
                                runtime_config: *const c_char,
-                               credentials: *const c_char,
+                               credentials_json: *const c_char,
                                cb: Option<extern fn(xcommand_handle: i32,
                                                     err: ErrorCode,
                                                     handle: i32)>) -> ErrorCode {
-    trace!("indy_open_wallet: >>> name: {:?}, runtime_config: {:?}, credentials: {:?}", name, runtime_config, credentials);
+    trace!("indy_open_wallet: >>> name: {:?}, runtime_config: {:?}, credentials_json: {:?}", name, runtime_config, credentials_json);
 
     check_useful_c_str!(name, ErrorCode::CommonInvalidParam2);
     check_useful_opt_c_str!(runtime_config, ErrorCode::CommonInvalidParam3);
-    check_useful_c_str!(credentials, ErrorCode::CommonInvalidParam4);
+    check_useful_c_str!(credentials_json, ErrorCode::CommonInvalidParam4);
     check_useful_c_callback!(cb, ErrorCode::CommonInvalidParam5);
 
-    trace!("indy_open_wallet: entities >>> name: {:?}, runtime_config: {:?}, credentials: {:?}", name, runtime_config, credentials);
+    trace!("indy_open_wallet: entities >>> name: {:?}, runtime_config: {:?}, credentials_json: {:?}", name, runtime_config, credentials_json);
 
     let result = CommandExecutor::instance()
         .send(Command::Wallet(WalletCommand::Open(
             name,
             runtime_config,
-            credentials,
+            credentials_json,
             Box::new(move |result| {
                 let (err, handle) = result_to_err_code_1!(result, 0);
                 trace!("indy_open_wallet: handle: {:?}", handle);
@@ -340,7 +340,7 @@ pub extern fn indy_close_wallet(command_handle: i32,
 ///
 /// #Params
 /// name: Name of the wallet to delete.
-/// credentials: Wallet credentials json. List of supported keys are defined by wallet type.
+/// credentials_json: Wallet credentials json. List of supported keys are defined by wallet type.
 ///                    if NULL, then default credentials will be used.
 ///
 /// #Returns
@@ -352,21 +352,21 @@ pub extern fn indy_close_wallet(command_handle: i32,
 #[no_mangle]
 pub extern fn indy_delete_wallet(command_handle: i32,
                                  name: *const c_char,
-                                 credentials: *const c_char,
+                                 credentials_json: *const c_char,
                                  cb: Option<extern fn(xcommand_handle: i32,
                                                       err: ErrorCode)>) -> ErrorCode {
-    trace!("indy_delete_wallet: >>> name: {:?}, credentials: {:?}", name, credentials);
+    trace!("indy_delete_wallet: >>> name: {:?}, credentials_json: {:?}", name, credentials_json);
 
     check_useful_c_str!(name, ErrorCode::CommonInvalidParam2);
-    check_useful_c_str!(credentials, ErrorCode::CommonInvalidParam3);
+    check_useful_c_str!(credentials_json, ErrorCode::CommonInvalidParam3);
     check_useful_c_callback!(cb, ErrorCode::CommonInvalidParam4);
 
-    trace!("indy_delete_wallet: entities >>> name: {:?}, credentials: {:?}", name, credentials);
+    trace!("indy_delete_wallet: entities >>> name: {:?}, credentials_json: {:?}", name, credentials_json);
 
     let result = CommandExecutor::instance()
         .send(Command::Wallet(WalletCommand::Delete(
             name,
-            credentials,
+            credentials_json,
             Box::new(move |result| {
                 let err = result_to_err_code!(result);
                 trace!("indy_delete_wallet:");
@@ -386,11 +386,11 @@ pub extern fn indy_delete_wallet(command_handle: i32,
 /// #Params
 /// name: wallet storage name (the same as wallet name)
 /// config: wallet storage config (For example, database config)
-/// credentials: wallet storage credentials (For example, database credentials)
+/// credentials_json: wallet storage credentials (For example, database credentials)
 /// metadata: wallet metadata (For example encrypted keys).
 pub type WalletCreate = extern fn(name: *const c_char,
                                   config: *const c_char,
-                                  credentials: *const c_char,
+                                  credentials_json: *const c_char,
                                   metadata: *const c_char) -> ErrorCode;
 
 /// Open the wallet storage (For example, opening database connection)
@@ -399,12 +399,12 @@ pub type WalletCreate = extern fn(name: *const c_char,
 /// name: wallet storage name (the same as wallet name)
 /// config: wallet storage config (For example, database config)
 /// runtime_config: wallet storage runtime config (For example, connection config)
-/// credentials: wallet storage credentials (For example, database credentials)
+/// credentials_json: wallet storage credentials (For example, database credentials)
 /// storage_handle_p: pointer to store opened storage handle
 pub type WalletOpen = extern fn(name: *const c_char,
                                 config: *const c_char,
                                 runtime_config: *const c_char,
-                                credentials: *const c_char,
+                                credentials_json: *const c_char,
                                 storage_handle_p: *mut i32) -> ErrorCode;
 
 /// Close the opened walled storage (For example, closing database connection)
@@ -418,10 +418,10 @@ pub type WalletClose = extern fn(storage_handle: i32) -> ErrorCode;
 /// #Params
 /// name: wallet storage name (the same as wallet name)
 /// config: wallet storage config (For example, database config)
-/// credentials: wallet storage credentials (For example, database credentials)
+/// credentials_json: wallet storage credentials (For example, database credentials)
 pub type WalletDelete = extern fn(name: *const c_char,
                                   config: *const c_char,
-                                  credentials: *const c_char) -> ErrorCode;
+                                  credentials_json: *const c_char) -> ErrorCode;
 
 /// Create a new record in the wallet storage
 ///

libindy/src/errors/pool.rs

@@ -16,7 +16,7 @@ pub enum PoolError {
     Terminate,
     Timeout,
     AlreadyExists(String),
-    CommonError(CommonError)
+    CommonError(CommonError),
 }
 
 impl fmt::Display for PoolError {
@@ -27,7 +27,7 @@ impl fmt::Display for PoolError {
             PoolError::Terminate => write!(f, "Pool work terminated"),
             PoolError::Timeout => write!(f, "Timeout"),
             PoolError::AlreadyExists(ref description) => write!(f, "Pool ledger config already exists {}", description),
-            PoolError::CommonError(ref err) => err.fmt(f)
+            PoolError::CommonError(ref err) => err.fmt(f),
         }
     }
 }
@@ -40,7 +40,7 @@ impl error::Error for PoolError {
             PoolError::Terminate => "Pool work terminated",
             PoolError::Timeout => "Timeout",
             PoolError::AlreadyExists(ref description) => description,
-            PoolError::CommonError(ref err) => err.description()
+            PoolError::CommonError(ref err) => err.description(),
         }
     }
 
@@ -49,7 +49,7 @@ impl error::Error for PoolError {
             PoolError::NotCreated(_) | PoolError::InvalidHandle(_) => None,
             PoolError::Terminate | PoolError::Timeout => None,
             PoolError::AlreadyExists(_) => None,
-            PoolError::CommonError(ref err) => Some(err)
+            PoolError::CommonError(ref err) => Some(err),
         }
     }
 }