Release v1.2 (#30)

* Support additional operators in field based search

* Add index on table lines

* Add index on account ID to lines

* Remove unnecessary logs

* Remove unnecessary logs

* Introduce in and is operators in range

* Fix issue with null values

* Add test cases for in and is operators

* Sort transactions by timestamp

* Fix lines sorting

* test uuid table

* Remove test table

* Fix docs

* Fix docs

* Fix docs

* Remove QBank specific indexes

* Add docs on environment variables

Author: ilayaraja89

README.md

@@ -5,15 +5,19 @@ Systems that manage money do so by managing its movement - by tracking where it
 
 The there are two primitives in the system: **accounts** and **transactions**. Money moves between accounts by means of a transaction.
 
-A **transaction** may have multiple *lines* - each line represents the change (*delta*) of money in one account. A valid transaction has a total delta of zero - no money is created or destroyed, and all money moved out of any account(s) has moved in to other account(s). QLedger validates all transactions made via the API with a zero delta check.
+A **transaction** may have multiple *lines* - each line represents the change (*delta*) of money in one *account*. A valid transaction has a total delta of zero - no money is created or destroyed, and all money moved out of any account(s) has moved in to other account(s). QLedger validates all transactions made via the API with a zero delta check.
 
 > Phrased another way, the law of conversation of money is formalized by the rules of double entry bookkeeping - money debited from any account must be credited to another account (and vice versa), implying that all transactions must have at least two entries (double entry) with a zero sum delta. QLedger makes it easy to follow these rules.
 
 Accounts do not need to be predefined - they are called into existence when they are first used.
 
-All accounts and transactions are identified by a string identifier, which also acts an idempotency and an immutability key. Transactions once sent to the ledger cannot be changed - any 'modification' or reversal requires a new transaction. The safe recovery mechanism for all network errors is also a simple retry - as long as the identifier does not change the transaction will never be indavertently duplicated.
+All accounts and transactions are identified by a string identifier, which also acts an idempotency and an immutability key. Transactions once sent to the ledger cannot be changed - any 'modification' or reversal requires a new transaction. The safe recovery mechanism for all network errors is also a simple retry - as long as the identifier does not change the transaction will never be inadvertently duplicated.
 
-#### POST `/v1/transactions`
+## Transactions
+
+Transaction can be created as follows:
+
+`POST /v1/transactions`
 ```
 {
   "id": "abcd1234",
@@ -32,27 +36,37 @@ All accounts and transactions are identified by a string identifier, which also
 ```
 > Transactions with a total delta not equal to zero will result in a `400 BAD REQUEST` error.
 
+Transaction `timestamp` by default will be the time at which it is created. If necessary(such as migration of existing
+transactions), can be overridden using the `timestamp` property in the payload as follows:
+
+`POST /v1/transactions`
+```
+{
+  "id": "abcd1234",
+  "timestamp": "2017-01-01 13:01:05.000",
+  ...
 
-#### Metadata for Querying and Reports
+}
+```
 
-Transactions and accounts can have arbitrary number of key-value pairs maintained as a single JSON `data` which helps in grouping and filtering them by one or more criteria.
+> The `timestamp` in the payload should be in the format `2006-01-02 15:04:05.000`.
 
-Both transactions and accounts can be updated multiple times with `data`. The existing `data` is always overwritten with the new `data` value.
+Transactions can have arbitrary number of key-value pairs maintained as a single JSON `data` which helps in grouping and filtering them by one or more criteria.
 
 The `data` can be arbitrary JSON value as follows:
 ```
 {
   "data": {
-    "k1": "",
-    "k2": "strval",
-    "k3": ["av1", "av2", "av3"],
-    "k4": {
-      "nest1": {
-        "nest2": "val"
+    "active": true,
+    "status": "completed",
+    "codes": ["O123", "C123", "F123"],
+    "client_data": {
+      "interval": {
+        "invoice": "monthly"
       }
     },
-    "k5": 2017,
-    "k6": "2017-12-01"
+    "amount": 2000,
+    "expiry": "2017-12-01T05:00:00Z"
   }
 }
 ```
@@ -87,20 +101,7 @@ The transactions can be created with `data` as follows:
 }
 ```
 
-The accounts can be created with `data` as follows:
-
-`POST /v1/accounts`
-```
-{
-  "id": "alice",
-  "data": {
-    "product": "qw",
-    "date": "2017-01-01"
-  }
-}
-```
-
-The transactions or accounts can be updated with `data` using endpoints `PUT /v1/transactions` and `PUT /v1/accounts`
+Transactions can be updated multiple times with `data`. The existing `data` is always overwritten with the new `data` value.
 
 The transaction with ID `abcd1234` is updated with `data` as follows:
 
@@ -125,24 +126,56 @@ The transaction with ID `abcd1234` is updated with `data` as follows:
 }
 ```
 
-#### Searching of accounts and transactions
+## Accounts
+
+An account with ID `alice` can be created with `data` as follows:
+
+`POST /v1/accounts`
+```
+{
+  "id": "alice",
+  "data": {
+    "product": "qw",
+    "date": "2017-01-01"
+  }
+}
+```
+
+An account can be updated with `data` as follows:
+
+`PUT /v1/accounts`
+```
+{
+  "id": "alice",
+  "data": {
+    "product": "qw",
+    "date": "2017-01-05"
+  }
+}
+```
+
+## Searching of accounts and transactions
 
 The transactions and accounts can be filtered from the endpoints `GET /v1/transactions` and `GET /v1/accounts` with the search query formed using the bool clauses(`must` and `should`) and query types(`fields`, `terms` and `ranges`).
 
-##### Query types:
+### Query types:
 
-###### - `fields` query
+##### `fields` query
 
 Find items where the specified column exists with the specified value in the specified range.
 
 Example fields:
 - Field `{"id": {"eq": "ACME.CREDIT"}}` filters items where the column `id` is equal to `ACME.CREDIT`
+- Field `{"balance": {"ne": 0}}` filters items where the column `balance` is not equal to `0`.
 - Field `{"balance": {"lt": 0}}` filters items where the column `balance` is less than `0`
 - Field `{"timestamp": {"gte": "2017-01-01T05:30"}}` filters items where `timestamp` is greater than or equal to `2017-01-01T05:30`
+- Field `{"id": {"ne": "ACME.CREDIT"}}` filters items where the column `id` is not equal to `ACME.CREDIT`
+- Field `{"id": {"like": "%.DEBIT"}}` filters items where the column `id` ends with `.DEBIT`
+- Field `{"id": {"notlike": "%.DEBIT"}}` filters items where the column `id` doesn't ends with `.DEBIT`
 
-> The supported range operators are `lt`(less than), `lte`(less than or equal), `gt`(greater than), `gte`(greater than or equal), `eq`(equal).
+> The supported field operators are `lt`(less than), `lte`(less than or equal), `gt`(greater than), `gte`(greater than or equal), `eq`(equal), `ne`(not equal), `like`(like patterns), `notlike`(not like patterns).
 
-###### - `terms` query
+##### `terms` query
 
 Filters items where the specified key-value pairs in a term exists in the `data` JSON.
 
@@ -151,20 +184,25 @@ Example terms:
 - Term `{"months": ["jan", "feb", "mar"]}` filters items where values `jan`, `feb` AND `mar` in `data.months` array
 - Term `{"products":{"qw":{"tax":18.0}}}` filters items where subset `{"qw": {"tax": 18.0}}` in `products` object
 
-
-###### - `range` query
+##### `range` query
 
 Filters items which the specified key in `data` JSON exists in the specified range of values.
 
 Example range:
 - Range `{"charge": {"gte": 2000, "lte": 4000}}` filters items where `data.charge >= 2000` AND `data.charge <= 4000`
 - Range `{"date": {"gt": "2017-01-01","lt": "2017-06-31"}}` filters items where `data.date > '2017-01-01'` AND `data.date < '2017-01-31'`
+- Range `{"type": {"is": null}}` filters items where `data.type` is `NIL`
+- Range `{"type": {"is": null}}` filters items where `data.type` is not `NIL`
+- Range `{"action": {"in": ["intent", "invoice"]}}` filters items where `data.action` is ANY of `("intent", "invoice")`
+- Range `{"action": {"nin": ["charge", "refund"]}}` filters items where `data.action` is NOT ANY of `("charge", "refund")`
 
+> The supported range operators are `lt`(less than), `lte`(less than or equal), `gt`(greater than), `gte`(greater than or equal), `eq`(equal), `ne`(not equal), `like`(like patterns), `notlike`(not like patterns), `is`(is null checks), `isnot`(not null checks), `in`(ANY of list), `nin`(NOT ANY of list).
 
-##### Bool clauses:
+
+### Bool clauses:
 The following bool clauses determine whether all or any of the queries needs to be satisfied.
 
-###### - `must` clause
+##### `must` clause
 All of the query items in the `must` clause must be satisfied to get results.
 
 > The `must` clause can be equated with boolean `AND`
@@ -198,9 +236,7 @@ Example: The following query matches requests to match accounts which satisfies
 }
 ```
 
-
-
-###### - `should` clause
+##### `should` clause
 Any of the query items in the `should` clause should be satisfied to get results.
 
 > The `should` clause can be equated with boolean `OR`
@@ -238,3 +274,10 @@ Example: The following query matches requests to match transactions which satisf
 -  Clients those doesn't support passing search payload in the `GET`, can alternatively use the `POST`  endpoints: `POST /v1/transactions/_search` and `POST /v1/accounts/_search`.
 
 - A search query can have both `must` and `should` clauses.
+
+- Transactions in the search result are ordered chronological by default.
+
+
+## Environment Variables:
+
+Please read the documentation of all QLedger environment variables [here](./context#environment-variables)

context/README.md

@@ -0,0 +1,43 @@
+## Environment Variables
+
+#### Server Port: [Optional]
+
+QLedger server by default runs in port `7000`, which can be overridden by the following:
+```
+export PORT=7000
+```
+
+#### Authentication Token:
+
+QLedger API requests are authenticated using the secret token, which can be set using the following:
+```
+export LEDGER_AUTH_TOKEN=XXXXX
+```
+
+#### Database URL:
+
+QLedger uses PostgreSQL database to store the accounts and transactions.
+
+The PostgreSQL database URL can be set using:
+```
+export DATABASE_URL="postgres://localhost/ledgerdb?sslmode=disable"
+```
+
+For the purpose of running test cases, a separate database URL can be set using:
+```
+export TEST_DATABASE_URL="postgres://localhost/qw_ledger_test?sslmode=disable"
+```
+
+**Note:**
+
+- The database URL can be in one of the mentioned formats here:
+https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNSTRING
+
+#### Sharing Load Balancer/Domain Name: [Optional]
+
+In staging/production environments, the services are usually deployed in the same domain, differentiated and routed using the definite path prefixes.
+
+To access all QLedger APIs with prefix `/qledger/api`, set the following:
+```
+export HOST_PREFIX=/qledger/api
+```

controllers/accounts.go

@@ -22,7 +22,6 @@ func GetAccounts(w http.ResponseWriter, r *http.Request, context *ledgerContext.
 	}
 	defer r.Body.Close()
 	query := string(body)
-	log.Println("Query:", query)
 
 	engine, aerr := models.NewSearchEngine(context.DB, models.SearchNamespaceAccounts)
 	if aerr != nil {

controllers/transactions.go

@@ -83,7 +83,7 @@ func MakeTransaction(w http.ResponseWriter, r *http.Request, context *ledgerCont
 		}
 		// Otherwise the transaction is just a duplicate
 		// The exactly duplicate transactions are ignored
-		log.Println("Transaction is duplicate:", transaction.ID)
+		// log.Println("Transaction is duplicate:", transaction.ID)
 		w.WriteHeader(http.StatusAccepted)
 		return
 	}
@@ -115,7 +115,6 @@ func GetTransactions(w http.ResponseWriter, r *http.Request, context *ledgerCont
 		return
 	}
 	query := string(body)
-	log.Println("Query:", query)
 
 	results, aerr := engine.Query(query)
 	if aerr != nil {

migrations/postgres/20171017001_add_index_lines_transaction_id.down.sql

@@ -0,0 +1 @@
+DROP INDEX IF EXISTS lines_transaction_id_idx;

migrations/postgres/20171017001_add_index_lines_transaction_id.up.sql

@@ -0,0 +1 @@
+CREATE INDEX lines_transaction_id_idx ON lines USING btree (transaction_id);

migrations/postgres/20171107001_add_index_lines_account_id.down.sql

@@ -0,0 +1 @@
+DROP INDEX IF EXISTS lines_account_id_idx;

migrations/postgres/20171107001_add_index_lines_account_id.up.sql

@@ -0,0 +1 @@
+CREATE INDEX lines_account_id_idx ON lines USING btree (account_id);

models/search.go

@@ -4,7 +4,6 @@ import (
 	"database/sql"
 	"encoding/json"
 	"errors"
-	"log"
 	"regexp"
 	"strconv"
 	"strings"
@@ -63,8 +62,6 @@ func (engine *SearchEngine) Query(q string) (interface{}, ledgerError.Applicatio
 	}
 
 	sqlQuery := rawQuery.ToSQLQuery(engine.namespace)
-	log.Println("sqlQuery SQL:", sqlQuery.sql)
-	log.Println("sqlQuery args:", sqlQuery.args)
 	rows, err := engine.db.Query(sqlQuery.sql, sqlQuery.args...)
 	if err != nil {
 		return nil, DBError(err)
@@ -259,6 +256,10 @@ func (rawQuery *SearchRawQuery) ToSQLQuery(namespace string) *SearchSQLQuery {
 		q = q + "(" + strings.Join(shouldWhere, " OR ") + ")"
 	}
 
+	if namespace == "transactions" {
+		q = q + " ORDER BY timestamp"
+	}
+
 	if offset > 0 {
 		q = q + " OFFSET " + strconv.Itoa(offset) + " "
 	}