Merging branch 'scanner'. Following changes:

 * `records` methods to iterate over large number of records
 * Faster test cases
 * prefix scan

Author: m-manu

README.md

@@ -8,44 +8,52 @@ An ultra-light-weight HBase ORM library that enables:
 
 
 ## Usage
-Let's say you've an HBase table `citizens` with row-key format of `country_code#UID`. Now, let's say your table is created with three column families `main`, `optional` and `tracked`, which may have columns `uid`, `name`, `salary` etc.
+Let's say you've an HBase table `citizens` with row-key format of `country_code#UID`. Now, let's say this table is created with three column families `main`, `optional` and `tracked`, which may have columns (qualifiers) `uid`, `name`, `salary` etc.
 
-This library enables to you represent your HBase table as a bean-like class, as below:
+This library enables to you represent your HBase table as a *bean-like class*, as below:
 
 ```java
-@HBTable(name = "citizens", families = {@Family(name = "main"), @Family(name = "optional", versions = 3), @Family(name = "tracked", versions = 10)})
+@HBTable(name = "citizens",
+  families = {
+    @Family(name = "main"),
+    @Family(name = "optional", versions = 3),
+    @Family(name = "tracked", versions = 10)
+  }
+)
 public class Citizen implements HBRecord<String> {
 
   @HBRowKey
   private String countryCode;
-  
+
   @HBRowKey
   private Integer uid;
-  
+
   @HBColumn(family = "main", column = "name")
   private String name;
-  
+
   @HBColumn(family = "optional", column = "age")
   private Short age;
-  
+
   @HBColumn(family = "optional", column = "salary")
   private Integer sal;
 
   @HBColumn(family = "optional", column = "counter")
   private Long counter;
-  
+
   @HBColumn(family = "optional", column = "custom_details")
   private Map<String, Integer> customDetails;
-  
+
   @HBColumn(family = "optional", column = "dependents")
   private Dependents dependents;
-  
+
   @HBColumnMultiVersion(family = "tracked", column = "phone_number")
   private NavigableMap<Long, Integer> phoneNumber;
-  
-  @HBColumn(family = "optional", column = "pincode", codecFlags = {@Flag(name = BestSuitCodec.SERIALIZE_AS_STRING, value = "true")})
+
+  @HBColumn(family = "optional", column = "pincode", codecFlags = {
+    @Flag(name = BestSuitCodec.SERIALIZE_AS_STRING, value = "true")
+  })
   private Integer pincode;
-  
+
   @Override
   public String composeRowKey() {
     return String.format("%s#%d", countryCode, uid);
@@ -57,22 +65,23 @@ public class Citizen implements HBRecord<String> {
     this.countryCode = pieces[0];
     this.uid = Integer.parseInt(pieces[1]);
   }
-  
+
   // Constructors, getters and setters
 }
 ```
 That is,
 
 * The above class `Citizen` represents the HBase table `citizens`, using the `@HBTable` annotation.
 * Logics for conversion of HBase row key to member variables of `Citizen` objects and vice-versa are implemented using `parseRowKey` and `composeRowKey` methods respectively.
-* The data type representing row key is the type parameter to `HBRecord` generic interface (in above case, `String`). Fields that form row key are annotated with `@HBRowKey`.
+* The data type representing row key is the type parameter to `HBRecord` generic interface (in above case, `String`).
+* Fields that form row key are annotated with `@HBRowKey` (just a marker annotation).
 * Names of columns and their column families are specified using `@HBColumn` or `@HBColumnMultiVersion` annotations.
 * The class may contain fields of simple data types (e.g. `String`, `Integer`), generic data types (e.g. `Map`, `List`), custom class (e.g. `Dependents`) or even generics of custom class (e.g. `List<Dependent>`) 
 * The `@HBColumnMultiVersion` annotation allows you to map multiple versions of column in a `NavigableMap<Long, ?>`. In above example, field `phoneNumber` is mapped to column `phone_number` within the column family `tracked` (which is configured for multiple versions)
 
 See source files [Citizen.java](./src/test/java/com/flipkart/hbaseobjectmapper/testcases/entities/Citizen.java) and [Employee.java](./src/test/java/com/flipkart/hbaseobjectmapper/testcases/entities/Employee.java) for detailed examples. Specifically, [Employee.java](./src/test/java/com/flipkart/hbaseobjectmapper/testcases/entities/Employee.java) demonstrates using "column inheritance" of this library, a useful feature if you have many HBase tables with common set of columns.
 
-Alternatively, you can model the class as below:
+Alternatively, you can model your class as below:
 
 ```java
 ...
@@ -82,9 +91,23 @@ public class Citizen implements HBRecord<CitizenKey> {
     String countryCode;
     Integer uid;
   }
+  
+  @HBRowKey
+  private CitizenKey rowKey;
+  
+  ...
+  
+  @Override
+  public CitizenKey composeRowKey() {
+    return return rowKey;
+  }
+
+  @Override
+  public void parseRowKey(CitizenKey rowKey) {
+    this.rowKey = rowKey;
+  }  
   ...
 }
-...
 ```
 
 
@@ -95,38 +118,34 @@ public class Citizen implements HBRecord<CitizenKey> {
   * uses [Jackson's JSON serializer](https://en.wikipedia.org/wiki/Jackson_(API)) for all other data types
   * serializes `null` as `null`
 * To customize serialization/deserialization behavior, you may define your own codec (by implementing the [Codec](./src/main/java/com/flipkart/hbaseobjectmapper/codec/Codec.java) interface) or you may extend the default codec.
-* The optional parameter `codecFlags` (supported by both `@HBColumn` and `@HBColumnMultiVersion` annotations) can be used to pass custom flags to the underlying codec. (e.g. You may write your codec to serialize field `Integer id` in `Citizen` class differently from field `Integer id` in `Employee` class)
+* The optional parameter `codecFlags` (supported by both `@HBColumn` and `@HBColumnMultiVersion` annotations) can be used to pass custom flags to the underlying codec. (e.g. You may want your codec to serialize field `Integer id` in `Citizen` class differently from field `Integer id` in `Employee` class)
 * The default codec class `BestSuitCodec` takes a flag `BestSuitCodec.SERIALIZE_AS_STRING`, whose value is "serializeAsString" (as in the above `Citizen` class example). When this flag is set to `true` on a field, the default codec serializes that field (even numerical fields) as strings.
   * Your custom codec may take other such flags to customize serialization/deserialization behavior at a **class field level**.
 
 ## Using this library for database access (DAO)
 This library provides an abstract class to define your own [data access object](https://en.wikipedia.org/wiki/Data_access_object). For example, you can create one for `Citizen` class in the above example as follows:
 
 ```java
-import org.apache.hadoop.conf.Configuration;
-
+import org.apache.hadoop.hbase.client.Connection;
 import java.io.IOException;
 
 public class CitizenDAO extends AbstractHBDAO<String, Citizen> {
 // in above, String is the row type of Citizen
 
-  public CitizenDAO(Configuration conf) throws IOException {
-    super(conf); // if you need to customize your codec, you may use super(conf, codec)
-    // alternatively, you can construct CitizenDAO by passing instance of 'Connection'
+  public CitizenDAO(Connection connection) throws IOException {
+    super(connection); // if you need to customize your codec, you may use super(connection, codec)
+    // alternatively, you can construct CitizenDAO by passing instance of 'org.apache.hadoop.conf.Configuration'
   }
 }
 ```
 (see [CitizenDAO.java](./src/test/java/com/flipkart/hbaseobjectmapper/testcases/daos/CitizenDAO.java))
 
-Once defined, you can access, manipulate and persist a row of `citizens` HBase table as below:
+Once defined, you can instantiate your *data access object* as below:
 
 ```java
-org.apache.hadoop.conf.Configuration configuration = getConf(); 
-
-// Create a data access object:
-CitizenDAO citizenDao = new CitizenDAO(configuration);
-// alternatively, in above, you can pass HBase client's Connection to your constructor
+CitizenDAO citizenDao = new CitizenDAO(connection);
 ```
+You can access, manipulate and persist records of `citizens` table as shown in below examples:
 
 Create new record:
 
@@ -135,22 +154,80 @@ String rowKey = citizenDao.persist(new Citizen("IND", 1, /* more params */));
 // In above, output of 'persist' is a String, because Citizen class implements HBRecord<String>
 ```
 
-Read data from HBase in various ways:
+Fetch a single record by its row key:
 
 ```java
-// Fetch a row from "citizens" HBase table with row key "IND#1":
+// Fetch row from "citizens" HBase table whose row key is "IND#1":
 Citizen pe = citizenDao.get("IND#1");
+```
+
+Fetch multiple records by their row keys:
 
+```java
 Citizen[] ape = citizenDao.get(new String[] {"IND#1", "IND#2"}); //bulk get
+```
+
+Fetch records by range of row keys (start row key, end row key):
+
+```java
+List<Citizen> lpe1 = citizenDao.get("IND#1", "IND#5");
+// above uses default behavior: start key inclusive, end key exclusive, 1 version
+
+List<Citizen> lpe2 = citizenDao.get("IND#1", true, "IND#9", true, 5, 10000);
+// above fetches with: start key inclusive, end key inclusive, 5 versions, caching set to 10,000 rows 
+
+```
+
+Iterate over *large number of records* by range of row keys:
+
+```java
+try (Records<Citizen> citizens = citizenDao.records("IND#000000001", true, "IND#100000000", true, 1, 10000)) {
+// using try-with-resources above to close the resources after iteration
+  for (Citizen citizen : citizens) {
+    // your code
+  }
+}
+```
+**Note:** All the `.records(...)` methods efficiently use iterators internally and do not load records upfront into memory. Hence, it's safe to fetch millions of records using them.
 
-// In below, note that "IND#1" is inclusive and "IND#5" is exclusive
-List<Citizen> lpe = citizenDao.get("IND#1", "IND#5"); //range get
-// ('versioned' variant above method is available)
+Fetch records by row key prefix:
 
-// for row keys in range ["IND#1", "IND#5"), fetch 3 versions of field 'phoneNumber' as a NavigableMap<row key, NavigableMap<timestamp, column value>>:
+```java
+// For small number of records:
+List<Citizen> lpe3 = citizenDao.getByPrefix(citizenDao.toBytes("IND#"));
+
+// For large number of records:
+try (Records<Citizen> citizens = citizenDao.recordsByPrefix(citizenDao.toBytes("IND#"))) {
+  for (Citizen citizen : citizens) {
+    // do something
+  }
+}
+```
+
+Fetch records by HBase's native `Scan` object: (for very advanced access patterns)
+
+```java
+Scan scan = new Scan().setAttribute(...)
+  .setReadType(...)
+  .setACL(...)
+  .withStartRow(...)
+  .withStopRow(...)
+  .readAllVersions(...);
+try (Records<Citizen> citizens = citizenDao.records(scan)) {
+  for (Citizen citizen : citizens) {
+    // do something
+  }
+}
+```
+
+
+Fetch specific field(s) for given row key(s):
+
+```java
+// for row keys in range ["IND#1", "IND#5"), fetch 3 versions of field 'phoneNumber':
 NavigableMap<String, NavigableMap<Long, Object>> phoneNumberHistory 
   = citizenDao.fetchFieldValues("IND#1", "IND#5", "phoneNumber", 3);
-// (bulk variants of above range method are also available)
+// bulk variants of above range method are also available
 ```
 
 Read data from HBase using HBase's native `Get`:
@@ -163,7 +240,7 @@ Get get2 = citizenDao.getGet("IND#2").setTimeRange(1, 5).setMaxVersions(2); // A
 counterDAO.getOnGets(get2);
 ```
 
-Manipulate and persist an object to HBase:
+Manipulate and persist an object back to HBase:
 
 ```java
 // change a field:
@@ -296,7 +373,7 @@ CitizenSummary citizenSummary = hbObjectMapper.readValue(
 ## Advantages
  * Your application code will be clean and minimal.
  * Your code need not worry about HBase methods or serialization/deserialization at all, thereby helping you maintain clear [separation of concerns](https://en.wikipedia.org/wiki/Separation_of_concerns).
- * Classes are **thread-safe**. You can just have to instantiate your DAO classes once at the start of your application and use them anywhere.
+ * Classes are **thread-safe**. You just have to instantiate your DAO classes once at the start of your application and use them anywhere!
  * Light weight: This library depends on just HBase Client and few other small libraries. It has very low overhead and hence is very fast.
  * Customizability/Extensibility: Want to use HBase native methods directly in some cases? No problem. Want to customize ser/deser in general or for a given class field? No problem. This library is high flexible.
 
@@ -313,21 +390,22 @@ Add below entry within the `dependencies` section of your `pom.xml`:
 <dependency>
   <groupId>com.flipkart</groupId>
   <artifactId>hbase-object-mapper</artifactId>
-  <version>1.12.1</version>
+  <version>1.13</version>
 </dependency>
 ```
+
 See artifact details: [com.flipkart:hbase-object-mapper on **Maven Central**](https://search.maven.org/search?q=g:com.flipkart%20AND%20a:hbase-object-mapper&core=gav) or
 [com.flipkart:hbase-object-mapper on **MVN Repository**](https://mvnrepository.com/artifact/com.flipkart/hbase-object-mapper).
 ## How to build?
 To build this project, follow below simple steps:
 
  1. Do a `git clone` of this repository
- 2. Checkout latest stable version `git checkout v1.12.1`
+ 2. Checkout latest stable version `git checkout v1.13`
  3. Execute `mvn clean install` from shell
 
 ### Please note:
 
- * Currently, projects that use this library are running on [Hortonworks Data Platform v3.1](https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/index.html) (corresponds to Hadoop 3.1 and HBase 2.0). However, if you are using a different version of Hadoop, you may change the versions in [pom.xml](./pom.xml) to desired ones and build the project.
+ * Currently, projects that use this library are running on [Hortonworks Data Platform v3.1](https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/index.html) (corresponds to Hadoop 3.1 and HBase 2.0). However, if you are using a different version of Hadoop/HBase, you may change the versions in [pom.xml](./pom.xml) to desired ones and build the project.
  * Test cases are **very comprehensive**. So, `mvn` build times can sometimes be longer, depending on your machine configuration.
  * By default, test cases spin an [in-memory HBase test cluster](https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java) to run data access related test cases (near-realworld scenario). 
     * If test cases are failing with time out errors, you may increase the timeout by setting environment variable `INMEMORY_CLUSTER_START_TIMEOUT` (seconds). For example, on Linux you may run the command `export INMEMORY_CLUSTER_START_TIMEOUT=8` on terminal, before running the aforementioned `mvn` command.

pom.xml

@@ -11,7 +11,7 @@
     <modelVersion>4.0.0</modelVersion>
     <groupId>com.flipkart</groupId>
     <artifactId>hbase-object-mapper</artifactId>
-    <version>1.12.1</version>
+    <version>1.13</version>
     <url>https://flipkart-incubator.github.io/hbase-orm/</url>
     <scm>
         <url>https://github.com/flipkart-incubator/hbase-orm/</url>
@@ -124,7 +124,6 @@
                 <configuration>
                     <source>1.8</source>
                     <target>1.8</target>
-                    <compilerArgument>-Xlint:all</compilerArgument>
                 </configuration>
             </plugin>
             <plugin>
@@ -148,7 +147,7 @@
             <plugin>
                 <groupId>org.apache.maven.plugins</groupId>
                 <artifactId>maven-source-plugin</artifactId>
-                <version>3.1.0</version>
+                <version>3.2.1</version>
                 <executions>
                     <execution>
                         <id>attach-sources</id>
@@ -175,7 +174,7 @@
             <plugin>
                 <groupId>org.jacoco</groupId>
                 <artifactId>jacoco-maven-plugin</artifactId>
-                <version>0.8.4</version>
+                <version>0.8.5</version>
                 <executions>
                     <execution>
                         <goals>