Added further options to preload classes (#139)

* Added further options to preload classes

Added methods to
- preload several classes in one call
- preload all classes annotated with @AerospikeRecord from a package
(specified by either a String or a Class in the package)

Also added a unit test for the same as well as updated the README.md
file.

* Replaced class.getPackage() with JDK 8 implementation

* Move componentType() to getComponentType()

* Minor formatting changes

* Fixed an indenting issue

---------

Co-authored-by: Tim Faulkes <[email protected]>

Author: tim-aero

README.md

@@ -243,6 +243,12 @@ The Builder constructor simply takes an IAerospikeClient which it uses for acces
 
 `.preLoadClass(Class<?>)`: Used to load a class before it is needed. The process of loading a class for the first time can be moderately expensive -- there is lots of introspection which goes on to determine how to map the classes to and from the database with the help of the annotations or configuration file. The results of this process are cached so it only has to happen once, and as few introspection calls as possible are called during the actual transformation. If a class is not preloaded, this computation will happen the first time an instance of that class is encountered, resulting in slowdown on the first call.
 
+Another reason to preload a class is situations where an abstract superclass might be read without the subclasses being seen by the AeroMapper first. For example, a list of `Animal` might be stored in the database, but `Animal` is an abstract class with concrete subclasses like `Dog`, `Cat`, etc. If the first call of the AeroMapper is to read a list of `Animal` from the database, there is not enough information to resolve the concrete sub-classes without preloading them.
+
+`.preLoadClasses(Class<?> ...)`: Use to preload several classes before they are called. This is a convenience mechanism which calls `.preLoadClass` for each of the classes in the list.
+
+`.preLoadClassesFromPackage(String | Class<?>)`: Preload all the classes in the specified package which are annotated with `@AerospikeRecord`. The package can be specified by passing a string of the package name or by passing a class in that package. The latter method is preferred as this is less brittle as code is refactored. Note that if a class is passed this class is used only for the package name and does not necessarily need to be a class annotated with `@AerospikeRecord`. Creating a 'marker' class in the package with no functionality and passing to this method is a good way of preventing breaking the preloading as classes are moved around.
+
 `withConfigurationFile`: Whilst mapping information from POJOs via annotations is efficient and has the mapping code inline with the POJO code, there are times when this is not available. For example, if an external library with POJOs is being used and it is desired to map those POJOs to the database, there is no easy way of annotating the source code. Another case this applies is if different mapping parameters are needed between different environments. For example, embedded objects might be stored in a map in development for ease of debugging, but stored in a list in production for compaction of stored data. In these cases an external configuration YAML file can be used to specify how to map the data to the database. See [External Configuration File](#external-configuration-file) for more details. There is an overload of this method which takes an additional boolean parameter -- if this is `true` and the configuration file is not valid, errors will be logged to `stderr` and the process continue. It is normally not recommended to set this parameter to true.
 
 If multiple configuration files are used and the same class is defined in multiple configuration files, the definitions in the first configuration file for a class will be used. 

src/main/java/com/aerospike/mapper/tools/AeroMapper.java

@@ -1,13 +1,18 @@
 package com.aerospike.mapper.tools;
 
+import java.io.BufferedReader;
 import java.io.File;
 import java.io.IOException;
 import java.io.InputStream;
+import java.io.InputStreamReader;
 import java.lang.reflect.Array;
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.List;
+import java.util.Set;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.function.Function;
+import java.util.stream.Collectors;
 
 import javax.validation.constraints.NotNull;
 
@@ -31,6 +36,7 @@
 import com.aerospike.client.query.Filter;
 import com.aerospike.client.query.RecordSet;
 import com.aerospike.client.query.Statement;
+import com.aerospike.mapper.annotations.AerospikeRecord;
 import com.aerospike.mapper.tools.ClassCache.PolicyType;
 import com.aerospike.mapper.tools.configuration.ClassConfig;
 import com.aerospike.mapper.tools.configuration.Configuration;
@@ -57,7 +63,8 @@ public Builder(IAerospikeClient client) {
         }
 
         /**
-         * Add in a custom type converter. The converter must have methods which implement the ToAerospike and FromAerospike annotation.
+         * Add in a custom type converter. The converter must have methods which
+         * implement the ToAerospike and FromAerospike annotation.
          *
          * @param converter The custom converter
          * @return this object
@@ -77,6 +84,64 @@ public Builder preLoadClass(Class<?> clazz) {
             return this;
         }
 
+        public Builder preLoadClasses(Class<?>... clazzes) {
+            if (classesToPreload == null) {
+                classesToPreload = new ArrayList<>();
+            }
+            classesToPreload.addAll(Arrays.asList(clazzes));
+            return this;
+        }
+
+        public String getPackageName(Class<?> clazz) {
+            Class<?> c;
+            if (clazz.isArray()) {
+                c = clazz.getComponentType();
+            } else {
+                c = clazz;
+            }
+            String pn;
+            if (c.isPrimitive()) {
+                pn = "java.lang";
+            } else {
+                String cn = c.getName();
+                int dot = cn.lastIndexOf('.');
+                pn = (dot != -1) ? cn.substring(0, dot).intern() : "";
+            }
+            return pn;
+        }
+
+        public Builder preLoadClassesFromPackage(Class<?> classInPackage) {
+            return preLoadClassesFromPackage(getPackageName(classInPackage));
+        }
+
+        public Builder preLoadClassesFromPackage(String thePackage) {
+            Set<Class<?>> clazzes = findAllClassesUsingClassLoader(thePackage);
+            for (Class<?> thisClazz : clazzes) {
+                // Only add classes with the AerospikeRecord annotation.
+                if (thisClazz.getAnnotation(AerospikeRecord.class) != null) {
+                    this.preLoadClass(thisClazz);
+                }
+            }
+            return this;
+        }
+
+        // See https://www.baeldung.com/java-find-all-classes-in-package
+        private Set<Class<?>> findAllClassesUsingClassLoader(String packageName) {
+            InputStream stream = ClassLoader.getSystemClassLoader()
+                    .getResourceAsStream(packageName.replaceAll("[.]", "/"));
+            BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
+            return reader.lines().filter(line -> line.endsWith(".class")).map(line -> getClass(line, packageName))
+                    .collect(Collectors.toSet());
+        }
+
+        private Class<?> getClass(String className, String packageName) {
+            try {
+                return Class.forName(packageName + "." + className.substring(0, className.lastIndexOf('.')));
+            } catch (ClassNotFoundException ignored) {
+            }
+            return null;
+        }
+
         public Builder withConfigurationFile(File file) throws IOException {
             return this.withConfigurationFile(file, false);
         }
@@ -103,7 +168,8 @@ public Builder withConfiguration(String configurationYaml) throws JsonProcessing
             return this.withConfiguration(configurationYaml, false);
         }
 
-        public Builder withConfiguration(String configurationYaml, boolean allowsInvalid) throws JsonProcessingException {
+        public Builder withConfiguration(String configurationYaml, boolean allowsInvalid)
+                throws JsonProcessingException {
             ObjectMapper objectMapper = new ObjectMapper(new YAMLFactory());
             Configuration configuration = objectMapper.readValue(configurationYaml, Configuration.class);
             this.loadConfiguration(configuration, allowsInvalid);
@@ -211,22 +277,25 @@ public void save(@NotNull Object object, String... binNames) throws AerospikeExc
     }
 
     @Override
-    public void save(@NotNull WritePolicy writePolicy, @NotNull Object object, String... binNames) throws AerospikeException {
+    public void save(@NotNull WritePolicy writePolicy, @NotNull Object object, String... binNames)
+            throws AerospikeException {
         save(writePolicy, object, null, binNames);
     }
 
     @SuppressWarnings("unchecked")
-    private <T> void save(WritePolicy writePolicy, @NotNull T object, RecordExistsAction recordExistsAction, String[] binNames) {
+    private <T> void save(WritePolicy writePolicy, @NotNull T object, RecordExistsAction recordExistsAction,
+            String[] binNames) {
         Class<T> clazz = (Class<T>) object.getClass();
         ClassCacheEntry<T> entry = MapperUtils.getEntryAndValidateNamespace(clazz, this);
         if (writePolicy == null) {
             writePolicy = new WritePolicy(entry.getWritePolicy());
             if (recordExistsAction != null) {
                 writePolicy.recordExistsAction = recordExistsAction;
             }
-            
-            // #132 -- Ensure that if an overriding TTL / sendkey is passed in the policy it is NOT overwritten. Hence
-            // only if the policy is null do we override these settings. 
+
+            // #132 -- Ensure that if an overriding TTL / sendkey is passed in the policy it
+            // is NOT overwritten. Hence
+            // only if the policy is null do we override these settings.
             Integer ttl = entry.getTtl();
             Boolean sendKey = entry.getSendKey();
 
@@ -261,19 +330,22 @@ public <T> T readFromDigest(@NotNull Class<T> clazz, @NotNull byte[] digest) thr
     }
 
     @Override
-    public <T> T readFromDigest(@NotNull Class<T> clazz, @NotNull byte[] digest, boolean resolveDependencies) throws AerospikeException {
+    public <T> T readFromDigest(@NotNull Class<T> clazz, @NotNull byte[] digest, boolean resolveDependencies)
+            throws AerospikeException {
         ClassCacheEntry<T> entry = MapperUtils.getEntryAndValidateNamespace(clazz, this);
         Key key = new Key(entry.getNamespace(), digest, entry.getSetName(), null);
         return this.read(null, clazz, key, entry, resolveDependencies);
     }
 
     @Override
-    public <T> T readFromDigest(Policy readPolicy, @NotNull Class<T> clazz, @NotNull byte[] digest) throws AerospikeException {
+    public <T> T readFromDigest(Policy readPolicy, @NotNull Class<T> clazz, @NotNull byte[] digest)
+            throws AerospikeException {
         return this.readFromDigest(readPolicy, clazz, digest, true);
     }
 
     @Override
-    public <T> T readFromDigest(Policy readPolicy, @NotNull Class<T> clazz, @NotNull byte[] digest, boolean resolveDependencies) throws AerospikeException {
+    public <T> T readFromDigest(Policy readPolicy, @NotNull Class<T> clazz, @NotNull byte[] digest,
+            boolean resolveDependencies) throws AerospikeException {
         ClassCacheEntry<T> entry = MapperUtils.getEntryAndValidateNamespace(clazz, this);
         Key key = new Key(entry.getNamespace(), digest, entry.getSetName(), null);
         return this.read(readPolicy, clazz, key, entry, resolveDependencies);
@@ -285,7 +357,8 @@ public <T> T read(@NotNull Class<T> clazz, @NotNull Object userKey) throws Aeros
     }
 
     @Override
-    public <T> T read(@NotNull Class<T> clazz, @NotNull Object userKey, boolean resolveDependencies) throws AerospikeException {
+    public <T> T read(@NotNull Class<T> clazz, @NotNull Object userKey, boolean resolveDependencies)
+            throws AerospikeException {
         ClassCacheEntry<T> entry = MapperUtils.getEntryAndValidateNamespace(clazz, this);
         String set = entry.getSetName();
         Key key = new Key(entry.getNamespace(), set, Value.get(entry.translateKeyToAerospikeKey(userKey)));
@@ -298,7 +371,8 @@ public <T> T read(Policy readPolicy, @NotNull Class<T> clazz, @NotNull Object us
     }
 
     @Override
-    public <T> T read(Policy readPolicy, @NotNull Class<T> clazz, @NotNull Object userKey, boolean resolveDependencies) throws AerospikeException {
+    public <T> T read(Policy readPolicy, @NotNull Class<T> clazz, @NotNull Object userKey, boolean resolveDependencies)
+            throws AerospikeException {
         ClassCacheEntry<T> entry = MapperUtils.getEntryAndValidateNamespace(clazz, this);
         String set = entry.getSetName();
         Key key = new Key(entry.getNamespace(), set, Value.get(entry.translateKeyToAerospikeKey(userKey)));
@@ -311,7 +385,8 @@ public <T> T[] read(@NotNull Class<T> clazz, @NotNull Object[] userKeys) throws
     }
 
     @Override
-    public <T> T[] read(BatchPolicy batchPolicy, @NotNull Class<T> clazz, @NotNull Object[] userKeys) throws AerospikeException {
+    public <T> T[] read(BatchPolicy batchPolicy, @NotNull Class<T> clazz, @NotNull Object[] userKeys)
+            throws AerospikeException {
         return read(batchPolicy, clazz, userKeys, (Operation[]) null);
     }
 
@@ -321,7 +396,8 @@ public <T> T[] read(@NotNull Class<T> clazz, @NotNull Object[] userKeys, Operati
     }
 
     @Override
-    public <T> T[] read(BatchPolicy batchPolicy, @NotNull Class<T> clazz, @NotNull Object[] userKeys, Operation... operations) {
+    public <T> T[] read(BatchPolicy batchPolicy, @NotNull Class<T> clazz, @NotNull Object[] userKeys,
+            Operation... operations) {
         ClassCacheEntry<T> entry = MapperUtils.getEntryAndValidateNamespace(clazz, this);
         String set = entry.getSetName();
         Key[] keys = new Key[userKeys.length];
@@ -336,8 +412,9 @@ public <T> T[] read(BatchPolicy batchPolicy, @NotNull Class<T> clazz, @NotNull O
         return readBatch(batchPolicy, clazz, keys, entry, operations);
     }
 
-    @SuppressWarnings({"unchecked"})
-    private <T> T read(Policy readPolicy, @NotNull Class<T> clazz, @NotNull Key key, @NotNull ClassCacheEntry<T> entry, boolean resolveDependencies) {
+    @SuppressWarnings({ "unchecked" })
+    private <T> T read(Policy readPolicy, @NotNull Class<T> clazz, @NotNull Key key, @NotNull ClassCacheEntry<T> entry,
+            boolean resolveDependencies) {
         if (readPolicy == null || readPolicy.filterExp == null) {
             Object objectForKey = LoadedObjectResolver.get(key);
             if (objectForKey != null) {
@@ -367,7 +444,7 @@ private <T> T read(Policy readPolicy, @NotNull Class<T> clazz, @NotNull Key key,
 
     @SuppressWarnings("unchecked")
     private <T> T[] readBatch(BatchPolicy batchPolicy, @NotNull Class<T> clazz, @NotNull Key[] keys,
-                              @NotNull ClassCacheEntry<T> entry, Operation... operations) {
+            @NotNull ClassCacheEntry<T> entry, Operation... operations) {
         if (batchPolicy == null) {
             batchPolicy = entry.getBatchPolicy();
         }
@@ -405,7 +482,8 @@ public <T> boolean delete(@NotNull Class<T> clazz, @NotNull Object userKey) thro
     }
 
     @Override
-    public <T> boolean delete(WritePolicy writePolicy, @NotNull Class<T> clazz, @NotNull Object userKey) throws AerospikeException {
+    public <T> boolean delete(WritePolicy writePolicy, @NotNull Class<T> clazz, @NotNull Object userKey)
+            throws AerospikeException {
         ClassCacheEntry<T> entry = MapperUtils.getEntryAndValidateNamespace(clazz, this);
         Object asKey = entry.translateKeyToAerospikeKey(userKey);
 
@@ -452,7 +530,8 @@ public <T> void find(@NotNull Class<T> clazz, Function<T, Boolean> function) thr
 
         RecordSet recordSet = null;
         try {
-            // TODO: set the policy (If this statement is thought to be useful, which is dubious)
+            // TODO: set the policy (If this statement is thought to be useful, which is
+            // dubious)
             recordSet = mClient.query(null, statement);
             T result;
             while (recordSet.next()) {
@@ -487,7 +566,8 @@ public <T> void scan(@NotNull Class<T> clazz, @NotNull Processor<T> processor, i
     }
 
     @Override
-    public <T> void scan(ScanPolicy policy, @NotNull Class<T> clazz, @NotNull Processor<T> processor, int recordsPerSecond) {
+    public <T> void scan(ScanPolicy policy, @NotNull Class<T> clazz, @NotNull Processor<T> processor,
+            int recordsPerSecond) {
         ClassCacheEntry<T> entry = MapperUtils.getEntryAndValidateNamespace(clazz, this);
         if (policy == null) {
             policy = entry.getScanPolicy();
@@ -583,7 +663,8 @@ public <T> VirtualList<T> asBackedList(@NotNull Object object, @NotNull String b
     }
 
     @Override
-    public <T> VirtualList<T> asBackedList(@NotNull Class<?> owningClazz, @NotNull Object key, @NotNull String binName, Class<T> elementClazz) {
+    public <T> VirtualList<T> asBackedList(@NotNull Class<?> owningClazz, @NotNull Object key, @NotNull String binName,
+            Class<T> elementClazz) {
         return new VirtualList<>(this, owningClazz, key, binName, elementClazz);
     }
 
@@ -631,18 +712,18 @@ private Policy getPolicyByClassAndType(Class<?> clazz, PolicyType policyType) {
         ClassCacheEntry<?> entry = ClassCache.getInstance().loadClass(clazz, this);
 
         switch (policyType) {
-            case READ:
-                return entry == null ? mClient.getReadPolicyDefault() : entry.getReadPolicy();
-            case WRITE:
-                return entry == null ? mClient.getWritePolicyDefault() : entry.getWritePolicy();
-            case BATCH:
-                return entry == null ? mClient.getBatchPolicyDefault() : entry.getBatchPolicy();
-            case SCAN:
-                return entry == null ? mClient.getScanPolicyDefault() : entry.getScanPolicy();
-            case QUERY:
-                return entry == null ? mClient.getQueryPolicyDefault() : entry.getQueryPolicy();
-            default:
-                throw new UnsupportedOperationException("Provided unsupported policy.");
+        case READ:
+            return entry == null ? mClient.getReadPolicyDefault() : entry.getReadPolicy();
+        case WRITE:
+            return entry == null ? mClient.getWritePolicyDefault() : entry.getWritePolicy();
+        case BATCH:
+            return entry == null ? mClient.getBatchPolicyDefault() : entry.getBatchPolicy();
+        case SCAN:
+            return entry == null ? mClient.getScanPolicyDefault() : entry.getScanPolicy();
+        case QUERY:
+            return entry == null ? mClient.getQueryPolicyDefault() : entry.getQueryPolicy();
+        default:
+            throw new UnsupportedOperationException("Provided unsupported policy.");
         }
     }
 }