- static method in Schema to infer a schema from various data sources

- Bug fixes in schema inferral

Author: JohannesJander

.gitignore

@@ -1,18 +1,10 @@
-MainTest.java
-nbproject
 
 # Compiled class file
 *.class
 
 # Log file
 *.log
 
-# BlueJ files
-*.ctxt
-
-# Mobile Tools for Java (J2ME)
-.mtj.tmp/
-
 # Package Files #
 *.jar
 *.war
@@ -33,7 +25,7 @@ hs_err_pid*
 .project
 .settings/
 
-
-
 /bin/
 /schema.json
+/test.csv
+/test.json

docs/creating-schemas.md

@@ -2,7 +2,7 @@
 
 - [Creating from scratch via Java methods](#via-java-methods)
 - [Creating from a serialized JSON representation](#from-json)
-- [Creating from sample data (inferring)](#inferring-a-schema-from-data)
+- [Creating from sample data (inferring)](#auto-generating-a-schema-from-data-inferring)
 - [Schema validation](#schema-validation)
 - [Writing a Schema to a File](#writing-a-schema-to-a-file)
 
@@ -72,9 +72,41 @@ String schemaFilePath = "/path/to/schema/file/shema.json";
 Schema schema = new Schema(schemaFilePath, true); // enforce validation with strict=true.
 ```
 
-## Inferring a Schema from data
+## Auto-generating a Schema from data (inferring)
 
-If you don't have a schema for a CSV and don't want to manually define one then you can generate it:
+If you don't have a schema for a CSV and don't want to manually define one then you can auto-generate it:
+
+```java
+String csvData = "id,name,age\n1,John,30\n2,Jane,25\n3,Bob,35";
+Schema schema = Schema.infer(csvData, StandardCharsets.UTF_8);
+```
+The type inferral algorithm tries to cast to available types and each successful type casting increments a popularity score for the successful type cast in question. At the end, the best score so far is returned.
+The inferral algorithm traverses all of the table's rows and attempts to cast every single value of the table.
+
+You can  infer a schema from 
+- a CSV file 
+- a URL pointing to a CSV file
+- a CSV containing String
+- a JSON array node
+- a String array containing multiple CSV data sets
+- a File List
+- a URL List
+- mixed data types (CSV, JSON, etc.)
+
+If you have more than one CSV file, you can use `infer()` to check that all files have the same schema:
+
+```java
+File testFile = getResourceFile("/testsuite-data/files/csv/1mb.csv");
+File testFile2 = getResourceFile("/testsuite-data/files/csv/10mb.csv");
+List<File> fileList = Arrays.asList(testFile, testFile2);
+
+Schema schema = Schema.infer(fileList, StandardCharsets.UTF_8);
+```
+
+If the CSV files have different headers, the `Schema.infer()` call will throw an Exception.
+
+In case you want to infer a schema and then use the data, it can be helpful to not use the static `Schema.infer()` 
+method, but first create a `Table` instance and then infer the schema from it. 
 
 ```java
 URL url = new URL("https://raw.githubusercontent.com/frictionlessdata/tableschema-java/master" +
@@ -88,8 +120,7 @@ System.out.println(schema.asJson());
 
 ```
 
-The type inferral algorithm tries to cast to available types and each successful type casting increments a popularity score for the successful type cast in question. At the end, the best score so far is returned.
-The inferral algorithm traverses all of the table's rows and attempts to cast every single value of the table. When dealing with large tables, you might want to limit the number of rows that the inferral algorithm processes:
+When dealing with large tables, you might want to limit the number of rows that the inferral algorithm processes:
 
 ```java
 // Only process the first 25 rows for type inferral.

pom.xml

@@ -3,7 +3,7 @@
     <modelVersion>4.0.0</modelVersion>
     <groupId>io.frictionlessdata</groupId>
     <artifactId>tableschema-java</artifactId>
-    <version>0.9.4-SNAPSHOT</version>
+    <version>0.9.5-SNAPSHOT</version>
     <packaging>jar</packaging>
     <issueManagement>
         <url>https://github.com/frictionlessdata/tableschema-java/issues</url>

src/main/java/io/frictionlessdata/tableschema/field/DateField.java

@@ -52,8 +52,12 @@ public LocalDate parseValue(String value, String format, Map<String, Object> opt
                     would correspond to dates like: 30/11/14
                  */
                 String regex = parseDateFormat(format);
-                DateTimeFormatter formatter = DateTimeFormatter.ofPattern(regex);
-                return LocalDate.from(formatter.parse(value));
+                try {
+                    DateTimeFormatter formatter = DateTimeFormatter.ofPattern(regex);
+                    return LocalDate.from(formatter.parse(value));
+                } catch (Exception ex) {
+                    throw new TypeInferringException("Invalid date format: " + format);
+                }
             }
             throw new TypeInferringException();
         }

src/main/java/io/frictionlessdata/tableschema/field/GeopointField.java

@@ -31,64 +31,91 @@ public GeopointField(String name, String format, String title, String descriptio
     public double[] parseValue(String value, String format, Map<String, Object> options)
             throws TypeInferringException {
         try{
-            if(format.equalsIgnoreCase(Field.FIELD_FORMAT_DEFAULT)){
-                String[] geopoint = value.split(", *");
-
-                if(geopoint.length == 2){
-                    double lon = Double.parseDouble(geopoint[0]);
-                    double lat = Double.parseDouble(geopoint[1]);
-
-                    // No exception? It's a valid geopoint object.
-                    return new double[]{lon, lat};
+            if (format.equalsIgnoreCase(Field.FIELD_FORMAT_DEFAULT)){
+                return parseDefaultString(value);
+            } else if (format.equalsIgnoreCase(Field.FIELD_FORMAT_ARRAY)){
+                return parseArrayString(value);
+            } else if(format.equalsIgnoreCase(Field.FIELD_FORMAT_OBJECT)) {
+                return parseObjectString(value);
+            }
+        } catch(Exception e){
+            if (e instanceof TypeInferringException) {
+                throw e;
+            }
+            throw new TypeInferringException(e);
+        }
+        throw new TypeInferringException("Invalid format for geopoint field: " + format);
+    }
 
-                }else{
-                    throw new TypeInferringException("Geo points must have two coordinates");
+    @Override
+    public boolean isCompatibleValue(String value, String format) {
+        try {
+            parseDefaultString(value);
+            return true;
+        } catch (Exception ex) {
+            try {
+                parseArrayString(value);
+                return true;
+            } catch (Exception ex2) {
+                try {
+                    parseObjectString(value);
+                    return true;
+                } catch (Exception ex3) {
+                    return false;
                 }
+            }
+        }
+    }
 
-            }else if(format.equalsIgnoreCase(Field.FIELD_FORMAT_ARRAY)){
+    private static double[] parseDefaultString(String value) throws TypeInferringException {
+        String[] geopoint = value.split(", *");
 
-                // This will throw an exception if the value is not an array.
-                ArrayNode jsonArray = JsonUtil.getInstance().createArrayNode(value);
+        if(geopoint.length == 2){
+            double lon = Double.parseDouble(geopoint[0]);
+            double lat = Double.parseDouble(geopoint[1]);
 
-                if (jsonArray.size() == 2){
-                    double lon = jsonArray.get(0).asDouble();
-                    double lat = jsonArray.get(1).asDouble();
+            // No exception? It's a valid geopoint object.
+            return new double[]{lon, lat};
 
-                    // No exception? It's a valid geopoint object.
-                    return new double[]{lon, lat};
+        }else{
+            throw new TypeInferringException("Geo points must have two coordinates");
+        }
+    }
 
-                }else{
-                    throw new TypeInferringException("Geo points must have two coordinates");
-                }
+    private static double[] parseArrayString(String value) throws TypeInferringException {
+        // This will throw an exception if the value is not an array.
+        ArrayNode jsonArray = JsonUtil.getInstance().createArrayNode(value);
 
-            }else if(format.equalsIgnoreCase(Field.FIELD_FORMAT_OBJECT)){
+        if (jsonArray.size() == 2){
+            double lon = jsonArray.get(0).asDouble();
+            double lat = jsonArray.get(1).asDouble();
 
-                // This will throw an exception if the value is not an object.
-                JsonNode jsonObj = JsonUtil.getInstance().createNode(value);
+            // No exception? It's a valid geopoint object.
+            return new double[]{lon, lat};
 
-                if (jsonObj.size() == 2 && jsonObj.has("lon") && jsonObj.has("lat")){
-                    double lon = jsonObj.get("lon").asDouble();
-                    double lat = jsonObj.get("lat").asDouble();
+        }else{
+            throw new TypeInferringException("Geo points must have two coordinates");
+        }
+    }
 
-                    // No exception? It's a valid geopoint object.
-                    return new double[]{lon, lat};
+    private static double[] parseObjectString(String value) throws TypeInferringException {
+        // This will throw an exception if the value is not an object.
+        JsonNode jsonObj = JsonUtil.getInstance().createNode(value);
 
-                }else{
-                    throw new TypeInferringException();
-                }
+        if (jsonObj.size() == 2 && jsonObj.has("lon") && jsonObj.has("lat")){
+            double lon = jsonObj.get("lon").asDouble();
+            double lat = jsonObj.get("lat").asDouble();
 
-            }else{
-                throw new TypeInferringException();
-            }
+            // No exception? It's a valid geopoint object.
+            return new double[]{lon, lat};
 
-        }catch(Exception e){
-            if (e instanceof TypeInferringException) {
-                throw e;
-            }
-            throw new TypeInferringException(e);
+        }else{
+            throw new TypeInferringException();
         }
+
     }
 
+
     @Override
     public String formatValueAsString(double[] value, String format, Map<String, Object> options) throws InvalidCastException, ConstraintsException {
         if ((null == format) || (format.equalsIgnoreCase(Field.FIELD_FORMAT_DEFAULT))){