Working on SQL

Author: cgivre

contrib/format-daffodil/README.md

@@ -1,25 +1,35 @@
-# Daffodil 'Format' Reader
+# Daffodil Format Reader
 This plugin enables Drill to read DFDL-described data from files by way of the Apache Daffodil DFDL implementation.
 
-## Validation
+## Configuration:
+To use Daffodil schemata, simply add the following to the `formats` section of a file-based storage plugin:
 
-Data read by Daffodil is always validated using Daffodil's Limited Validation mode.
+```json
+"daffodil": {
+          "type": "daffodil",
+          "extensions": [
+            "dat"
+          ]
+        }
+```
+There are four other optional parameters which you can specify:
+* `schemaURI`: Pre-compiled dfdl schema (.bin extension) or DFDL schema source (.xsd extension)
+* `validationMode`: Use `true` to request Daffodil built-in limited validation. Use `false` for no validation.
+* `rootName`: Local name of root element of the message. Can be null to use the first element declaration of the primary schema file. Ignored if reloading a pre-compiled schema.
+* `rootNameSpace`:  Namespace URI as a string. Can be `null` to use the target namespace of the primary schema file or if it is unambiguous what element is the rootName. Ignored if reloading a pre-compiled schema.
 
-TBD: do we need an option to control escalating validation errors to fatal? Currently this is not provided.
+## Usage:
 
-## Limitations:  TBD
 
-At the moment, the DFDL schema is found on the local file system, which won't support Drill's distributed architecture.
 
-There are restrictions on the DFDL schemas that this can handle.
+## Limitations:
+At the moment, the DFDL schema is found on the local file system, which won't support Drill's distributed architecture.
 
-In particular, all element children must have distinct element names, including across choice branches.
-(This rules out a number of large DFDL schemas.)
+There are restrictions on the DFDL schemas that this can handle.  In particular, all element children must have distinct element names, including across choice branches.  Unfortunately, this rules out a number of large DFDL schemas.
 
 TBD: Auto renaming as part of the Daffodil-to-Drill metadata mapping?
 
-The data is parsed fully from its native form into a Drill data structure held in memory.
-No attempt is made to avoid access to parts of the DFDL-described data that are not needed to answer the query.
+The data is parsed fully from its native form into a Drill data structure held in memory. No attempt is made to avoid access to parts of the DFDL-described data that are not needed to answer the query.
 
 If the data is not well-formed, an error occurs and the query fails.
 

contrib/format-daffodil/src/test/java/org/apache/drill/exec/store/daffodil/TestDynamicSchemata.java

@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.store.daffodil;
+
+import org.apache.drill.test.BaseTestQuery;
+import org.junit.Test;
+
+public class TestDynamicSchemata  extends BaseTestQuery {
+
+  @Test
+  public void testSchema() throws Exception {
+    test("CREATE DAFFODIL SCHEMA USING JAR 'xxx.jar'");
+
+  }
+}

exec/java-exec/src/main/codegen/data/Parser.tdd

@@ -30,6 +30,7 @@
 
   # List of keywords.
   keywords: [
+    "DAFFODIL",
     "DATABASES",
     "SCHEMAS",
     "TABLES",
@@ -67,6 +68,8 @@
     "SqlRefreshMetadata()",
     "SqlCreateFunction()",
     "SqlDropFunction()",
+    "SqlCreateDaffodilSchema()",
+    "SqlDropDaffodilSchema()",
     "SqlAnalyzeTable()",
     "DrillSqlSetOption(Span.of(), null)",
     "DrillSqlResetOption(Span.of(), null)",

exec/java-exec/src/main/codegen/includes/parserImpls.ftl

@@ -687,6 +687,48 @@ SqlNode SqlDropFunction() :
    }
 }
 
+/**
+* Parse create Daffodil Schema statement
+* CREATE DAFFODIL SCHEMA USING JAR 'jar_name'
+*/
+SqlNode SqlCreateDaffodilSchema() :
+{
+  SqlParserPos pos;
+  SqlNode jar;
+}
+{
+  <CREATE> { pos = getPos(); }
+  <DAFFODIL>
+  <SCHEMA>
+  <USING>
+  <JAR>
+  jar = StringLiteral()
+  {
+     return new SqlCreateDaffodilSchema(pos, jar);
+   }
+ }
+
+/**
+* Parse drop UDF statement
+* DROP DAFFODIL SCHEMA USING JAR 'jar_name'
+*/
+SqlNode SqlDropDaffodilSchema() :
+{
+SqlParserPos pos;
+SqlNode jar;
+}
+{
+<DROP> { pos = getPos(); }
+<DAFFODIL>
+<SCHEMA>
+<USING>
+  <JAR>
+  jar = StringLiteral()
+  {
+    return new SqlDropDaffodilSchema(pos, jar);
+  }
+}
+
 /**
  * Parses a analyze statements:
  * <ul>