Skip to content

update dependencies and add convenient setup script #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mbeckerle wants to merge 3 commits into
base: master
Choose a base branch
from
Open

update dependencies and add convenient setup script #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
854f2c1
changes to work with daffodil 3.2.0-SNAPSHOT
mbeckerle Jun 2, 2021
6399fe1
Insures file/line/col information are removed.
mbeckerle Jun 2, 2021
b28e42a
Update dependencies, add handy script.
mbeckerle Dec 21, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,19 @@ organization := "io.github.openDFDL"

version := "1.0.0-SNAPSHOT"

scalaVersion := "2.12.11"
scalaVersion := "2.12.15"

parallelExecution in Test := false
Test / parallelExecution := false

testOptions += Tests.Argument(TestFrameworks.JUnit, "-v")

crossScalaVersions := Seq("2.12.11")
crossScalaVersions := Seq("2.12.15")

libraryDependencies := Seq(
"junit" % "junit" % "4.13.1" % "test",
"com.novocode" % "junit-interface" % "0.11" % "test",
"org.apache.daffodil" %% "daffodil-tdml-lib" % "3.1.0-SNAPSHOT",
"org.scala-lang.modules" %% "scala-xml" % "1.3.0",
"com.github.sbt" % "junit-interface" % "0.13.2" % "test",
"org.apache.daffodil" %% "daffodil-tdml-lib" % "3.2.0",
"org.scala-lang.modules" %% "scala-xml" % "2.0.0",
"javax.xml.bind" % "jaxb-api" % "2.3.1"
Copy link
Copy Markdown

@tuxji tuxji Dec 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider configuring Scala Steward to bump these versions automatically. Instructions:

Open a pull request that adds the repository of your Scala project to repos-github.md. Once that PR is merged, @scala-steward will check periodically for version number updates in your project and will open pull requests for updates it finds.

)

Expand Down
46 changes: 46 additions & 0 deletions make-symlinks-to-ibm-dfdl.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
#!/bin/sh
#
# A script to setup symlinks to IBM DFDL libraries and example files
# so that you can 'sbt publishLocal' this cross tester.
#
# Assumes you have unpacked the IBM DFDL jar per the instructions
# in the README.md file, and just want to "copy" the pieces into
# this module as those instructions suggest.
#
# Uses symlinks, not copying, so is dependent on operating systems
# that support symlinks. (You can change it to copy pretty easily
# by replacing 'ln -s' below with 'cp'.)
#
# Once you have run this script, then 'sbt test' should self-test
# the cross tester, and 'sbt publishLocal' should publish the
# component in your local ".ivy2" cache where managed dependency
# systems (like sbt) can find it.
#
# Then if you setup the sbt plugin, you can run DFDL schemas against
# IBM DFDL via a one-line change in the build.sbt, as is described
# in the README.md file.
#

cd ../ibm-dfdl # assumes the ibm libraries are in a peer directory
export IBMDIR=`pwd`
cd -
echo IBM DFDL found at $IBMDIR

echo Updating `pwd`/lib as symlink to $IBMDIR/lib
rm -rf lib
ln -s $IBMDIR/lib lib
touch lib/.keep

echo Setting up src/test/resources for example company.* files
cd src/test/resources
ln -s $IBMDIR/src/test/resources/company.txt company.txt
ln -s $IBMDIR/src/test/resources/company.xml company.xml
ln -s $IBMDIR/src/test/resources/company.xsd company.xsd
cd -

echo Setting up src/test/resources/IBMdefined for example DFDL schema
cd src/test/resources/IBMdefined
ln -s $IBMDIR/src/test/resources/IBMdefined/RecordSeparatedFieldFormat.xsd RecordSeparatedFieldFormat.xsd
cd -


2 changes: 1 addition & 1 deletion project/build.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
sbt.version=1.3.9
sbt.version=1.5.3
Copy link
Copy Markdown

@tuxji tuxji Dec 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not the latest sbt version, how come?

43 changes: 34 additions & 9 deletions src/main/scala/org/apache/daffodil/tdml/processor/IBM_DFDL.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,7 @@ package org.apache.daffodil.tdml.processor

import java.io.StringReader
import java.net.URI

import scala.xml.Node

import org.apache.commons.io.input.ReaderInputStream
import org.apache.daffodil.api.DaffodilSchemaSource
import org.apache.daffodil.api.DataLocation
Expand All @@ -37,7 +35,6 @@ import org.xml.sax.ErrorHandler
import org.xml.sax.InputSource
import org.xml.sax.SAXParseException
import org.xml.sax.helpers.XMLReaderFactory

import com.ibm.dfdl.grammar.DFDLGrammarFactory
import com.ibm.dfdl.grammar.IDFDLGrammar
import com.ibm.dfdl.processor.DFDLProcessorFactory
Expand All @@ -47,16 +44,17 @@ import com.ibm.dfdl.processor.IDFDLProcessor
import com.ibm.dfdl.processor.IDFDLProcessorErrorHandler
import com.ibm.dfdl.processor.exceptions.DFDLException
import com.ibm.dfdl.processor.types.DFDLDiagnosticType

import com.ibm.dfdl.sample.sax.reader.DFDLReader
import com.ibm.dfdl.sample.sax.writer.SAXToDFDLEventAdapter

import io.github.openDFDL.TraceListener
import io.github.openDFDL.DFDLReader2
import io.github.openDFDL.XMLSAXContentHandler1

import org.apache.daffodil.api.EmbeddedSchemaSource
import org.apache.daffodil.tdml.TDMLTestNotCompatibleException

import scala.xml.transform.RewriteRule
import scala.xml.transform.RuleTransformer

object IBMDFDLMode extends Enumeration {
type Type = Value
val Parse, Unparse, Compile, Configuration = Value
Expand Down Expand Up @@ -185,7 +183,8 @@ final class TDMLDFDLProcessorFactory private (
schemaSource: DaffodilSchemaSource,
useSerializedProcessor: Boolean,
optRootName: Option[String],
optRootNamespace: Option[String]): TDML.CompileResult = {
optRootNamespace: Option[String],
tunables: Map[String, String]): TDML.CompileResult = {

val rootNamespace = optRootNamespace.getOrElse(null)

Expand All @@ -194,8 +193,17 @@ final class TDMLDFDLProcessorFactory private (
val grammarFactory = new DFDLGrammarFactory
grammarFactory.setErrorHandler(grammarErrorHandler)
grammarFactory.setServiceTraceListener(traceListener)
val schemaUri: URI = schemaSource.uriForLoading
Assert.invariant(schemaSource.isInstanceOf[URISchemaSource])
//
// We must strip file/line/col information off the schema
// if present.
//
val preparedSchemaSource = schemaSource match {
case ess: EmbeddedSchemaSource =>
EmbeddedSchemaSource(XMLUtility.stripLineColInfo(ess.node), ess.nameHint)
case _ => schemaSource
}
val schemaUri: URI = preparedSchemaSource.uriForLoading
val grammar =
try {
//
Expand Down Expand Up @@ -229,6 +237,22 @@ final class TDMLDFDLProcessorFactory private (
private lazy val compileErrorHandler = new DFDLErrorHandler(IBMDFDLMode.Compile)
}

object XMLUtility {
private val removeLineColInfo = new RewriteRule {
override def transform(n: Node) = n match {
case e: scala.xml.Elem => {
e.copy(attributes = n.attributes.filter(a =>
(a.prefixedKey != "dafint:" + XMLUtils.COLUMN_ATTRIBUTE_NAME) &&
(a.prefixedKey != "dafint:" + XMLUtils.LINE_ATTRIBUTE_NAME) &&
(a.prefixedKey != "dafint:" + XMLUtils.FILE_ATTRIBUTE_NAME)))
}
case _ => n
}
}

val stripLineColInfo = new RuleTransformer(removeLineColInfo)
}

sealed trait DiagnosticsMixin {

def implementationName: String = "ibm"
Expand Down Expand Up @@ -415,8 +439,9 @@ final class IBMTDMLDFDLProcessor private (
new IBMTDMLParseResult(diagnostics, dfdlReader, sb)
}

override def unparse(infosetXML: scala.xml.Node, outputStream: java.io.OutputStream): TDMLUnparseResult = {
override def unparse(infosetXMLRaw: scala.xml.Node, outputStream: java.io.OutputStream): TDMLUnparseResult = {

val infosetXML = XMLUtility.stripLineColInfo(infosetXMLRaw)
val serializer = processorFactory.createSerializer
serializer.setGrammar(grammar)

Expand Down