Update DLP to use new RC of tensorframes (v0.2.9-rc3) (#45)

* Update DLP to use new RC of tensorframes (v0.2.9-rc0)

* Update tensorflow dependency from 1.1 -> 1.3, add pandas dependency

* Update test_tf_consistency to only compare graph nodes (exclude 'version', 'library' fields from comparison)

* Add cython dependency

* Move cython up in requirements file, hopefully pip installs it first?

* fix typo

* Update pandas to version that has cython dependency

* Remove cython dependency

* Update numpy version from 1.11.2 --> 1.13.1

* Update python dependencies to minimum version requirements instead
of exact requirements

* Add upper version bound on pillow

* Temporarily set keras version to 2.0.4 to see if build succeeds

* Debugging, do not merge

* Try tests with keras >= 2.0.4

* Fix keras version to 2.0.4, add explanatory comment

* Update tensorframes dep to 0.2.9-rc1

* Remove print statements

* Update tensorframes version to 0.2.9-rc1

* Upgrade to tensorframes RC2

* Update to tensorframes 0.2.9-rc3

Author: smurching

build.sbt

@@ -34,7 +34,7 @@ sparkComponents ++= Seq("mllib-local", "mllib", "sql")
 
 // add any Spark Package dependencies using spDependencies.
 // e.g. spDependencies += "databricks/spark-avro:0.1"
-spDependencies += s"databricks/tensorframes:0.2.8-s_${scalaMajorVersion}"
+spDependencies += s"databricks/tensorframes:0.2.9-rc3-s_${scalaMajorVersion}"
 
 
 // These versions are ancient, but they cross-compile around scala 2.10 and 2.11.

python/requirements.txt

@@ -1,10 +1,11 @@
 # This file should list any python package dependencies.
-coverage==4.4.1
-h5py==2.7.0
-keras==2.0.4
-nose==1.3.7  # for testing
-numpy==1.11.2
-pillow==4.1.1
-pygments==2.2.0
-tensorflow==1.1.0
-six==1.10.0
+coverage>=4.4.1
+h5py>=2.7.0
+keras==2.0.4 # NOTE: this package has only been tested with keras 2.0.4 and may not work with other releases
+nose>=1.3.7  # for testing
+numpy>=1.11.2
+pillow>=4.1.1,<4.2
+pygments>=2.2.0
+tensorflow==1.3.0
+pandas>=0.19.1
+six>=1.10.0

python/spark-package-deps.txt

@@ -1,3 +1,3 @@
 # This file should list any spark package dependencies as:
 # :package_name==:version   e.g. databricks/spark-csv==0.1
-databricks/tensorframes==0.2.8
+databricks/tensorframes==0.2.9-rc3

python/tests/graph/test_builder.py

@@ -58,9 +58,13 @@ def test_tf_consistency(self):
 
         self.assertEqual(z_ref, z_tgt)
 
-        # Version texts are not essential part of the graph, ignore them
-        gdef_ref.ClearField("versions")
-        gfn.graph_def.ClearField("versions")
+        # Remove all fields besides "node" from the graph definition, since we only
+        # care that the nodes are equal
+        # TODO(sid.murching) find a cleaner way of removing all fields besides "node"
+        nonessentialFields = ["versions", "version", "library"]
+        for fieldName in nonessentialFields:
+            gdef_ref.ClearField(fieldName)
+            gfn.graph_def.ClearField(fieldName)
 
         # The GraphDef contained in the GraphFunction object
         # should be the same as that in the one exported directly from TensorFlow session