Merge pull request #36 from paulgc/master

Project import generated by Copybara.

Author: paulgc17

README.md

@@ -20,7 +20,7 @@ TF Data Validation includes:
 *    An anomalies viewer so that you can see what features have anomalies and
      learn more in order to correct them.
 
-For instructions on using TFDV, see the [get started guide](g3doc/get_started.md)
+For instructions on using TFDV, see the [get started guide](https://github.com/tensorflow/data-validation/blob/master/g3doc/get_started.md)
 and try out the [example notebook](https://colab.research.google.com/github/tensorflow/data-validation/blob/master/examples/chicago_taxi/chicago_taxi_tfdv.ipynb).
 
 Caution: TFDV may be backwards incompatible before version 1.0.
@@ -34,14 +34,6 @@ The recommended way to install TFDV is using the
 pip install tensorflow-data-validation
 ```
 
-TFDV 0.9.0 currently requires TensorFlow Transform 0.9.0. Make sure to
-force install Transform 0.9.0 after installing TFDV, using the following
-command.
-
-```bash
-pip install tensorflow_transform==0.9.0
-```
-
 ## Installing from source
 
 ### 1. Prerequisites
@@ -55,7 +47,7 @@ directions](https://www.scipy.org/scipylib/download.html).
 
 #### Install Bazel
 
-If bazel is not installed on your system, install it now by following [these
+If Bazel is not installed on your system, install it now by following [these
 directions](https://bazel.build/versions/master/docs/install.html).
 
 ### 2. Clone the TFDV repository
@@ -115,7 +107,8 @@ other *untested* combinations may also work.
 
 |tensorflow-data-validation |tensorflow    |apache-beam[gcp]|
 |---------------------------|--------------|----------------|
-|GitHub master              |nightly (1.x) |2.6.0           |
+|GitHub master              |nightly (1.x) |2.8.0           |
+|0.11.0                     |1.11          |2.8.0           |
 |0.9.0                      |1.9           |2.6.0           |
 
 ## Questions

RELEASE.md

@@ -1,20 +1,27 @@
-# Current version (not yet released; still in development)
+# Release 0.11.0
 
 ## Major Features and Improvements
 
+* Add option to infer feature types from schema when generating statistics over CSV data.
+* Add utility method `set_domain` to set the domain of a feature in the schema.
+* Add option to compute weighted statistics by providing a weight feature.
 * Add a PTransform for decoding TF examples.
-* Add utility methods to write and load the schema protocol buffer.
+* Add utility methods `write_schema_text` and `load_schema_text` to write and load the schema protocol buffer.
 * Add option to compute statistics over a sample.
-* Add support for computing weighted common statistics.
+* Optimize performance of statistics computation (~2x improvement on benchmark datasets).
 
 ## Bug Fixes and Other Changes
 
+* Depends on `apache-beam[gcp]>=2.8,<3`.
+* Depends on `tensorflow-transform>=0.11,<0.12`.
+* Depends on `tensorflow-metadata>=0.9,<0.10`.
 * Fix bug in clearing oneof domain\_info field in Feature proto.
 * Fix overflow error for large integers by casting them to STRING type.
 * Added API docs.
 
 ## Breaking changes
 
+* Requires pre-installed `tensorflow>=1.11,<2`.
 * Make tf.Example decoder to represent a feature with no value list as a
   missing value (None).
 * Make StatsOptions as a class.

WORKSPACE

@@ -9,10 +9,11 @@ workspace(name = "tensorflow_data_validation")
 #    reliable downloads.
 load("//tensorflow_data_validation:repo.bzl", "tensorflow_http_archive")
 
+# v1.11.0
 tensorflow_http_archive(
     name = "org_tensorflow",
-    sha256 = "696c4906d6536ed8d9f8f13c4927d3ccf36dcf3e13bb352ab80cba6b1b9038d4",
-    git_commit = "25c197e02393bd44f50079945409009dd4d434f8",
+    sha256 = "025b47263af34475dc75da40c76a87934a70f69611e9b0b88445d65730f0fc73",
+    git_commit = "c19e29306ce1777456b2dbb3a14f511edf7883a8",
 )
 
 # TensorFlow depends on "io_bazel_rules_closure" so we need this here.

g3doc/api_docs/python/_toc.yaml

@@ -24,8 +24,12 @@ toc:
       path: /tfx/data_validation/api_docs/python/tfdv/get_feature
     - title: infer_schema
       path: /tfx/data_validation/api_docs/python/tfdv/infer_schema
+    - title: load_schema_text
+      path: /tfx/data_validation/api_docs/python/tfdv/load_schema_text
     - title: load_statistics
       path: /tfx/data_validation/api_docs/python/tfdv/load_statistics
+    - title: set_domain
+      path: /tfx/data_validation/api_docs/python/tfdv/set_domain
     - title: StatsOptions
       path: /tfx/data_validation/api_docs/python/tfdv/StatsOptions
     - title: TFExampleDecoder
@@ -36,3 +40,5 @@ toc:
       path: /tfx/data_validation/api_docs/python/tfdv/validate_statistics
     - title: visualize_statistics
       path: /tfx/data_validation/api_docs/python/tfdv/visualize_statistics
+    - title: write_schema_text
+      path: /tfx/data_validation/api_docs/python/tfdv/write_schema_text

g3doc/api_docs/python/index.md

@@ -14,6 +14,9 @@
 *  <a href="./tfdv/get_domain.md"><code>tfdv.get_domain</code></a>
 *  <a href="./tfdv/get_feature.md"><code>tfdv.get_feature</code></a>
 *  <a href="./tfdv/infer_schema.md"><code>tfdv.infer_schema</code></a>
+*  <a href="./tfdv/load_schema_text.md"><code>tfdv.load_schema_text</code></a>
 *  <a href="./tfdv/load_statistics.md"><code>tfdv.load_statistics</code></a>
+*  <a href="./tfdv/set_domain.md"><code>tfdv.set_domain</code></a>
 *  <a href="./tfdv/validate_statistics.md"><code>tfdv.validate_statistics</code></a>
-*  <a href="./tfdv/visualize_statistics.md"><code>tfdv.visualize_statistics</code></a>
+*  <a href="./tfdv/visualize_statistics.md"><code>tfdv.visualize_statistics</code></a>
+*  <a href="./tfdv/write_schema_text.md"><code>tfdv.write_schema_text</code></a>

g3doc/api_docs/python/tfdv.md

@@ -13,9 +13,9 @@ Init module for TensorFlow Data Validation.
 
 [`class DecodeCSV`](./tfdv/DecodeCSV.md): Decodes CSV records into an in-memory dict representation.
 
-[`class GenerateStatistics`](./tfdv/GenerateStatistics.md): Public API for generating data statistics.
+[`class GenerateStatistics`](./tfdv/GenerateStatistics.md): API for generating data statistics.
 
-[`class StatsOptions`](./tfdv/StatsOptions.md): Options for generating data statistics.
+[`class StatsOptions`](./tfdv/StatsOptions.md): Options for generating statistics.
 
 [`class TFExampleDecoder`](./tfdv/TFExampleDecoder.md): A decoder for decoding TF examples into tf data validation datasets.
 
@@ -37,9 +37,15 @@ Init module for TensorFlow Data Validation.
 
 [`infer_schema(...)`](./tfdv/infer_schema.md): Infer schema from the input statistics.
 
+[`load_schema_text(...)`](./tfdv/load_schema_text.md): Loads the schema stored in text format in the input path.
+
 [`load_statistics(...)`](./tfdv/load_statistics.md): Loads data statistics proto from file.
 
+[`set_domain(...)`](./tfdv/set_domain.md): Sets the domain for the input feature in the schema.
+
 [`validate_statistics(...)`](./tfdv/validate_statistics.md): Validate the input statistics against the provided input schema.
 
 [`visualize_statistics(...)`](./tfdv/visualize_statistics.md): Visualize the input statistics using Facets.
 
+[`write_schema_text(...)`](./tfdv/write_schema_text.md): Writes input schema to a file in text format.
+