v2.11.0-rc0

Release 2.11.0-rc0

Bug Fixes and Other Changes

• tensorflow:
  • Moving logging.h and bitmap from tf/core to tf/tsl.
• BOISE TF op:
  • Add main C++ functions for converting to/from BOISE labels to offsets
  • Add main C++ functions for converting to/from BOISE labels to offsets
  • Add main C++ functions for converting to/from BOISE labels to offsets
  • Add kernel code and Python API for OffsetsToBoiseTags op
• Other:
  • Add link to KPLs, fix typo in Neural machine translation with attention tutorial
  • Update README.md
  • Publish the tensorflow_models.nlp guide docs to tensorflow.org
  • Add missing dependency to constrained sequence kernel.
  • Add missing absl status dependency to sentence breaking utils.
  • Another missing absl status dependency. this time for sentence fragmenter.
  • Add absl status to sentence fragmenter v2.
  • Update pybind11 to 2.10.0 to match tensorflow.
  • Better error message for WordPiece when the vocabulary file has unicode issues.
  • Update Transformer tutorial with Keras MultiHeadAttention
  • transformers.ipynb: fix length filter and target slicing
  • transformers.ipynb: cleanup wording, create a PositionalEmbedding layer.
  • Replace tensorflow::Status::OK() with tensorflow::OkStatus().
  • Update README with note about various OS releases.
  • Cast the step type.
  • Reactivate TFLite ByteSplitter test.
  • Modify tokenizer to process pt_examples to tokenizers.pt
  • fix words alignment in documentation
  • Update nmt_with_attention:
  • transformers.ipynb: Factor out CrossAttention, GlobalSelfAttention, and CausalSelfAttention layers.
  • Switch the transformer to train with Model.fit.
  • Whitespace changes to force republish.
  • Add a phrase based tokenzier
  • Fix tutorial display, again.
  • Update version

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

satojkovic

v2.10.0

Release 2.10.0

Major Features and Improvements

• New ByteSplitter which tokenizes strings into bytes.
• New tutorial: Fine tune BERT with Orbit [will be added to tensorflow.org/text soon].
• Fixed an issue where dynamic TF Lite tensors were not getting resized correctly.

Bug Fixes and Other Changes

• Fix typo error in subwords_tokenizer guide with text.WordpieceTokenizer
• Fixes prepare_tf_dep.sh for OSX.
• Add cross-links to tensorflow_models.nlp API reference.
• (Generated change) Update tf.Text versions and/or docs.
• Update shape inference of kernel template for fast wordpiece and activate the op test.
• Update configure.sh for Apple Silicon.
• Export Trimmer ABC to be usable as tf_text.Trimmer
• Fix TensorFlow checkpoint and trackable imports.
• Correct tutorial explanation: meaning of attention weights
• Modernize fine_tune_bert.
• Lint and update the Fine-tuning a BERT model tutorial
• Use pointer for pointer math instead of iterator. Fixes c++17 compilation for regex_split on windows.
• Add install_bazel.sh script to make it easy to install the correctly needed version of Bazel. (#946)
• Make install_bazel.sh script executable.
• Prevent runtime errors from happening due to invalid regular expressions using regex_split & RegexSplitter.
• Centralize tensorflow-models docs into a top-level docs/ directory.
• Remove link to non-existant section on tf.org.
• Move fine_tune_bert guide.
• Updated the spelling mistakes in subwords_tokenizer.ipynb
• Fixes a bug caused by passing an empty tensor into SentencepieceTokenizer's detokenize method.
• Update build for Sentencepiece. Darts was not properly being depended on.
• Improve Sentencepiece build by adding missing dependency - str_format.
• Fix typos and lint Neural machine translation with attention tutorial
• Fix external link formatting, lint NMT with attention tutorial

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

gadagashwini, mnahinkhan, Steve R. Sun, synandi

v2.10.0-rc0

Release 2.10.0-rc0

Major Features and Improvements

• New ByteSplitter which tokenizes strings into bytes.
• New tutorial: Fine tune BERT with Orbit [will be added to tensorflow.org/text soon].
• Fixed an issue where dynamic TF Lite tensors were not getting resized correctly.

Bug Fixes and Other Changes

• Fix typo error in subwords_tokenizer guide with text.WordpieceTokenizer
• Fixes prepare_tf_dep.sh for OSX.
• Add cross-links to tensorflow_models.nlp API reference.
• (Generated change) Update tf.Text versions and/or docs.
• Update shape inference of kernel template for fast wordpiece and activate the op test.
• Update configure.sh for Apple Silicon.
• Export Trimmer ABC to be usable as tf_text.Trimmer
• Fix TensorFlow checkpoint and trackable imports.
• Correct tutorial explanation: meaning of attention weights
• Modernize fine_tune_bert.
• Lint and update the Fine-tuning a BERT model tutorial
• Use pointer for pointer math instead of iterator. Fixes c++17 compilation for regex_split on windows.
• Add install_bazel.sh script to make it easy to install the correctly needed version of Bazel. (#946)
• Make install_bazel.sh script executable.
• Prevent runtime errors from happening due to invalid regular expressions using regex_split & RegexSplitter.
• Centralize tensorflow-models docs into a top-level docs/ directory.
• Remove link to non-existant section on tf.org.
• Move fine_tune_bert guide.
• Updated the spelling mistakes in subwords_tokenizer.ipynb
• Fixes a bug caused by passing an empty tensor into SentencepieceTokenizer's detokenize method.
• Update build for Sentencepiece. Darts was not properly being depended on.
• Improve Sentencepiece build by adding missing dependency - str_format.
• Fix typos and lint Neural machine translation with attention tutorial
• Fix external link formatting, lint NMT with attention tutorial

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

gadagashwini, mnahinkhan, Steve R. Sun, synandi

v2.9.0

Release 2.9

Major Features and Improvements

• New FastBertNormalizer that improves speed for BERT normalization and is convertible to TF Lite.
• New FastBertTokenizer that combines FastBertNormalizer and FastWordpieceTokenizer.
• New ngrams kernel for handling STRING_JOIN reductions.

Bug Fixes and Other Changes

• NgramsStringJoin shape inference fixed to handle unranked tensors
• Upgrade pybind11 and reenable tests that were broken.
• Rename a couple files to match the naming of the other tflite kernels. Also adds some deps to tflite_ops that were missing and causing an error when testing :all.
• Add to TF Lite documentation that ngrams is a convertible op.
• Fix public access and missing ICU data to build_fast_bert_normalizer_model and enable the disabled tests.
• Update the doc for FastWordpieceTokenizer.
• Refine the doc for FastWordpieceTokenizer.
• Bug fix: make BertTokenizer work for RaggedTensors with row_splits_dtype=int32
• Fix typo error text.WordpieceTokenizer
• Added comma at missing places in emoticons for normalizer
• Refactor build and test scripts to use prepare_tf_dep.sh
• Fixes prepare_tf_dep.sh for OSX.
• Fixed bug in setup.py that was requiring the wrong version.
• Updated package with the correct versions of Python we release on.
• Update documentation on TF Lite convertible ops.
• Transition to use TF's version of bazel.
• Transition to use TF's bazel configuration.
• Add missing symbols for tokenization layers
• Fix typo in text_generation.ipynb
• Fix grammar typo
• Allow fast wordpiece tokenizer to take in external wordpiece model.
• Internal change
• Improvement to guide where mean call is redundant. See #810 for more info.
• Update broken link and fix typo in BERT-SNGP demo notebook
• Consolidate disparate test-related files into a single testing_infra folder.
• Pin tf-text version to guides & tutorials.
• Fix bug in constrained sequence op. Added a check on an edge case where num_steps = 0 should do nothing and prevent it from SIGSEV crashes.
• Remove outdated Keras tests due to them no longer making the testing utilities available.
• Update bert preprocessing by padding correct tensors
• Update tensorflow-text notebooks from 2.7 to 2.8
• Optimize FastWordPiece to only generate requested outputs.
• Add a note about byte-indexing vs character indexing.
• Add a MAX_TOKENS to the transformer tutorial.
• Only export tensorflow symbols from shared libs.
• (Generated change) Update tf.Text versions and/or docs.
• Do not run the prepare_tf_dep script for Apple M1 macs.
• Update text_classification_rnn.ipynb
• Fix the exported symbols for the linker test. By adding it to the share objects instead of the c++ code, it allows for the code to be compiled together in one large shared lib.
• Implement FastBertNormalizer based on codepoint-wise mappings.
• Add pybind for fast_bert_normalizer_model_builder.
• Remove unused comments related to Python 2 compatibility.
• update transformer.ipynb
• Update toolchain & temporarily disable tf lite tests.
• Define manylinux2014 for the new toolchain target, and have presubmits use it.
• Move tflite build deps to custom target.
• Add FastBertTokenizer.
• Update bazel version to 5.1.0
• Update TF Text to use new Ngrams kernel.
• Don't try to set dimension if shape is unknown for ngrams.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Aflah, Connor Brinton, devnev39, Janak Ramakrishnan, Martin, Nathan Luehr, Pierre Dulac, Rabin Adhikari, gadagashwini, mohantym, rtg0795

v2.10.0-b2

Release 2.10.0-b2

Major Features and Improvements

• Added FastSentencepieceTokenizer which is convertible to TF Lite. Please note the op name in the graph will change, so any models trained with this version will need to be retrained when the release candidate for 2.10 is released.

Important Notes

• This beta release is outside the normal release cycle and is meant to work with TF versions 2.8.x.
• Again, the op name for FSP will change in future releases.

v2.8.2

Release 2.8.2

Major Features and Improvements

• 📦️ Fix macOS packaging so it works with package managers like Poetry (#838)

Bug Fixes and Other Changes

• Package metadata updated with the correct available python versions.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Connor Brinton

v2.9.0-rc1

Release 2.9.0-rc1

Major Features and Improvements

• New FastBertNormalizer that improves speed for BERT normalization and is convertible to TF Lite.
• New FastBertTokenizer that combines FastBertNormalizer and FastWordpieceTokenizer.
• New ngrams kernel for handling STRING_JOIN reductions.

Bug Fixes and Other Changes

• Fixed bug in setup.py that was requiring the wrong version.
• Updated package with the correct versions of Python we release on.
• Update documentation on TF Lite convertible ops.
• Transition to use TF's version of bazel.
• Transition to use TF's bazel configuration.
• Add missing symbols for tokenization layers
• Fix typo in text_generation.ipynb
• Fix grammar typo
• Allow fast wordpiece tokenizer to take in external wordpiece model.
• Internal change
• Improvement to guide where mean call is redundant. See #810 for more info.
• Update broken link and fix typo in BERT-SNGP demo notebook
• Consolidate disparate test-related files into a single testing_infra folder.
• Pin tf-text version to guides & tutorials.
• Fix bug in constrained sequence op. Added a check on an edge case where num_steps = 0 should do nothing and prevent it from SIGSEV crashes.
• Remove outdated Keras tests due to them no longer making the testing utilities available.
• Update bert preprocessing by padding correct tensors
• Update tensorflow-text notebooks from 2.7 to 2.8
• Optimize FastWordPiece to only generate requested outputs.
• Add a note about byte-indexing vs character indexing.
• Add a MAX_TOKENS to the transformer tutorial.
• Only export tensorflow symbols from shared libs.
• (Generated change) Update tf.Text versions and/or docs.
• Do not run the prepare_tf_dep script for Apple M1 macs.
• Update text_classification_rnn.ipynb
• Fix the exported symbols for the linker test. By adding it to the share objects instead of the c++ code, it allows for the code to be compiled together in one large shared lib.
• Implement FastBertNormalizer based on codepoint-wise mappings.
• Add pybind for fast_bert_normalizer_model_builder.
• Remove unused comments related to Python 2 compatibility.
• update transformer.ipynb
• Update toolchain & temporarily disable tf lite tests.
• Define manylinux2014 for the new toolchain target, and have presubmits use it.
• Move tflite build deps to custom target.
• Add FastBertTokenizer.
• Update bazel version to 5.1.0
• Update TF Text to use new Ngrams kernel.
• Don't try to set dimension if shape is unknown for ngrams.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Aflah, Connor Brinton, devnev39, Janak Ramakrishnan, Martin, Nathan Luehr, Pierre Dulac, Rabin Adhikari

v2.9.0-rc0

Release 2.9.0-rc0

Major Features and Improvements

• New FastBertNormalizer that improves speed for BERT normalization and is convertible to TF Lite.
• New FastBertTokenizer that combines FastBertNormalizer and FastWordpieceTokenizer.
• New ngrams kernel for handling STRING_JOIN reductions.

Bug Fixes and Other Changes

• Add missing symbols for tokenization layers
• Fix typo in text_generation.ipynb
• Fix grammar typo
• Allow fast wordpiece tokenizer to take in external wordpiece model.
• Internal change
• Improvement to guide where mean call is redundant. See #810 for more info.
• Update broken link and fix typo in BERT-SNGP demo notebook
• Consolidate disparate test-related files into a single testing_infra folder.
• Pin tf-text version to guides & tutorials.
• Fix bug in constrained sequence op. Added a check on an edge case where num_steps = 0 should do nothing and prevent it from SIGSEV crashes.
• Remove outdated Keras tests due to them no longer making the testing utilities available.
• Update bert preprocessing by padding correct tensors
• Update tensorflow-text notebooks from 2.7 to 2.8
• Optimize FastWordPiece to only generate requested outputs.
• Add a note about byte-indexing vs character indexing.
• Add a MAX_TOKENS to the transformer tutorial.
• Only export tensorflow symbols from shared libs.
• (Generated change) Update tf.Text versions and/or docs.
• Do not run the prepare_tf_dep script for Apple M1 macs.
• Update text_classification_rnn.ipynb
• Fix the exported symbols for the linker test. By adding it to the share objects instead of the c++ code, it allows for the code to be compiled together in one large shared lib.
• Implement FastBertNormalizer based on codepoint-wise mappings.
• Add pybind for fast_bert_normalizer_model_builder.
• Remove unused comments related to Python 2 compatibility.
• update transformer.ipynb
• Update toolchain & temporarily disable tf lite tests.
• Define manylinux2014 for the new toolchain target, and have presubmits use it.
• Move tflite build deps to custom target.
• Add FastBertTokenizer.
• Update bazel version to 5.1.0
• Update TF Text to use new Ngrams kernel.
• Don't try to set dimension if shape is unknown for ngrams.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Aflah, Connor Brinton, devnev39, Janak Ramakrishnan, Martin, Nathan Luehr, Pierre Dulac, Rabin Adhikari

v2.8.1

Release 2.8.1

Major Features and Improvements

• Upgrade Sentencepiece to v0.1.96
• Adds new trimmer ShrinkLongestTrimmer

Bug Fixes and Other Changes

• Upgrade bazel to 4.2.2
• Create .bazelversion file to guarantee using correct version
• Update tf.Text versions and docs.
• Add Apple Silicon support for manual builds.
• Update configure.sh
• Only Apple Silicon will be installed with tensorflow-macos
• Fix merge error & add SP patch for building on Windows
• Fix inclusion of missing libraries for Mac & Windows
• Update word_embeddings.ipynb
• Update classify_text_with_bert.ipynb
• Update tensorflow_text tutorials to new preprocessing layer symbol path
• Fixes typo in guide
• Update Apple Silicon's requires.
• release script to use tf nighly
• Fix typo in ragged tensor link.
• Update requires for setup. It wasn't catching non-M1 Macs.
• Add missing symbols for tokenization layers
• Fix typo in text_generation.ipynb
• Fix grammar typo
• Allow fast word piece tokenizer to take in external word piece model.
• Update guide with redundant mean call.
• Update broken link and fix typo in BERT-SNGP demo notebook.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Abhijeet Manhas, chunduriv, Dean Wyatte, Feiteng, jaymessina3, Mao, Olivier Bacs, RenuPatelGoogle, Steve R. Sun, Stonepia, sun1638650145, Tharaka De Silva, thuang513, Xiaoquan Kong, devnev39, Janak Ramakrishnan, Pierre Dulac

v2.8.0-rc0

Release 2.8.0-rc0

Major Features and Improvements

• Upgrade Sentencepiece to v0.1.96
• Adds new trimmer ShrinkLongestTrimmer

Bug Fixes and Other Changes

• Upgrade bazel to 4.2.2
• Create .bazelversion file to guarantee using correct version
• (Generated change) Update tf.Text versions and/or docs.
• Add Apple Silicon support for manual builds.
• Update configure.sh
• Only Apple Silicon will be installed with tensorflow-macos
• Fix merge error & add SP patch for building on Windows
• Fix inclusion of missing libraries for Mac & Windows
• Update word_embeddings.ipynb
• Update classify_text_with_bert.ipynb
• Update tensorflow_text tutorials to new preprocessing layer symbol path
• Fixes typo in guide
• Update Apple Silicon's requires.
• release script to use tf nighly
• Fix typo in ragged tensor link.
• Update requires for setup. It wasn't catching non-M1 Macs.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Abhijeet Manhas, chunduriv, Dean Wyatte, Feiteng, jaymessina3, Mao, Olivier Bacs, RenuPatelGoogle, Steve R. Sun, Stonepia, sun1638650145, Tharaka De Silva, thuang513, Xiaoquan Kong