v0.4.0

[mattdangerw]

The 0.4 release adds support for pretrained models to the library via keras_nlp.models. You can read an
introduction to the new API in our Getting Started Guide.

If you encounter any problems or have questions, please open an issue!

Breaking Changes

• Renamed keras_nlp.layers.MLMHead -> keras_nlp.layers.MaskedLMHead.
• Renamed keras_nlp.layers.MLMMaskGenerator -> keras_nlp.layers.MaskedLMMaskGenerator.
• Renamed keras_nlp.layers.UnicodeCharacterTokenizer -> keras_nlp.layers.UnicodeCodepointTokenizer.
• Switched the default of lowercase in keras_nlp.tokenizers.WordPieceTokenizer from True to False.
• Renamed the token id output of MaskedLMMaskGenerator from "tokens" to "tokens_ids".

Summary

• Added the keras_nlp.models API.
  • Added support for BERT, DistilBERT, RoBERTa, and XLM-RoBERTa models and pretrained checkpoints.
  • See our Getting Started Guide for more details.
• Added new metrics.
  • keras_nlp.metrics.Bleu and keras_nlp.metrics.EditDistance.
• Added new vocabulary training utilities.
  • keras_nlp.tokenizers.compute_word_piece_vocabulary and keras_nlp.tokenizers.compute_sentence_piece_proto.
• Added new preprocessing layers.
  • keras_nlp.layers.RandomSwap and keras_nlp.layers.RandomDeletion.

What's Changed

• Add Edit Distance Metric by @abheesht17 in Add Edit Distance Metric #231
• Minor fix to simplify and test handling of max_length prompts by @jbischof in Minor fix to simplify and test handling of max_length prompts #258
• Remove split regex args for WordPieceTokenizer by @mattdangerw in Remove split regex args for WordPieceTokenizer #255
• Add instructions on installing the latest changes by @mattdangerw in Add instructions on installing the latest changes #261
• Add warning when k > vocab_size in top_k_search by @jbischof in Add warning when k > vocab_size in top_k_search #260
• Fix keras library imports and usage by @jbischof in Fix keras library imports and usage #262
• Add BLEU Score by @abheesht17 in Add BLEU Score  #222
• Configure GKE-based accelerator testing by @chenmoneygithub in Configure GKE-based accelerator testing #265
• Added WordPieceTokenizer training function by @jessechancy in Added WordPieceTokenizer training function #256
• Add requirements.txt for cloud build by @chenmoneygithub in Add requirements.txt for cloud build #267
• Global Seed Bug Fix by @jessechancy in Global Seed Bug Fix #269
• Update accelerator testing to use the new GCP project by @chenmoneygithub in Update accelerator testing to use the new GCP project #272
• Fixed typo: "recieved" by @ehrencrona in Fixed typo: "recieved" #273
• Reuse dense pooled output for fine tuning by @mattdangerw in Reuse dense pooled output for fine tuning #251
• Simplify BERT modeling, use keras embeddings by @mattdangerw in Simplify BERT modeling, use keras embeddings #253
• Rename UnicodeCharacterTokenizer>UnicodeCodepointTokenizer by @mattdangerw in Rename UnicodeCharacterTokenizer>UnicodeCodepointTokenizer #254
• Add README for accelerator testing config folder by @chenmoneygithub in Add README for accelerator testing config folder #276
• Random Deletion Layer by @aflah02 in Random Deletion Layer #214
• Made trainer more efficient. Loading full files instead of using TextLineDataset. by @jessechancy in Made trainer more efficient. Loading full files instead of using TextLineDataset. #280
• Use KerasNLP for BERT preprocessing for GLUE by @mattdangerw in Use KerasNLP for BERT preprocessing for GLUE #252
• Minor fixes to the Random Deletion Layer by @aflah02 in Minor fixes to the Random Deletion Layer #286
• Fixes for WordPieceTrainer by @aflah02 in Fixes for WordPieceTrainer #293
• Update default to strip_accents=False by @jessechancy in Update default to strip_accents=False #289
• Move Bert to models folder by @jbischof in Move Bert to models folder #288
• Make Decoding Functions Graph-compatible (with XLA Support!) by @abheesht17 in Make Decoding Functions Graph-compatible (with XLA Support!) #271
• SentencePieceTrainer by @aflah02 in SentencePieceTrainer #281
• Rename models.Bert() to models.BertCustom() by @jbischof in Rename models.Bert() to models.BertCustom() #310
• Add a test for variable sequence length inputs by @mattdangerw in Add a test for variable sequence length inputs #313
• Support checkpoint loading for BertBase by @jbischof in Support checkpoint loading for BertBase #299
• RoBERTa pretrained model forward pass by @jessechancy in RoBERTa pretrained model forward pass #304
• Register objects as serializable by @mattdangerw in Register objects as serializable #292
• Style merging for Bert and Roberta by @jbischof in Style merging for Bert and Roberta #315
• Streamline and speed up tests by @jbischof in Streamline and speed up tests #324
• Add Support for CJK Char Splitting for WordPiece Tokenizer by @abheesht17 in Add Support for CJK Char Splitting for WordPiece Tokenizer #318
• Clean up model input names for consistency by @mattdangerw in Clean up model input names for consistency #327
• Return a single tensor from roberta by @mattdangerw in Return a single tensor from roberta #328
• BERT, RoBERTa: Add model.compile UTs by @abheesht17 in BERT, RoBERTa: Add model.compile UTs #330
• Continue rename of bert model inputs by @mattdangerw in Continue rename of bert model inputs #329
• Text Generation Utilities: Add Support for Ragged Inputs by @abheesht17 in Text Generation Utilities: Add Support for Ragged Inputs #300
• bert_base_zh, bert_base_multi_cased: Add BERT Base Variants by @abheesht17 in bert_base_zh, bert_base_multi_cased: Add BERT Base Variants #319
• WordPiece vocabularies trainer on Wikipedia dataset by @jessechancy in WordPiece vocabularies trainer on Wikipedia dataset #316
• Use the exported ragged ops for RandomDeletion by @mattdangerw in Use the exported ragged ops for RandomDeletion #332
• Random Swap Layer by @aflah02 in Random Swap Layer #224
• Fixes for Random Deletion Layer by @aflah02 in Fixes for Random Deletion Layer #339
• Move cloudbuild to a hidden directory by @mattdangerw in Move cloudbuild to a hidden directory #345
• Fix the build by @mattdangerw in Fix the build #349
• Migrating from Datasets to TFDS for GLUE Example by @aflah02 in Migrating from Datasets to TFDS for GLUE Example #340
• Move network_tests into keras_nlp/ by @mattdangerw in Move network_tests into keras_nlp/ #344
• Stop hardcoding 2.9 by @mattdangerw in Stop hardcoding 2.9 #351
• Add BERT Large by @abheesht17 in Add BERT Large #331
• Add normalize_first arg to Transformer Layers by @abheesht17 in Add normalize_first arg to Transformer Layers #350
• Add Small BERT Variants by @abheesht17 in Add Small BERT Variants #338
• Beam Search: Add Ragged and XLA Support by @abheesht17 in Beam Search: Add Ragged and XLA Support #341
• Fix download paths for bert weights by @mattdangerw in Fix download paths for bert weights #356
• Add a BertPreprocessor class by @mattdangerw in Add a BertPreprocessor class #343
• Text Generation Functions: Add Benchmark Script by @abheesht17 in Text Generation Functions: Add Benchmark Script #342
• Improve readability for encoder/decoder blocks by @mattdangerw in Improve readability for encoder/decoder blocks #353
• Add GPT-2 Model and its Variants by @abheesht17 in Add GPT-2 Model and its Variants #354
• Clean up BERT, RoBERTa doc-strings by @abheesht17 in Clean up BERT, RoBERTa doc-strings #359
• Create unique string id for each BERT backbone by @jbischof in Create unique string id for each BERT backbone #361
• Use model.fit() for BERT Example by @abheesht17 in Use model.fit() for BERT Example #360
• Minor Fixes in BertPreprocessor Layer by @abheesht17 in Minor Fixes in BertPreprocessor Layer #373
• Clone user passed initializers called multiple times by @mattdangerw in Clone user passed initializers called multiple times #371
• Update BERT model file structure by @mattdangerw in Update BERT model file structure #376
• Move gpt model code into a directory by @mattdangerw in Move gpt model code into a directory #379
• Move roberta model code into a directory by @mattdangerw in Move roberta model code into a directory #380
• Reorg test directories by @mattdangerw in Reorg test directories #384
• Add XLM-RoBERTa by @abheesht17 in Add XLM-RoBERTa #372
• Add DistilBERT by @abheesht17 in Add DistilBERT #382
• Stop running CI on Windows by @mattdangerw in Stop running CI on Windows #386
• Fix Bert serialization by @mattdangerw in Fix Bert serialization #385
• Improve MacOS support and pin tensorflow version during testing by @mattdangerw in Improve MacOS support and pin tensorflow version during testing #383
• Unify BERT model API in one class by @jbischof in Unify BERT model API in one class #387
• Add from_preset constructor to BertPreprocessor by @jbischof in Add from_preset constructor to BertPreprocessor #390
• More robustly test BERT preprocessing by @mattdangerw in More robustly test BERT preprocessing #394
• Move name and trainable to kwargs by @jbischof in Move name and trainable to kwargs #399
• Add backbone as property for task models by @jbischof in Add backbone as property for task models #398
• Set default name of Bert instance to "backbone" by @jbischof in Set default name of Bert instance to "backbone" #397
• Fix gpt2 serialization by @mattdangerw in Fix gpt2 serialization #391
• Fix distilbert serialization by @mattdangerw in Fix distilbert serialization #392
• Fix roberta and xlm-roberta serialization by @mattdangerw in Fix roberta and xlm-roberta serialization #393
• Register the BertPreprocessor as serializable by @mattdangerw in Register the BertPreprocessor as serializable #401
• BPE tokenizer by @chenmoneygithub in BPE tokenizer #389
• Change GPT-2's Format to Mirror BERT's by @abheesht17 in Change GPT-2's Format to Mirror BERT's #418
• Fix bert preprocessing docstring so it is runnable by @mattdangerw in Fix bert preprocessing docstring so it is runnable #421
• Change RoBERTa and XLM-RoBERTa's Format to Mirror BERT's by @abheesht17 in Change RoBERTa and XLM-RoBERTa's Format to Mirror BERT's #417
• Update distilbert to mirror recent bert changes by @mattdangerw in Update distilbert to mirror recent bert changes #406
• Change gpt2 to GPT2 by @sampathweb in Change gpt2 to GPT2 #425
• Fix byte pair detokenization of 2d arrays by @mattdangerw in Fix byte pair detokenization of 2d arrays #423
• Never pass Raggeds to user function when generating text by @mattdangerw in Never pass Raggeds to user function when generating text #424
• Add XLMRobertaClassifier by @abheesht17 in Add XLMRobertaClassifier #422
• Add RobertaPreprocessor Layer by @abheesht17 in Add RobertaPreprocessor Layer #419
• Update Style Guide for naming of Models and Layers by @sampathweb in Update Style Guide for naming of Models and Layers #434
• Support String Output for BytePairTokenizer by @abheesht17 in Support String Output for BytePairTokenizer #438
• Improve our continuous testing for model presets by @mattdangerw in Improve our continuous testing for model presets #357
• Remove remote files from BPE docstring by @jbischof in Remove remote files from BPE docstring #440
• Add DistilBertClassifier by @abheesht17 in Add DistilBertClassifier #437
• Remove lingering reference to BertCustom by @mattdangerw in Remove lingering reference to BertCustom #441
• Add XLM-RoBERTa Tokenizer (SPM) by @abheesht17 in Add XLM-RoBERTa Tokenizer (SPM) #428
• Add a disclaimer for use of model checkpoints by @mattdangerw in Add a disclaimer for use of model checkpoints #430
• Add a disclaimer to our README by @mattdangerw in Add a disclaimer to our README #431
• Fix our BERT GLUE example so it runs again by @mattdangerw in Fix our BERT GLUE example so it runs again #444
• Add backbone presets to task classes by @jbischof in Add backbone presets to task classes  #448
• Split the Bert tokenizer to a separate class by @mattdangerw in Split the Bert tokenizer to a separate class #449
• Conditionally import tf text by @mattdangerw in Conditionally import tf text #452
• Copy our model disclaimer to the distilbert classifier by @mattdangerw in Copy our model disclaimer to the distilbert classifier #453
• Fix regex string for BPE by @chenmoneygithub in Fix regex string for BPE #458
• Fix docstrings and add note to style guide by @jbischof in Fix docstrings and add note to style guide #464
• Allow formatting our docstrings inline by @mattdangerw in Allow formatting our docstrings inline #450
• Update self.assertEquals with self.assertEqual by @MaximSmolskiy in Update self.assertEquals with self.assertEqual #466
• Document our release process by @mattdangerw in Document our release process #473
• Add RobertaTokenizer by @abheesht17 in Add RobertaTokenizer #468
• Add DistilBertTokenizer by @abheesht17 in Add DistilBertTokenizer #469
• Modify XLMRobertaTokenizer to Match BERT by @abheesht17 in Modify XLMRobertaTokenizer to Match BERT #471
• Add GPT2Tokenizer by @abheesht17 in Add GPT2Tokenizer #470
• Minor fix to git commands by @mattdangerw in Minor fix to git commands #475
• Version bump to 0.4.0 by @mattdangerw in Version bump to 0.4.0 #476
• Clarify comment on BERT preset testing by @mattdangerw in Clarify comment on BERT preset testing #477
• Fix the nightly build by @mattdangerw in Fix the nightly build #484
• Bump tf and tf-text to 2.11 by @mattdangerw in Bump tf and tf-text to 2.11 #490
• Consolidate preset testing by @mattdangerw in Consolidate preset testing #480
• Allow BertPreprocessor to map labeled datasets by @mattdangerw in Allow BertPreprocessor to map labeled datasets #478
• Glue eval script by @chenmoneygithub in Glue eval script  #445
• Update Requirements and Python version in setup.py by @sampathweb in Update Requirements and Python version in setup.py #495
• First task-level preset with BertClassifier by @jbischof in First task-level preset with BertClassifier #494
• Add a helper model to automatically apply preprocessing by @mattdangerw in Add a helper model to automatically apply preprocessing #346
• Add GPT2 Presets by @abheesht17 in Add GPT2 Presets #472
• fix incorrect flag by @chenmoneygithub in fix incorrect flag #496
• Add instructions on how to update deps of GPU testing by @chenmoneygithub in Add instructions on how to update deps of GPU testing #499
• Add XLM-RoBERTa Presets by @abheesht17 in Add XLM-RoBERTa Presets #482
• Add DistilBERT Presets by @abheesht17 in Add DistilBERT Presets #479
• Add RoBERTa Presets by @abheesht17 in Add RoBERTa Presets #506
• Fix nightly builds by @mattdangerw in Fix nightly builds #522
• Remove typo by @jbischof in Remove typo #515
• Fix Model Doc-string Examples by @abheesht17 in Fix Model Doc-string Examples #516
• Mark format.sh executable again by @mattdangerw in Mark format.sh executable again #518
• Make BertClassifier operate directly on raw string inputs by @mattdangerw in Make BertClassifier operate directly on raw string inputs #485
• Fix nightlies take two by @mattdangerw in Fix nightlies take two #525
• Use tf.ones for docstring example input by @jbischof in Use tf.ones for docstring example input #524
• Fix the index order of GLUE script and other bugs by @chenmoneygithub in Fix the index order of GLUE script and other bugs #517
• Standardize on "backbone" naming for BERT by @jbischof in Standardize on "backbone" naming for BERT #536
• Add dropout to BertClassifier by @chenmoneygithub in Add dropout to BertClassifier #540
• Preprocess string lists as a batch of single segments by @mattdangerw in Preprocess string lists as a batch of single segments #504
• Update model and file names for DistilBert by @ADITYADAS1999 in Update model and file names for DistilBert #541
• Rename filenames in models/ to match classnames by @mattdangerw in Rename filenames in models/ to match classnames #548
• Split tokenizers into their own file by @mattdangerw in Split tokenizers into their own file #549
• Add a note about our new file naming conventions by @mattdangerw in Add a note about our new file naming conventions #553
• Add distribution support for GLUE script by @chenmoneygithub in Add distribution support for GLUE script #544
• Make out preset ids more consistent by @mattdangerw in Make out preset ids more consistent #552
• Rename DistilBert -> DistilBertBackbone by @mattdangerw in Rename DistilBert -> DistilBertBackbone #551
• Rename Roberta -> RobertaBackbone (and same for XLM*) by @mattdangerw in Rename Roberta -> RobertaBackbone (and same for XLM*) #550
• Fix link in glue_benchmark README by @mattdangerw in Fix link in glue_benchmark README #557
• Remove qualification for PRESET_NAMES by @jbischof in Remove qualification for PRESET_NAMES #554
• Use black[jupyter] to format notebooks by @jbischof in Use black[jupyter] to format notebooks #556
• Temporarily drop GPT2 from our init.py by @mattdangerw in Temporarily drop GPT2 from our __init__.py #560
• Replicate Standardize on "backbone" naming for BERT #536 changing GPT2 -> GPT2Backbone by @ADITYADAS1999 in Replicate #536 changing GPT2 -> GPT2Backbone #558
• Change Backbone Names by @abheesht17 in Change Backbone Names #559
• Raise a friendly error message for unbatched input by @mattdangerw in Raise a friendly error message for unbatched input #545
• Fix Typo in Backbone Doc-strings by @abheesht17 in Fix Typo in Backbone Doc-strings #565
• Fix LinearDecayWithWarmup crash in BERT model. by @reedwm in Fix LinearDecayWithWarmup crash in BERT model. #564
• Update XLMRobertaPreprocessor to mirror recent changes by @mattdangerw in Update XLMRobertaPreprocessor to mirror recent changes #568
• Update RobertaPreprocessor to mirror recent changes by @mattdangerw in Update RobertaPreprocessor to mirror recent changes #567
• Fix minor typo in BertPreprocessor layer by @mattdangerw in Fix minor typo in BertPreprocessor layer #569
• Update DistilBertPreprocessor to mirror recent changes by @mattdangerw in Update DistilBertPreprocessor to mirror recent changes #566
• Point presets to url containing "v1/" by @chenmoneygithub in Point presets to url containing "v1/"  #577
• Stop testing h5 saved model format, start testing keras_v3 by @mattdangerw in Stop testing h5 saved model format, start testing keras_v3 #521
• fix the gpu testing by @chenmoneygithub in fix the gpu testing #581
• Make DistilBertClassifier operate directly on raw string inputs by @mattdangerw in Make DistilBertClassifier operate directly on raw string inputs #578
• Make RobertaClassifier operate directly on raw string inputs by @chenmoneygithub in Make RobertaClassifier operate directly on raw string inputs #579
• Export XLM-Roberta Classifier by @mattdangerw in Export XLM-Roberta Classifier #580
• Make XLMRobertaClassifier operate directly on raw string inputs by @mattdangerw in Make XLMRobertaClassifier operate directly on raw string inputs #583
• Fix some misc typos for distilbert by @mattdangerw in Fix some misc typos for distilbert #584
• Report GLUE score and hyperparameter settings by @chenmoneygithub in Report GLUE score and hyperparameter settings #585
• Remove support for Python 3.7 to align with keras-nightly by @sampathweb in Remove support for Python 3.7 to align with keras-nightly #590
• Add DeBERTa v3 Model by @abheesht17 in Add DeBERTa v3 Model #435
• Add DeBERTa Tokenizer and Preprocessor Classes by @abheesht17 in Add DeBERTa Tokenizer and Preprocessor Classes #589
• Add Dropout to *Classifier Doc-strings by @abheesht17 in Add Dropout to *Classifier Doc-strings #595
• Rename MLM -> MaskedLM for all library symbols by @mattdangerw in Rename MLM -> MaskedLM for all library symbols #598
• File-level Doc-string Changes for Classifiers and Presets by @abheesht17 in File-level Doc-string Changes for Classifiers and Presets #604
• Add DebertaClassifier and DeBERTa Presets by @abheesht17 in Add DebertaClassifier and DeBERTa Presets #594
• Rename Deberta -> DebertaV3 by @mattdangerw in Rename Deberta -> DebertaV3 #605
• Version bump to 0.4.0.dev0 by @mattdangerw in Version bump to 0.4.0.dev0 #609
• Add mixed precision support for glue script by @chenmoneygithub in Add mixed precision support for glue script #608
• Remove deberta from 0.4 release by @chenmoneygithub in Remove deberta from 0.4 release #607
• Update README for v0.4 by @jbischof in Update README for v0.4 #588
• Remove the dev prefix for final release by @mattdangerw in Remove the dev prefix for final release #613
• Rename preset IDs for consistency by @mattdangerw in Rename preset IDs for consistency #612

New Contributors

• @jbischof made their first contribution in Minor fix to simplify and test handling of max_length prompts #258
• @ehrencrona made their first contribution in Fixed typo: "recieved" #273
• @sampathweb made their first contribution in Change gpt2 to GPT2 #425
• @MaximSmolskiy made their first contribution in Update self.assertEquals with self.assertEqual #466
• @ADITYADAS1999 made their first contribution in Update model and file names for DistilBert #541
• @reedwm made their first contribution in Fix LinearDecayWithWarmup crash in BERT model. #564

Full Changelog: v0.3.0...v0.4.0

---

This discussion was created from the release v0.4.0.