v0.4.1

[mattdangerw]

The 0.4.1 release is a minor release with new model architectures and compilation defaults for task models. If you encounter any problems or have questions, please open an issue!

Summary

• Added compilation defaults for all task models (e.g. keras_nlp.models.BertClassifier). No existing functionality is changed, but users of task models can now skip calling .compile() and use default learning rates and optimizer strategies provided by the library.
• Added keras_nlp.models.AlbertBackbone, keras_nlp.models.AlbertClassifier, preprocessor, and tokenizer layers for pre-trained ALBERT models.
• Added keras_nlp.models.FNetBackbone, keras_nlp.models.FNetClassifier, preprocessor, and tokenizer layers for pre-trained FNet models.
• Added keras_nlp.models.DebertaV3Backbone, keras_nlp.models.DebertaV3Classifier, preprocessor, and tokenizer layers for pre-trained DeBERTaV3 models.

What's Changed

• Update python version in readme to 3.8 by @haifeng-jin in Update python version in readme to 3.8 #618
• Modify our pip install line so we upgrade tf by @mattdangerw in Modify our pip install line so we upgrade tf #616
• Use Adam optimizer for quick start by @mattdangerw in Use Adam optimizer for quick start #620
• Clean up class name and self in calls to super() by @mbrukman in Clean up class name and self in calls to super() #628
• Update word_piece_tokenizer.py by @ADITYADAS1999 in Update word_piece_tokenizer.py #617
• Add DeBERTaV3 Conversion Script by @abheesht17 in Add DeBERTaV3 Conversion Script #633
• Add AlbertTokenizer and AlbertPreprocessor by @abheesht17 in Add AlbertTokenizer and AlbertPreprocessor #627
• Create Backbone base class by @jbischof in Create Backbone base class #621
• Add TPU testing by @chenmoneygithub in Add TPU testing #591
• Add Base Preprocessor Class by @abheesht17 in Add Base Preprocessor Class #638
• Add keras_nlp.samplers by @chenmoneygithub in Add keras_nlp.samplers #563
• Add ALBERT Backbone by @abheesht17 in Add ALBERT Backbone #622
• Add a small script to count parameters in our presets by @mattdangerw in Add a small script to count parameters in our presets #610
• Clean up examples/ directory by @ADITYADAS1999 in Clean up examples/ directory #637
• Fix Small BERT Typo by @abheesht17 in Fix Small BERT Typo #651
• Rename examples/bert -> examples/bert_pretraining by @mattdangerw in Rename examples/bert -> examples/bert_pretraining #647
• Add FNet Preprocessor by @abheesht17 in Add FNet Preprocessor #646
• Add FNet Backbone by @abheesht17 in Add FNet Backbone #643
• Small DeBERTa Docstring Fixes by @abheesht17 in Small DeBERTa Docstring Fixes #666
• Add Fenced Docstring Testing by @abheesht17 in Add Fenced Docstring Testing #640
• Corrected the epsilon value by @soma2000-lang in Corrected the epsilon value  #665
• Consolidate docstring formatting weirdness in Backbone and Preprocessor base classes by @mattdangerw in Consolidate docstring formatting weirdness in Backbone and Preprocessor base classes #654
• Fix value_dim in TransformerDecoder's cross-attn layer by @abheesht17 in Fix value_dim in TransformerDecoder's cross-attn layer #667
• Add ALBERT Presets by @abheesht17 in Add ALBERT Presets #655
• Add Base Task Class by @abheesht17 in Add Base Task Class #671
• Implement TopP, TopK and Beam samplers by @chenmoneygithub in Implement TopP, TopK and Beam samplers #652
• Add FNet Presets by @abheesht17 in Add FNet Presets #659
• Bump the year to 2023 by @mattdangerw in Bump the year to 2023 #679
• Add BART Backbone by @abheesht17 in Add BART Backbone #661
• Handle trainable and name in the backbone base class by @mattdangerw in Handle trainable and name in the backbone base class #680
• Ignore Task Docstring for Testing by @abheesht17 in Ignore Task Docstring for Testing #683
• Light-weight benchmarking script by @NusretOzates in Light-weight benchmarking script #664
• Conditionally import tf_text everywhere by @mattdangerw in Conditionally import tf_text everywhere #684
• Expose token_embedding as a Backbone Property by @abheesht17 in Expose token_embedding as a Backbone Property #676
• Move from_preset to base tokenizer classes by @shivance in Move from_preset to base tokenizer classes #673
• add f_net_classifier and f_net_classifier_test by @ADITYADAS1999 in add f_net_classifier and f_net_classifier_test #670
• import rouge_scorer directly from rouge_score package by @sampathweb in import rouge_scorer directly from rouge_score package #691
• Fix typo in requirements file juypter -> jupyter by @mattdangerw in Fix typo in requirements file juypter -> jupyter #693
• Temporary fix to get nightly green again by @mattdangerw in Temporary fix to get nightly green again #696
• GPT2 Text Generation APIs by @chenmoneygithub in GPT2 Text Generation APIs #592
• Run keras saving tests on nightly and fix RobertaClassifier test by @mattdangerw in Run keras saving tests on nightly and fix RobertaClassifier test #692
• Speed up pip install keras-nlp; simplify deps by @mattdangerw in Speed up pip install keras-nlp; simplify deps #697
• Add AlbertClassifier by @shivance in Add AlbertClassifier #668
• Make tokenizer, backbone, preprocessor properties settable on base class by @mattdangerw in Make tokenizer, backbone, preprocessor properties settable on base class #700
• Update to latest black by @mattdangerw in Update to latest black #708
• RobertaMaskedLM task and preprocessor by @mattdangerw in RobertaMaskedLM task and preprocessor #653
• Default compilation for BERT/RoBERTa classifiers by @jbischof in Default compilation for BERT/RoBERTa classifiers #695
• Add start/end token padding to GPT2Preprocessor by @chenmoneygithub in Add start/end token padding to GPT2Preprocessor #704
• Don't install tf stable when building our nightly image by @mattdangerw in Don't install tf stable when building our nightly image #711
• Add OPT Backbone and Tokenizer by @mattdangerw in Add OPT Backbone and Tokenizer #699
• Small OPT Doc-string Edits by @abheesht17 in Small OPT Doc-string Edits #716
• Default compilation other classifiers by @Plutone11011 in Default compilation other classifiers #714
• Add BartTokenizer and BART Presets by @abheesht17 in Add BartTokenizer and BART Presets #685
• Add an add_prefix_space Arg in BytePairTokenizer by @shivance in Add an add_prefix_space Arg in BytePairTokenizer #715
• Opt presets by @mattdangerw in Opt presets #707
• fix import of tensorflow_text in tf_utils by @sampathweb in fix import of tensorflow_text in tf_utils #723
• Check for masked token in roberta tokenizer by @mattdangerw in Check for masked token in roberta tokenizer #742
• Improve test coverage for special tokens in model tokenizers by @mattdangerw in Improve test coverage for special tokens in model tokenizers #743
• Fix the sampler truncation strategy by @chenmoneygithub in Fix the sampler truncation strategy #713
• Add ALBERT Conversion Script by @abheesht17 in Add ALBERT Conversion Script #736
• Add FNet Conversion Script by @abheesht17 in Add FNet Conversion Script #737
• Add BART Conversion Script by @abheesht17 in Add BART Conversion Script #739
• Pass Correct LayerNorm Epsilon value to TransformerEncoder in Backbones by @TheAthleticCoder in Pass Correct LayerNorm Epsilon value to TransformerEncoder in Backbones #731
• Improving the layer Description. by @Neeshamraghav012 in Improving the layer Description. #734
• Adding ragged support to SinePositionEncoding by @apupneja in Adding ragged support to SinePositionEncoding #751
• Fix trailing space by @mattdangerw in Fix trailing space #755
• Adding an AlbertMaskedLM task model and preprocessor by @shivance in Adding an AlbertMaskedLM task + Fix Projection layer dimension in MaskedLMHead #725
• New docstring example for TokenAndPosition Embedding layer. by @Neeshamraghav012 in New docstring example for TokenAndPosition Embedding layer. #760
• Add a note for TPU issues for deberta_v3 by @mattdangerw in Add a note for TPU issues for deberta_v3 #758
• Add missing exports to models API by @mattdangerw in Add missing exports to models API #763
• Autogenerate preset table by @Cyber-Machine in Autogenerate preset table #690
• Remove work in progress API for 0.4.1 release by @mattdangerw in Remove work in progress API for 0.4.1 release #765
• Version bump to 0.4.1.dev0 by @mattdangerw in Version bump to 0.4.1.dev0 #766
• Version bump to 0.4.1 by @mattdangerw in Version bump to 0.4.1 #768

New Contributors

• @haifeng-jin made their first contribution in Update python version in readme to 3.8 #618
• @mbrukman made their first contribution in Clean up class name and self in calls to super() #628
• @soma2000-lang made their first contribution in Corrected the epsilon value  #665
• @NusretOzates made their first contribution in Light-weight benchmarking script #664
• @shivance made their first contribution in Move from_preset to base tokenizer classes #673
• @Plutone11011 made their first contribution in Default compilation other classifiers #714
• @TheAthleticCoder made their first contribution in Pass Correct LayerNorm Epsilon value to TransformerEncoder in Backbones #731
• @Neeshamraghav012 made their first contribution in Improving the layer Description. #734
• @Cyber-Machine made their first contribution in Autogenerate preset table #690

Full Changelog: v0.4.0...v0.4.1

---

This discussion was created from the release v0.4.1.

[shivance]

@mattdangerw We also fixed Mistake in projection layer dimension of MaskedLMHead #733 through #725

[mattdangerw]

@shivance indeed we did! Though actually we are not exposing the new XXMaskedLM task models quite yet (because they are still in the process of landing for all models). Overall that is something we will highlight more for the upcoming 0.5 release.

The fix for MaskedLMHead head is live on 0.4.1 though and much appreciated!