You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
-[ ] An `xx/xx_backbone.py` file which has the model graph \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_backbone.py)\].
26
-
-[ ] An `xx/xx_backbone_test.py` file which has unit tests for the backbone \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_backbone_test.py)\].
25
+
-[ ] An `xx/xx_backbone.py` file which has the model graph \[[Example](https://github.com/keras-team/keras-hub/blob/master/keras_hub/src/models/distil_bert/distil_bert_backbone.py)\].
26
+
-[ ] An `xx/xx_backbone_test.py` file which has unit tests for the backbone \[[Example](https://github.com/keras-team/keras-hub/blob/master/keras_hub/src/models/distil_bert/distil_bert_backbone_test.py)\].
27
27
-[ ] A Colab notebook link in the PR description which matches the outputs of the implemented backbone model with the original source \[[Example](https://colab.research.google.com/drive/1SeZWJorKWmwWJax8ORSdxKrxE25BfhHa?usp=sharing)\].
28
28
29
29
### Step 3: PR #2 - Add XXTokenizer
30
30
31
-
-[ ] An `xx/xx_tokenizer.py` file which has the tokenizer for the model \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_tokenizer.py)\].
32
-
-[ ] An `xx/xx_tokenizer_test.py` file which has unit tests for the model tokenizer \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_tokenizer_test.py)\].
31
+
-[ ] An `xx/xx_tokenizer.py` file which has the tokenizer for the model \[[Example](https://github.com/keras-team/keras-hub/blob/master/keras_hub/src/models/distil_bert/distil_bert_tokenizer.py)\].
32
+
-[ ] An `xx/xx_tokenizer_test.py` file which has unit tests for the model tokenizer \[[Example](https://github.com/keras-team/keras-hub/blob/master/keras_hub/src/models/distil_bert/distil_bert_tokenizer_test.py)\].
33
33
-[ ] A Colab notebook link in the PR description, demonstrating that the output of the tokenizer matches the original tokenizer \[[Example](https://colab.research.google.com/drive/1MH_rpuFB1Nz_NkKIAvVtVae2HFLjXZDA?usp=sharing)].
34
34
35
35
### Step 4: PR #3 - Add XX Presets
36
36
37
-
-[ ] An `xx/xx_presets.py` file with links to weights uploaded to a personal GCP bucket/Google Drive \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_presets.py)\].
38
-
-[ ] A `tools/checkpoint_conversion/convert_xx_checkpoints.py` which is reusable script for converting checkpoints \[[Example](https://github.com/keras-team/keras-nlp/blob/master/tools/checkpoint_conversion/convert_distilbert_checkpoints.py)\].
37
+
-[ ] An `xx/xx_presets.py` file with links to weights uploaded to a personal GCP bucket/Google Drive \[[Example](https://github.com/keras-team/keras-hub/blob/master/keras_hub/src/models/distil_bert/distil_bert_presets.py)\].
38
+
-[ ] A `tools/checkpoint_conversion/convert_xx_checkpoints.py` which is reusable script for converting checkpoints \[[Example](https://github.com/keras-team/keras-hub/blob/master/tools/checkpoint_conversion/convert_distilbert_checkpoints.py)\].
39
39
-[ ] A Colab notebook link in the PR description, showing an end-to-end task such as text classification, etc. The task model can be built using the backbone model, with the task head on top \[[Example](https://gist.github.com/mattdangerw/bf0ca07fb66b6738150c8b56ee5bab4e)\].
40
40
41
41
### Step 5: PR #4 and Beyond - Add XX Tasks and Preprocessors
42
42
43
43
This PR is optional.
44
44
45
-
-[ ] An `xx/xx_<task>.py` file for adding a task model like classifier, masked LM, etc. \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_classifier.py)\]
46
-
-[ ] An `xx/xx_<task>_preprocessor.py` file which has the preprocessor and can be used to get inputs suitable for the task model \[[Example](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_preprocessor.py)\].
47
-
-[ ]`xx/xx_<task>_test.py` file and `xx/xx_<task>_preprocessor_test.py` files which have unit tests for the above two modules \[[Example 1](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_classifier_test.py) and [Example 2](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_preprocessor_test.py)\].
45
+
-[ ] An `xx/xx_<task>.py` file for adding a task model like classifier, masked LM, etc. \[[Example](https://github.com/keras-team/keras-hub/blob/master/keras_hub/src/models/distil_bert/distil_bert_classifier.py)\]
46
+
-[ ] An `xx/xx_<task>_preprocessor.py` file which has the preprocessor and can be used to get inputs suitable for the task model \[[Example](https://github.com/keras-team/keras-hub/blob/master/keras_hub/src/models/distil_bert/distil_bert_preprocessor.py)\].
47
+
-[ ]`xx/xx_<task>_test.py` file and `xx/xx_<task>_preprocessor_test.py` files which have unit tests for the above two modules \[[Example 1](https://github.com/keras-team/keras-hub/blob/master/keras_hub/src/models/distil_bert/distil_bert_classifier_test.py) and [Example 2](https://github.com/keras-team/keras-hub/blob/master/keras_hub/src/models/distil_bert/distil_bert_preprocessor_test.py)\].
48
48
-[ ] A Colab notebook link in the PR description, demonstrating that the output of the preprocessor matches the output of the original preprocessor \[[Example](https://colab.research.google.com/drive/1GFFC7Y1I_2PtYlWDToqKvzYhHWv1b3nC?usp=sharing)].
49
49
50
50
## Detailed Instructions
@@ -54,7 +54,7 @@ This section discusses, in details, every necessary step.
54
54
### Step 1: Open an issue/Find an open issue
55
55
56
56
Before getting started with the code, it's important to check if there are any
@@ -117,13 +117,13 @@ This is when things might slightly get complicated.
117
117
If the model introduces a paradigm shift, such as using relative attention instead
118
118
of vanilla attention, the contributor will have to implement complete custom layers. A case
119
119
in point is `keras_hub.models.DebertaV3Backbone` where we had to [implement layers
120
-
from scratch](https://github.com/keras-team/keras-nlp/tree/master/keras_hub/models/deberta_v3).
120
+
from scratch](https://github.com/keras-team/keras-hub/tree/master/keras_hub/models/deberta_v3).
121
121
122
122
On the other hand, if the model has a small tweak, something simpler can be done.
123
123
For instance, in the Whisper model, the self-attention and cross-attention mechanism
124
124
is exactly the same as vanilla attention, with the exception that the key projection
125
125
layer does not have a bias term. In this case, we can inherit the custom layer
126
-
from one of the standard layers and make minor modifications. See [this PR](https://github.com/keras-team/keras-nlp/pull/801/files#diff-8533ae3a7755c0dbe95ccbb71f85c677297f687bf3884fadefc64f1d0fdce51aR22) for
126
+
from one of the standard layers and make minor modifications. See [this PR](https://github.com/keras-team/keras-hub/pull/801/files#diff-8533ae3a7755c0dbe95ccbb71f85c677297f687bf3884fadefc64f1d0fdce51aR22) for
127
127
more details.
128
128
129
129
Since the first PR is only to add the model backbone class, you should omit the
@@ -165,7 +165,7 @@ All the underlying actual tokenization will be taken care of by the superclass.
165
165
The important thing here is adding "special tokens". Most models have
166
166
special tokens such as beginning-of-sequence token, end-of-sequence token,
167
167
mask token, pad token, etc. These have to be
168
-
[added as member attributes](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_tokenizer.py#L91-L105)
168
+
[added as member attributes](https://github.com/keras-team/keras-hub/blob/master/keras_hub/src/models/distil_bert/distil_bert_tokenizer.py#L91-L105)
169
169
to the tokenizer class. These member attributes are then accessed by the
170
170
preprocessor layers.
171
171
@@ -193,7 +193,7 @@ files. These files will then be uploaded to GCP by us!
193
193
After wrapping up the preset configuration file, you need to
194
194
add the `from_preset` function to all three classes, i.e., `DistilBertBackbone`,
The testing for presets is divided into two: "large" and "extra large".
199
199
For "large" tests, we pick the smallest preset (in terms of number of parameters)
@@ -228,9 +228,9 @@ and return the dictionary in the form expected by the model.
228
228
229
229
The preprocessor class might have a few intricacies depending on the model. For example,
230
230
the DeBERTaV3 tokenizer does not have the `[MASK]` in the provided sentencepiece
231
-
proto file, and we had to make some modifications [here](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/models/deberta_v3/deberta_v3_preprocessor.py). Secondly, we have
231
+
proto file, and we had to make some modifications [here](https://github.com/keras-team/keras-hub/blob/master/keras_hub/models/deberta_v3/deberta_v3_preprocessor.py). Secondly, we have
232
232
a separate preprocessor class for every task. This is because different tasks
233
-
might require different input formats. For instance, we have a [separate preprocessor](https://github.com/keras-team/keras-nlp/blob/master/keras_hub/src/models/distil_bert/distil_bert_masked_lm_preprocessor.py)
233
+
might require different input formats. For instance, we have a [separate preprocessor](https://github.com/keras-team/keras-hub/blob/master/keras_hub/src/models/distil_bert/distil_bert_masked_lm_preprocessor.py)
234
234
for masked language modeling (MLM) for DistilBERT.
0 commit comments