1.6.2

to_tensor only for requested fileds. Use add_to_tensor_inputs to add custom fields.

1.6.1

to_tensor only for valid instances

1.6

• pad_batch function more general
• pad logic for custom fields using add_padding_ops
• model output can include all the HuggingFace outputs, using return_all=True in TransformerEmbedder

1.5.1

Exposed has_double_sep and has_starting_token as Tokenizer properties

1.4.8

Fix Tokenizer regression

1.4.7

• Added properties to Tokenizer, exposing model special tokens
• Expose num_special_tokens to get the number of special tokens the model needs

1.4.6

Exposed save_pretrained in TransformerEmbedder

1.4.5

Fix sentence length bug

1.4.4

Fixed bug with double-SEP models

1.4.3

Return dictionary has word_mask parameter, in the size of the original sentence (before sub-tokens)