On padding the input with -1

[hquan7395]

The comment at

event-ssm/event_ssm/dataloading.py

Line 89 in a127667

# pad tokens with -1, which results in a zero vector with embedding look-ups

says that padding -1 corresponds to zero in the embedding layer. But when I check the trained params with

state['params']['encoder']['encoder']['embedding'][-1]

I got

Array([-0.06191076, 0.18668179, -0.25837645, 0.1488393 , 0.11525428, 0.18183647, -0.30500978, 0.07464553, -0.3888737 , 0.10938968, -0.02655256, -0.349307 , -0.03291983, -0.040106 , 0.2369387 , -0.01890478, -0.21027383, -0.00453783, -0.1900245 , 0.01763803, -0.09983198, -0.3064041 , -0.14400026, -0.05854894, 0.10559192, 0.04855605, 0.1330904 , 0.00867153, -0.04014621, 0.23416986, 0.14465654, 0.3036504 ], dtype=float32)

Is this not intended?

[markschoene]

Flax linen nn.Embed returns a zero vector for negative indices. However, explicitly querying [-1] on the tensor will return the last entry of the tensor sliced along the corresponding dimension (typical Python behavior). I’d suggest to test by calling the embedding object directly instead of the state dict. If this still yields non zero tensors, use the version indicated in the README.md

In any case, if you use the classification model, there will be a mask applied to the loss that removes all the terms that were tokenized as -1 anyways. The above steps should help even if you want to use the model without masking.

[markschoene]

Could you let me know if this addresses your concerns? I’d be happy to answer more questions if there are any in the process.