Dataset too large

[NHendrickson9616]

I am using the run_mlm.py file but I have my own copy because I changed where the tokenizer is going to since it is a different path from the model which is local.

While intially working with this method, I used the first two lines of my dataset and it was working just fine, but now that I have expanded the input, I am getting this error:

IndexError                                Traceback (most recent call last)
Cell In[58], line 5
      3 scorer = MaskedLM('/data/user/home/nchendri/LongRun/')
      4 text =  dsMap['test']['text']
----> 5 ppl = scorer.get_perplexity(text, batch=32)
      6 print(ppl)
      7 print(list(zip(text, ppl)))

Cell In[57], line 162, in MaskedLM.get_perplexity(self, input_texts, batch)
    159     return _e
    161 if self.max_length is not None:
--> 162     data.append([encode_mask(i) for i in range(min(self.max_length - len(self.sp_token_prefix), len(x)))])
    163 else:
    164     data.append([encode_mask(i) for i in range(len(x))])

Cell In[57], line 162, in <listcomp>(.0)
    159     return _e
    161 if self.max_length is not None:
--> 162     data.append([encode_mask(i) for i in range(min(self.max_length - len(self.sp_token_prefix), len(x)))])
    163 else:
    164     data.append([encode_mask(i) for i in range(len(x))])

Cell In[57], line 157, in MaskedLM.get_perplexity.<locals>.encode_mask(mask_position)
    155 # add the correct token id as the label
    156 label = [PAD_TOKEN_LABEL_ID] * _e['input_ids'].shape[1]
--> 157 label[mask_position + len(self.sp_token_prefix)] = masked_token_id
    158 _e['labels'] = torch.tensor([label], dtype=torch.long)
    159 return _e

IndexError: list assignment index out of range

[asahi417]

Would it possible to share the sentence that causes the error? I can run it locally to see if I can reproduce the error.

[NHendrickson9616]

It is not a sentence, it is a a list of lines from .txt files, so it is essentially a lot of sentences in a list and I want to evaluate them all

[NHendrickson9616]

And unfortunately I cannot share the dataset anywhere because it has PHI in it

[asahi417]

Ok. Probably, something to do with the sequence length. Which model are you using to compute perplexity? Also, could you tell me the number of the maximum character in a single line in your file?

[NHendrickson9616]

I am fairly sure that it is 512, I think I need help on how to batch my dataset

[asahi417]

If you run it on a single instance, instead of passing a list, you should be able to find the line that causes the error.

text =  dsMap['test']['text']
for t in text:
    ppl = scorer.get_perplexity([t])

[NHendrickson9616]

I am running a version of BERT, but the line that caused issues has only 129 characters in it

[asahi417]

Does the sentence contain only roman characters?

[NHendrickson9616]

It has dashes, commas, a period, and a colon, I don't know if those count, but other than that, yes

[NHendrickson9616]

I have pretrained BERT on this same dataset before with the same tokenizer and model.

[asahi417]

Could you try to compute perplexity on the same files but with roberta-base? It might be BERT specific issue.

[NHendrickson9616]

Same error, same spot

[NHendrickson9616]

But it did take it a little longer to get there although that may be because it was loading a new model in

[asahi417]

So RoBERTa and BERT raises the same error in the same line, correct?

[NHendrickson9616]

Correct.