Clarification on Padding Process in BERT Model Construction #107
Description
Hi there,
In my endeavor to construct a BERT model from the ground up for the purpose of gaining hands-on experience and a comprehensive understanding of the model, I have encountered a point of confusion regarding the padding process.
According to my interpretation of the dataset.py file, the standard procedure involves concatenating two sentences and adding pad tokens at the end of the combined sequence, similar to typical padding practices in other tasks. Here's an example:
Sent_A= The cat is walking (Len is 4)
Sent_B= The dog is barking at the tree (Len is 7)
Maximum sequence length: 15
Sequence: The cat is walking The dog is barking at the tree [PAD] [PAD] [PAD] [PAD]
Segment: 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0
However, in alternative implementations 1 2, I've observed padding aimed at equalizing the lengths of the two sentences, resulting in a structure like this:
Sent_A= The cat is walking (Len is 4)
Sent_B= The dog is barking at the tree (Len is 7)
Maximum sequence length: 15
Sequence: The cat is walking [PAD] [PAD] [PAD] The dog is barking at the tree [PAD]
Segment: 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0
This discrepancy has left me in a state of uncertainty—whether my understanding from the dataset.py file is accurate, or if the alternative implementations I've come across have a different rationale behind their padding approach. Could you kindly provide clarification on this matter?