[chapter 02]Why need special process for whitespace in SimpleTokenizerV1

[myme5261314]

We could see SimpleTokenizerV1's code in 30th cell of ch02/01_main-chapter-code/ch02.ipynb.

class SimpleTokenizerV1:
    def __init__(self, vocab):
        self.str_to_int = vocab
        self.int_to_str = {i:s for s,i in vocab.items()}
    
    def encode(self, text):
        preprocessed = re.split(r'([,.:;?_!"()\']|--|\s)', text)
                                
        preprocessed = [
            item.strip() for item in preprocessed if item.strip()
        ]
        ids = [self.str_to_int[s] for s in preprocessed]
        return ids
        
    def decode(self, ids):
        text = " ".join([self.int_to_str[i] for i in ids])
        # Replace spaces before the specified punctuations
        text = re.sub(r'\s+([,.?!"()\'])', r'\1', text)
        return text

I've append two lines of code in 31th cell of ch02/01_main-chapter-code/ch02.ipynb.

tokenizer = SimpleTokenizerV1(vocab)

text = """"It's the last he painted, you know," 
           Mrs. Gisburn said with pardonable pride."""
ids = tokenizer.encode(text)
print(ids)
print(tokenizer.decode(ids))
print(text)

The running result is like below:

[1, 56, 2, 850, 988, 602, 533, 746, 5, 1126, 596, 5, 1, 67, 7, 38, 851, 1108, 754, 793, 7]
" It' s the last he painted, you know," Mrs. Gisburn said with pardonable pride.
"It's the last he painted, you know," 
           Mrs. Gisburn said with pardonable pride.

As we can see, the text does not equal to its result after .encode.decode, the differences lie in the whitespace and the newline characters.

I don't know why we need to treat whitespace specially through str.stirp() and " ".join in SimpleTokenizerV1.
Maybe because whitespace has less meaning in English?
But consider some other languages which whitespace is much more meaningful, such as Chinese.
There's no default whitespace between words and punctuation marks.

[rasbt]

That's a good point. To be honest, I just implemented something super simple here for illustration purposes. We don't end up using this tokenizer later in the book. Btw I have a BPE from-scratch implementation here: https://github.com/rasbt/LLMs-from-scratch/blob/main/ch02/05_bpe-from-scratch/bpe-from-scratch.ipynb