Quotation marks on lemmatization after read file with Pandas

[k4waiiiii]

I have tokenization and lemmatization then saved it to a file called 'data after lemmatization.xlsx'

This is the data before saved it to file:

This is the data after saved it to file with Pandas and then read the data file using Pandas:

Why I have quotation marks on column lemmatization but on column tokenizing is not?

import pandas as pd
from spacy.lang.id import Indonesian

nlp = Indonesian()
nlp.add_pipe('lemmatizer')
nlp.initialize()

data = [
    'pergi untuk melakukan penanganan banjir'
]

df = pd.DataFrame({'text': data})

#Tokenization
def tokenizer(words):
    return [token for token in nlp(words)]

#Lemmatization
def lemmatizer(token):
    return [lem.lemma_ for lem in token]


df['tokenizing'] = df['text'].apply(tokenizer)
df['lemmatizaton'] = df['tokenizing'].apply(lemmatizer)
df.to_excel('./data after lemmatization.xlsx', index=False)

dc = pd.read_excel('./data after lemmatization.xlsx')
dc

[polm]

It looks like the contents of your tokenizing column are Tokens, while the contents of your lemmatization column are strings. These are displayed differently because of the way repr works in Python - it renders strings with single quotes, and spaCy tokens just print the text.

You can make them the same by converting the spaCy tokens to strings.

Note this isn't really a spaCy specific issue.

[k4waiiiii]

How converting the spaCy tokens to strings? Trying with token.text but got an error on lemma_ the error is AttributeError: 'str' object has no attribute 'lemma_'

[polm]

token.text is the right way to get a string from a Token. But strings don't have lemmas, like the error says. You have to:

1. Get the tokens
2. Get the lemmas from the tokens
3. Convert the tokens to strings

It seems like what you are currently doing is:

1. Get the tokens
2. Convert the tokens to strings
3. Get the lemmas from the strings (won't work)

[k4waiiiii]

Ok. I see. Thanks for answering.