Inconsistent Result while using a fix random seed

[swageeth]

I have been using Spacy - Text Categoriser to classify text messages. Python version 3.10

Below is the the training model and I get the Unknown category with the highest score for this specific message:

#Import training data
with open ('SID - Commercial.txt', "r", encoding="utf8") as a:
    Commercial = a.read().splitlines()

with open ('SID - Crypto.txt', "r", encoding="utf8") as b:
    Crypto = b.read().splitlines()
    
with open ('SID - Extortion.txt', "r", encoding="utf8") as c:
    Extortion = c.read().splitlines()
    
with open ('SID - Financial.txt', "r", encoding="utf8") as d:
    Financial = d.read().splitlines()
    
with open ('SID - Gambling.txt', "r", encoding="utf8") as e:
    Gambling = e.read().splitlines()
    
with open ('SID - Gift.txt', "r", encoding="utf8") as f:
    Gift = f.read().splitlines()

with open ('SID - Investment.txt', "r", encoding="utf8") as g:
    Investment = g.read().splitlines()    

with open ('SID - Invoice.txt', "r", encoding="utf8") as h:
    Invoice = h.read().splitlines()  
    
with open ('SID - Phishing.txt', "r", encoding="utf8") as i:
    Phishing = i.read().splitlines() 
    
with open ('SID - Romance.txt', "r", encoding="utf8") as j:
    Romance = j.read().splitlines() 

with open ('SID - Unknown.txt', "r", encoding="utf8") as k:
    Unknown = k.read().splitlines() 
    
data = {}
data["Commercial"] = Commercial
data["Crypto"] = Crypto
data["Extortion"] = Extortion
data["Financial"] = Financial
data["Gambling"] = Gambling
data["Gift"] = Gift
data["Investment"] = Investment
data["Invoice"] = Invoice
data["Phishing"] = Phishing
data["Romance"] = Romance
data["Unknown"] = Unknown

# NLP model
spacy.util.fix_random_seed(0)
nlp = spacy.load("en_core_web_md")
nlp.add_pipe("text_categorizer", 
    config={
        "data": data,
        "model": "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
        "cat_type": "multi-label",
        "device": "gpu"
    }
)

print(nlp("FW: ��: ��� ���� ��� (�) ����� ������� ��������. #622460835")._.cats)

Result (which is correct as the Unknown as the highest score):
{'Commercial': 0.13948287736862833, 'Crypto': 0.015437351941468657, 'Extortion': 0.0860014895963152, 'Financial': 0.01987490991768424, 'Gambling': 0.029074990906618126, 'Gift': 0.06850244399154756, 'Investment': 0.012729882351053419, 'Invoice': 0.0718818617408037, 'Phishing': 0.046637490542787444, 'Romance': 0.05515818363916855, 'Unknown': 0.45521851800392493}

Importing test dataset:

# Import the dataset and assign scores
Messages = pd.read_csv('November SID2.csv', encoding='utf8')
    
Messages['Body'] = Messages['Body'].astype(str)
Messages['NLP_Result'] = Messages['Body'].apply(lambda x: nlp(x)._.cats)

#Find the category based on highest score and join back to the original dataset

Scores = pd.json_normalize(Messages.NLP_Result)
Scores['Category'] = Scores.idxmax(axis=1)
Scores['Category'] = Scores['Category'].replace('_', ' ', regex=True)

Messages_Final = pd.concat([Messages, Scores], axis=1)
Messages_Final.to_csv('out.csv', index=False)

The result in csv file for that same statement:

<style> </style>

Body	NLP_Result	Commercial	Crypto	Extortion	Financial	Gambling	Gift	Investment	Invoice	Phishing	Romance	Unknown	Category
FW: ��: ��� ���� ��� (�) ����� ������� ��������. #622460835	{'Commercial': 0.03343028275903707, 'Crypto': 0.012076486026176284, 'Extortion': 0.08983918751534335, 'Financial': 0.07360790896376578, 'Gambling': 0.014564933067751274, 'Gift': 0.08460245841797985, 'Investment': 0.017324353297565327, 'Invoice': 0.1522007262418396, 'Phishing': 0.4507937431127887, 'Romance': 0.010566873139864728, 'Unknown': 0.060993047457888194}	0.03343	0.012076	0.089839	0.073608	0.014565	0.084602	0.017324	0.152201	0.450794	0.010567	0.060993	Phishing

Why are they inconsistent even when my training model has 'spacy.util.fix_random_seed(0)'?

Thank you

[polm]

Two things:

1. please don't post screenshots of text or code, please paste them as text using markdown code blocks. Screenshots are very hard to read or make judgements from. For example, we can't tell that your "specific example" is the same.
2. Classy classification is a a third-party component that we don't develop, so it might be better to ask at their repo.

[swageeth]

Thank you, I will post on Classy repo to see whether they have a solution as well. Regarding what you mentioned -
Using GPU means it would be used for both training and test, correct? Do you recommend CPU instead?

I used UTF8 encoding in this case. Is there a better encoding that could be used for these sorts of texts? Unfortunately, I don’t have much control over the input dataset to make it better as it’s how the specific system export to the user. So wondering whether there is anything I could do in the code to improve the model.

[polm]

Regarding what you mentioned -Using GPU means it would be used for both training and test, correct? Do you recommend CPU instead?

It looks like you're not doing any training. I wouldn't recommend CPU as such - it will be too slow with transformers - but GPUs introduce more randomness issues.

I used UTF8 encoding in this case. Is there a better encoding that could be used for these sorts of texts? Unfortunately, I don’t have much control over the input dataset to make it better as it’s how the specific system export to the user. So wondering whether there is anything I could do in the code to improve the model.

There's no problem with UTF8, I wouldn't suggest you use any other encoding. The problem is that the example text you have is not valid UTF8, it's mojibake - it looks like it wasn't UTF8 but you loaded it as that anyway. I wouldn't trust the result of any classifier on it. You need to fix your encoding issue. Maybe look at ftfy.

[swageeth]

Thank you. I think the issue is it has both mojibake and foreign characters. Do you know any easy way to filter all mojibake and foreign characters out before the text is fed into Spacy by any chance?
I asked this same question in the classy repo and this is the response I got - "This is the case because the classifiers are not spacy specific. They use torch or scikit-learn." I'm just not sure how that would lead to inconsistent results.

[polm]

You can use a language detection library or Python's isascii to check if a string contains mojibake.

Again, I suggest you fix your mojibake by checking the encoding of your input text when you load it and correcting it. It looks like it's an email subject - note that subjects can have a different encoding than the body of the mail.

[polm]

I asked this same question in the classy repo and this is the response I got - "This is the case because the classifiers are not spacy specific. They use torch or scikit-learn." I'm just not sure how that would lead to inconsistent results.

To link to your issue... I think the idea is that the spaCy function only removes randomness in spaCy, but if you look at the implementation it's using the Python random library and torch. So I also don't understand that answer.