Remove punctuation ';"{}[]/?.,! Lowercase the data since it is going to happen anyway also stopwords are lower case Try different embedding See if better performance with stopwords included Find out what words are not in the embeddings?
Include magic features Start recording statistics Try to merge less popular words?