Misaligned tokenizer on embeddings?

[eugeniosegala]

Issue description

The nomic-embed-text-v1.5-GGUF model does not seem to calculate embedding dimensions properly.

Expected Behavior

No errors or warnings, embedding dimensions are calculated properly. As per now, I'm getting non-relevant embeddings during similarity searches.

It works okay with Ollama and the Python bindings of llama.cpp.

Actual Behavior

These errors/warnings is produced when loading the model:

[node-llama-cpp] llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
[node-llama-cpp] Using this model ("./nomic-embed-text-v1.5.f16.gguf") to tokenize text and then detokenize it resulted in a different text. There might be an issue with the model or the tokenizer implementation. Using this model may not work as intended

As you will notice, the embeddings are generated but I suspect the way they are calculated is wrong, as similarity searches on vectors do not relevant content.

Steps to reproduce

Download the model https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF

And then run it using node-llama-cpp:

import { getLlama } from "node-llama-cpp";
import path from "path";
import { fileURLToPath } from "url";

const __dirname = path.dirname(fileURLToPath(import.meta.url));

async function embedDocuments(documents) {
  const embeddings = new Map();

  await Promise.all(
    documents.map(async (document) => {
      const embedding = await context.getEmbeddingFor(document);
      embeddings.set(document, embedding);

      console.debug(
        `${embeddings.size}/${documents.length} documents embedded`
      );
    })
  );

  return embeddings;
}

function findSimilarDocuments(
  embedding,
  documentEmbeddings,
) {
  const similarities = new Map();
  for (const [otherDocument, otherDocumentEmbedding] of documentEmbeddings)
    similarities.set(
      otherDocument,
      embedding.calculateCosineSimilarity(otherDocumentEmbedding)
    );

  return Array.from(similarities.keys())
    .sort((a, b) => similarities.get(b) - similarities.get(a));
}

const llama = await getLlama();

// https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF
const model = await llama.loadModel({
  modelPath: path.join(__dirname, "nomic-embed-text-v1.5.f16.gguf"),
});

const context = await model.createEmbeddingContext();

const response = await context.getEmbeddingFor("crime");


const documentEmbeddings = await embedDocuments([
  "The sky is clear and blue today",
  "I love eating pizza with extra cheese",
  "Dogs love to play fetch with their owners",
  "The capital of France is Paris",
  "Drinking water is important for staying hydrated",
  "Mount Everest is the tallest mountain in the world",
  "A warm cup of tea is perfect for a cold winter day",
  "Painting is a form of creative expression",
  "Not all the things that shine are made of gold",
  "Cleaning the house is a good way to keep it tidy"
]);

const query = "Do you like pizza?";
const queryEmbedding = await context.getEmbeddingFor(query);

const similarDocuments = findSimilarDocuments(
  queryEmbedding,
  documentEmbeddings
);
const topSimilarDocument = similarDocuments[0];

console.log("query:", query);
console.log("Document:", topSimilarDocument); // Drinking water is important for staying hydrated

The returned vector will be:

"Drinking water is important for staying hydrated"

But it should be:

"I love eating pizza with extra cheese",

My Environment

OS: macOS 24.1.0 (arm64)
Node: 20.18.1 (arm64)
TypeScript: 5.6.3
node-llama-cpp: 3.3.0

Metal: available

Metal device: Apple M3 Pro
Metal used VRAM: 0% (80KB/27GB)
Metal free VRAM: 99.99% (27GB/27GB)
Metal unified memory: 27GB (100%)

CPU model: Apple M3 Pro
Math cores: 6
Used RAM: 96.66% (34.8GB/36GB)
Free RAM: 3.33% (1.2GB/36GB)
Used swap: 0% (0B/0B)
Max swap size: dynamic

[eugeniosegala]

I tested the exact same model version in both Ollama and node-llama-cpp, strangely I'm getting different embeddings back.

Ollama is working properly for some reasons.

I also tried to convert the latest version of nomic-embed-text using llama.cpp but I had no luck, I still get very strange embeddings + the usual errors reported above.

[eugeniosegala]

I can confirm that the equivalent binding library in Python works as Ollama.

I think there's a problem specifically inside node-llama-cpp.

I'm trying to debug it/solve it. I suspect is has to do with the tokenisation process.

[eugeniosegala]

Exact same logic in Python:

import numpy as np
from llama_cpp import Llama

def embed_documents(llm, documents):
    embeddings = {}
    for idx, document in enumerate(documents):
        embedding = llm.create_embedding(document)
        embeddings[document] = embedding["data"][0]["embedding"]
        print(f"{idx + 1}/{len(documents)} documents embedded")
    return embeddings

def find_similar_documents(query_embedding, document_embeddings):
    similarities = {}
    query_vector = np.array(query_embedding)
    for doc, embedding in document_embeddings.items():
        other_vector = np.array(embedding)
        # Compute cosine similarity
        similarity = np.dot(query_vector, other_vector) / (
            np.linalg.norm(query_vector) * np.linalg.norm(other_vector)
        )
        similarities[doc] = similarity

    # Sort documents by similarity in descending order
    sorted_docs = sorted(similarities.items(), key=lambda x: x[1], reverse=True)
    return sorted_docs

# Initialize the Llama model
model_path = "output_file.gguf"
llm = Llama(model_path=model_path, embedding=True, n_threads=8)

# Embed the query
query = "Do you like pizza?"
query_response = llm.create_embedding(query)

print(query_response)
query_embedding = query_response["data"][0]["embedding"]

# Embed a list of documents
documents = [
    "The sky is clear and blue today",
    "I love eating pizza with extra cheese",
    "Dogs love to play fetch with their owners",
    "The capital of France is Paris",
    "Drinking water is important for staying hydrated",
    "Mount Everest is the tallest mountain in the world",
    "A warm cup of tea is perfect for a cold winter day",
    "Painting is a form of creative expression",
    "Not all the things that shine are made of gold",
    "Cleaning the house is a good way to keep it tidy"
]

document_embeddings = embed_documents(llm, documents)

# Find similar documents to the query
similar_documents = find_similar_documents(query_embedding, document_embeddings)
top_similar_document = similar_documents[0]

# Output the results
print("Query:", query)
print("Top Similar Document:", top_similar_document[0])
print("Similarity Score:", top_similar_document[1])

Result: I love eating pizza with extra cheese

[giladgd]

@eugeniosegala Thanks for investigating this issue and providing examples.
I have an idea of what may be the cause of this; I'll take a look and let you know what I find.

[giladgd]

@eugeniosegala It appears I overlooked the WPM vocabulary type models case in the tokenization of the input, which caused it to behave a bit differently with some models in some cases.
I've opened PR #393 to fix that, which includes a test based on your example.
Thanks for reporting and looking into it :)

[eugeniosegala]

Thank you @giladgd! Much appreciated.

[github-actions]

🎉 This issue has been resolved in version 3.3.1 🎉

The release is available on:

• v3.3.1
• npm package (@latest dist-tag)
• GitHub release
• v3.3.1

Your semantic-release bot 📦🚀